Optimal approximation of continuous functions by very deep ReLU networks.

We prove that deep ReLU neural networks with conventional fully-connected architectures with $W$ weights can approximate continuous $\nu$-variate functions $f$ with uniform error not exceeding $a_\nu\omega_f(c_\nu W^{-2/\nu}),$ where $\omega_f$ is the modulus of continuity of $f$ and $a_\nu, c_\nu$ are some $\nu$-dependent constants. This bound is tight. Our construction is inherently deep and nonlinear: the obtained approximation rate cannot be achieved by networks with fewer than $\Omega(W/\ln W)$ layers or by networks with weights continuously depending on $f$.