The model is trained on labeled data, where the correct answer is provided for every input example.
Images with 28\times 28 pixels.
Data: \{x_i\}_{i=1}^N\subset \mathcal{X}, \{y_i\}_{i=1}^N\subset \mathcal{Y}.
\begin{aligned} X: \;&\Omega = \{1, \cdots, N\}\longrightarrow\mathcal{X} \\ \;\;i&\longmapsto x_i \end{aligned}
\begin{aligned} Y: \;&\Omega \longrightarrow\mathcal{Y} \\ \;\;i&\longmapsto y_i \end{aligned}
Model functions \rho_{\textbf{w}}: \mathcal{X}\to \mathcal{Y}, where \textbf{w}\in\mathcal{W} and \mathcal{W} is the parameter space.
A statistical divergence \ell:\mathbb{M}\times \mathbb{M}\to \mathbb{R}, where \mathbb{M} is a statistical manifold, and a cost function c:\mathcal{W}\mapsto\mathbb{R}
Our goal then becomes: find the parameter \textbf{w}^* that minimizes the function c(\textbf{w}).
Our ideal hypothesis then becomes: \rho_{\textbf{w}^*}.
\begin{aligned} n_{\textbf{w}}: \;&\mathbb{R}^{d_0}\longrightarrow\mathbb{R}^{d_1} \\ \;\;x_i&\longmapsto n_\textbf{w}(x_i)=\sigma\circ g_\textbf{w}(x_i) = \sigma\left( W x_i + b\right) \end{aligned}
where
\sigma:\mathbb{R}^{d_1} \to \mathbb{R}^{d_1}: activation function (ReLU, sigmoid, sign), acting componentwise;
g_\textbf{w}(x) = Wx+b: affine transformation, with W\in\mathbb{R}^{d_1\times d_0} the weight matrix and b\in\mathbb{R}^{d_1} the bias vector;
\textbf{w}=(W,b) is an element of the parameter space \mathcal{W}.
The most commonly used ones are:
Let n_j^{(p-1)} be the j-th neuron of layer p-1. Then the i-th neuron of layer p is given by n_i^{(p)}=\sigma^{(p)}\left(\sum_{j=1}^{d_{p-1}}w_{ij}^{(p-1)}\,n_j^{(p-1)}+b_i^{(p-1)}\right).
Neural networks in matrix form. Let P\in\mathbb{N}, d_0,d_1,\dots,d_{P+1}\in\mathbb{N} and let \sigma^{(l)}:\mathbb{R}^{d_l}\to\mathbb{R}^{d_l} be activation functions. A function \phi_\textbf{w}:\mathbb{R}^{d_0}\to\mathbb{R}^{d_{P+1}} is called a neural network of depth P if there exist matrices W^{(l)}\in\mathbb{R}^{d_{l+1}\times d_{l}} and vectors b^{(l)}\in\mathbb{R}^{d_{l+1}}, l=0,1,\dots,P, such that
and \phi_\textbf{w}(x)=N^{(P+1)}. Equivalently, \phi_\textbf{w}=\sigma^{(P+1)}\circ G^{(P+1)}\circ\sigma^{(P)}\circ G^{(P)}\circ\cdots\circ\sigma^{(1)}\circ G^{(1)}, \qquad \text{where } G^{(l)}(z)=W^{(l-1)}z+b^{(l-1)}.
We call P the depth and \max_{l} d_l the width; the vector \textbf{w}=\big((W^{(0)},b^{(0)}),\dots,(W^{(P)},b^{(P)})\big) collects all the parameters.
Composing the two neurons above with a third one, also ReLU:
The Universal Approximation Theorem!
(Compact convergence) Let d\in\mathbb{N}. A sequence of functions f_n:\mathbb{R}^d\to \mathbb{R}, n\in \mathbb{N}, converges compactly to f:\mathbb{R}^d\to \mathbb{R} if, for every compact set K\subseteq \mathbb{R}^d, we have \lim_{n\to \infty} \sup _{x\in K}|f_n(x) - f(x)| =0. In this case we write f_n\overset{cc}{\to} f.
(Universal approximator) Let d\in\mathbb{N}. A set of functions \mathcal{H} from \mathbb{R}^d to \mathbb{R} is a universal approximator (of C^{0}(\mathbb{R}^d)) if, for every \varepsilon>0, every compact set K\subseteq \mathbb{R}^d and every f\in C^0(\mathbb{R}^d), there exists g\in\mathcal{H} such that \sup_{x\in K}|f(x)-g(x)|<\varepsilon.
Lemma. Let d\in \mathbb{N} and let \mathcal{H} be a set of functions from \mathbb{R}^d to \mathbb{R}. Then \mathcal{H} is a universal approximator of C^0 if and only if C^0 (\mathbb{R}^d) \subseteq \overline{\mathcal{H}}^{cc}.
(Stone–Weierstrass theorem) Let d\in \mathbb{N}, let K\subseteq\mathbb{R}^d be compact and let \mathcal{H}\subseteq C^0(K,\mathbb{R}) satisfy
Then \mathcal{H} is dense in C^0(K).
(The space of neural networks) Let d,\,m,\,L,\,n\in\mathbb{N} and \sigma:\mathbb{R}\to\mathbb{R}. The set of all functions realized by neural networks with input dimension d, output dimension m, depth at most L, width at most n and activation \sigma is denoted by \mathcal{N}_d^m(\sigma;L,n) := \big\{\phi:\mathbb{R}^d\to \mathbb{R}^m \;\big|\; \phi \text{ a neural network},\ \operatorname{depth}(\phi)\le L,\ \operatorname{width}(\phi)\le n \big\}. Moreover, \mathcal{N}^m_d(\sigma; L) = \bigcup_{n\in \mathbb{N}}\mathcal{N}_d^m(\sigma;L,n).
Theorem (Universal Approximation — Leshno–Lin–Pinkus–Schocken; Petersen–Zech, Thm. 3.9) Let d\in \mathbb{N} and \sigma \in \mathcal{M}. Then \mathcal{N}_d^1(\sigma; 1) is a universal approximator of C^0(\mathbb{R}^d) if and only if \sigma is not a polynomial.
(Admissible set of activations) \mathcal{M} is the set of functions \sigma\in L^\infty_{\mathrm{loc}}(\mathbb{R}) for which there exist a finite M\in\mathbb{N} and intervals I_1,\dots,I_M partitioning \mathbb{R} with \sigma\in C^0(I_j) for every j=1,\dots,M.
Proof idea. The strategy comes down to the three reductions below:
If C^0(\mathbb{R}^1)\subseteq \overline{\mathcal{N}^1_1(\sigma; 1)}^{cc} then C^0(\mathbb{R}^d)\subseteq \overline{\mathcal{N}_d^1(\sigma; 1)}^{cc}.
If \sigma \in C^{\infty}(\mathbb{R}) is not a polynomial, then C^0(\mathbb{R}^1)\subseteq\overline{\mathcal{N}_1^1(\sigma; 1)}^{cc}.
If \sigma \in \mathcal{M} is not a polynomial, then there exists \tilde\sigma\in C^{\infty}(\mathbb{R}) \cap \overline{\mathcal{N}_1^1(\sigma; 1)}^{cc} which is not a polynomial.
Lemma. Assume that \mathcal{H} is a universal approximator of C^0(\mathbb{R}). Then, for every d\in\mathbb{N}, \operatorname{span} \{x\to g(\textbf{w}\cdot x) \;|\; \textbf{w}\in \mathbb{R}^d, g\in \mathcal{H}\} is a universal approximator of C^0(\mathbb{R}^d).
Lemma. If \sigma \in C^\infty(\mathbb{R}) and \sigma is not a polynomial, then \mathcal{N}^1_1(\sigma;1) is dense in C^0(\mathbb{R}).
Lemma. Let \sigma\in\mathcal{M}. Then, for every \varphi\in C_{c}^\infty(\mathbb{R}), we have \sigma * \varphi \in \overline{\mathcal{N}_1^1(\sigma;1)}^{cc}.
Lemma. If \sigma \in \mathcal{M} and \sigma * \varphi is a polynomial for every \varphi \in C_{c}^{\infty}(\mathbb{R}), then \sigma is a polynomial.
Given the samples \{(x_i,y_i)\}_{i=1}^{N}, we look for \textbf{w}^* \in \operatorname*{arg\,min}_{\textbf{w} \in \mathcal{W}} \; \frac{1}{N}\sum_{i=1}^{N} \ell\big(y_i, \phi_\textbf{w}(x_i)\big) \;=\; \operatorname*{arg\,min}_{\textbf{w} \in \mathcal{W}} c(\textbf{w}), where \ell(\cdot,\cdot) measures the divergence between the prediction and the expected value.
To solve \min_{\textbf{w}\in \mathcal{W}} c(\textbf{w}) we follow the direction -\nabla c, iterating \textbf{w}_{k+1}=\textbf{w}_k - \alpha_k \nabla c(\textbf{w}_k), where \alpha_k>0 is the learning rate.
Let f:\mathcal{W}\to \mathbb{R}. How do we describe its minimum point?
f:\mathcal{W}\to\mathbb{R} could, for instance, model the cost of the problems we have seen so far.
Given a point \textbf{w}_0\in \mathcal{W} and a retraction R on \mathcal{W}, the Riemannian gradient descent algorithm iterates \textbf{w}_{k+1}=R_{\textbf{w}_k}\big(-\alpha_k \operatorname{grad} f(\textbf{w}_k)\big), where \operatorname{grad} f(\textbf{w}_k)\in T_{\textbf{w}_k}\mathcal{W} is the Riemannian gradient.
A retraction on a manifold \mathcal{W} is a smooth map R:T\mathcal{W}\to\mathcal{W}, (\textbf{w},v)\mapsto R_\textbf{w}(v), such that, for every \textbf{w}\in\mathcal{W}:
In other words: R returns a point of the manifold and, to first order, behaves like the linear step we wanted to take in the tangent space. The Riemannian exponential map is one example.
Theorem (Local convergence). Let \mathcal{W} be a complete Riemannian manifold with a retraction R and let f:\mathcal{W}\to \mathbb{R} be smooth. Let w_*\in \mathcal{W} be a critical point of f, that is, \operatorname{grad} f(w_*)=0. Assume that all eigenvalues of \operatorname{Hess} f(w_*) are positive, say 0 < \lambda_1\le \cdots \le \lambda_{\max}<L. Define F:\mathcal{W}\to\mathcal{W} by F(w)=R_w\left(-\tfrac{1}{L} \operatorname{grad} f(w)\right). Then:
In other words: Riemannian gradient descent converges to the local minimum w_* provided that the initial point w_0 lies in U.
Assumption 1 (lower bound). f is bounded below: there exists f_{\text{low}}\in \mathbb{R} with f_{\text{low}}\le f(\textbf{w}) for every \textbf{w} \in \mathcal{W}.
Assumption 2 (regularity). Given S\subseteq T\mathcal{W}, there exists L>0 such that, for every v\in S_\textbf{w}, f\big(R_{\textbf{w}}(v)\big)\le f(\textbf{w})+\big\langle \operatorname{grad} f(\textbf{w}),\, v\big\rangle+\frac{L}{2}\|v\|^2.
This is the Riemannian version of the L-Lipschitz gradient condition: it guarantees that the linear model on T_\textbf{w}\mathcal{W} does not underestimate the function too much along the retraction.
Armijo–Goldstein condition f(\textbf{w})-f\big(R_\textbf{w}(-\alpha \operatorname{grad} f(\textbf{w}))\big) \;\ge\; r\,\alpha \,\|\operatorname{grad} f(\textbf{w})\|^2 for some constant r\in (0,1).
Backtracking line-search algorithm
Lemma. Let f be smooth on the Riemannian manifold \mathcal{W}. Fix \textbf{w}\in \mathcal{W}, a retraction R_\textbf{w}, an initial step \bar{\alpha}>0 and constants \tau, r \in (0,1). Assume that Assumption 2 holds for f\circ R_\textbf{w}(s) with constant L, where s=-\alpha\operatorname{grad} f(\textbf{w}), \alpha \in [0, \bar{\alpha}]. Then:
Proposition (criticality of accumulation points). Let f be smooth and satisfy Assumption 1 on a Riemannian manifold \mathcal{W}. Fix \tau, r\in(0,1) and S\subseteq T\mathcal{W} on which Assumption 2 holds with the retraction R. Assume:
Then \displaystyle\lim_{k\to\infty}\|\operatorname{grad} f(\textbf{w}_k)\|=0.
Proof idea. Since the learning rate satisfies the Armijo–Goldstein condition, f(\textbf{w}_k) - f\big(R_{\textbf{w}_k}(-\alpha_k \operatorname{grad} f(\textbf{w}_k))\big) \;\ge\; r\,\alpha_k\, \|\operatorname{grad} f(\textbf{w}_k)\|^2.
But the steps are bounded below: \alpha_k\ \ge\ \min\left(\bar{\alpha},\, \frac{2\tau(1-r)}{L}\right)=:d\ >\ 0 .
Hence, with c:=r\,d>0, f(\textbf{w}_k) - f(\textbf{w}_{k+1}) \;\ge\; c\,\|\operatorname{grad} f(\textbf{w}_k)\|^2 , that is, \{\textbf{w}_k\} is sufficiently decreasing.
Sufficiently decreasing. At every iteration the algorithm achieves sufficient decrease: there exists c>0 such that, for every k, f(\textbf{w}_k)-f(\textbf{w}_{k+1})\ \ge\ c\,\|\operatorname{grad} f(\textbf{w}_k)\|^2.
Lemma. Let f be smooth and satisfy Assumption 1 on a Riemannian manifold \mathcal{W}, and let w_0, w_1, w_2,\dots be the sequence of iterates of a sufficiently decreasing algorithm with constant c. Then \lim_{k\to \infty}\|\operatorname{grad} f(w_k)\| =0.
In other words: every accumulation point is critical.
A second-order model function would be m_\textbf{w}(s)= f(\textbf{w})+\big\langle\operatorname{grad} f(\textbf{w}),\,s\big\rangle +\tfrac{1}{2}\big\langle[\operatorname{Hess} f(\textbf{w})]s,\,s\big\rangle .
Newton’s algorithm:
Newton’s method has two drawbacks:
Conjugate gradient algorithm:
Trust-region algorithm