Seminars on Machine Learning, Geometry and Topology

Talk 1 and 2:
Introduction to Machine Learning and why do neural networks work?

Speaker: Monica Maria Funk Drechsler
PhD student in Mathematics — IME USP

Advisor: Marcos Alexandrino

What is machine learning?

  • Supervised learning:

The model is trained on labeled data, where the correct answer is provided for every input example.

  • Unsupervised learning: The model is trained on unlabeled data, and it looks for patterns and structure on its own.

Clustering MNIST — UMAP, uniform manifold approximation and projection for dimension reduction

Images with 28\times 28 pixels.

  • Reinforcement learning: The model learns by trial and error, receiving rewards or penalties for its actions.

Supervised learning in more detail

Data: \{x_i\}_{i=1}^N\subset \mathcal{X}, \{y_i\}_{i=1}^N\subset \mathcal{Y}.

  • \begin{aligned} X: \;&\Omega = \{1, \cdots, N\}\longrightarrow\mathcal{X} \\ \;\;i&\longmapsto x_i \end{aligned}

  • \begin{aligned} Y: \;&\Omega \longrightarrow\mathcal{Y} \\ \;\;i&\longmapsto y_i \end{aligned}

Model functions \rho_{\textbf{w}}: \mathcal{X}\to \mathcal{Y}, where \textbf{w}\in\mathcal{W} and \mathcal{W} is the parameter space.

  • \begin{aligned} Y_{\textbf W}: \;&\Omega \longrightarrow\mathcal{Y} \\ \;\;i&\longmapsto \rho_{\textbf w} \circ X(i) \end{aligned}

A statistical divergence \ell:\mathbb{M}\times \mathbb{M}\to \mathbb{R}, where \mathbb{M} is a statistical manifold, and a cost function c:\mathcal{W}\mapsto\mathbb{R}

  • \displaystyle c(\textbf{w}) \;=\; \frac{1}{N}\sum_{i=1}^{N}\ell\big(Y(i),\; \rho_{\textbf{w}}\circ X(i)\big)

Our goal then becomes: find the parameter \textbf{w}^* that minimizes the function c(\textbf{w}).

Our ideal hypothesis then becomes: \rho_{\textbf{w}^*}.

Examples of model functions — deep learning

Definition of a neuron

\begin{aligned} n_{\textbf{w}}: \;&\mathbb{R}^{d_0}\longrightarrow\mathbb{R}^{d_1} \\ \;\;x_i&\longmapsto n_\textbf{w}(x_i)=\sigma\circ g_\textbf{w}(x_i) = \sigma\left( W x_i + b\right) \end{aligned}

where

\sigma:\mathbb{R}^{d_1} \to \mathbb{R}^{d_1}: activation function (ReLU, sigmoid, sign), acting componentwise;

g_\textbf{w}(x) = Wx+b: affine transformation, with W\in\mathbb{R}^{d_1\times d_0} the weight matrix and b\in\mathbb{R}^{d_1} the bias vector;

\textbf{w}=(W,b) is an element of the parameter space \mathcal{W}.

Activation functions

The most commonly used ones are:

  • Sign \begin{aligned} s(x) = \begin{cases} 1 &\text{if } x\ge 0 \\ -1 &\text{if } x<0 \end{cases} \end{aligned}
  • ReLU: \ \operatorname{ReLU}(x)=\max\{0,x\}
  • Sigmoid: \ \sigma_{\text{sig}}(x)=\dfrac{1}{1+e^{-x}}
  • Softmax: \ \sigma_{\text{soft}}(z)_i = \dfrac{e^{z_i}}{\sum_{c=1}^{C} e^{z_c}}

A neuron inside a neural network

Let n_j^{(p-1)} be the j-th neuron of layer p-1. Then the i-th neuron of layer p is given by n_i^{(p)}=\sigma^{(p)}\left(\sum_{j=1}^{d_{p-1}}w_{ij}^{(p-1)}\,n_j^{(p-1)}+b_i^{(p-1)}\right).

Neural networks in matrix form. Let P\in\mathbb{N}, d_0,d_1,\dots,d_{P+1}\in\mathbb{N} and let \sigma^{(l)}:\mathbb{R}^{d_l}\to\mathbb{R}^{d_l} be activation functions. A function \phi_\textbf{w}:\mathbb{R}^{d_0}\to\mathbb{R}^{d_{P+1}} is called a neural network of depth P if there exist matrices W^{(l)}\in\mathbb{R}^{d_{l+1}\times d_{l}} and vectors b^{(l)}\in\mathbb{R}^{d_{l+1}}, l=0,1,\dots,P, such that

  • N^{(0)} = x   (input)
  • N^{(p)} = \sigma^{(p)}\big(W^{(p-1)} N^{(p-1)}+b^{(p-1)}\big) = \sigma^{(p)}\circ G^{(p)}\big(N^{(p-1)}\big), for 0<p<P+1
  • N^{(P+1)} = \sigma^{(P+1)}\big( W^{(P)} N^{(P)}+b^{(P)}\big)   (output)

and \phi_\textbf{w}(x)=N^{(P+1)}. Equivalently, \phi_\textbf{w}=\sigma^{(P+1)}\circ G^{(P+1)}\circ\sigma^{(P)}\circ G^{(P)}\circ\cdots\circ\sigma^{(1)}\circ G^{(1)}, \qquad \text{where } G^{(l)}(z)=W^{(l-1)}z+b^{(l-1)}.

We call P the depth and \max_{l} d_l the width; the vector \textbf{w}=\big((W^{(0)},b^{(0)}),\dots,(W^{(P)},b^{(P)})\big) collects all the parameters.

And what do these networks do?

Output of a neuron with the ReLU activation function

Output of the second layer

Composing the two neurons above with a third one, also ReLU:

The same construction with the sigmoid

Second layer with the sigmoid

Why do these networks work so well?

The Universal Approximation Theorem!

(Compact convergence) Let d\in\mathbb{N}. A sequence of functions f_n:\mathbb{R}^d\to \mathbb{R}, n\in \mathbb{N}, converges compactly to f:\mathbb{R}^d\to \mathbb{R} if, for every compact set K\subseteq \mathbb{R}^d, we have \lim_{n\to \infty} \sup _{x\in K}|f_n(x) - f(x)| =0. In this case we write f_n\overset{cc}{\to} f.

(Universal approximator) Let d\in\mathbb{N}. A set of functions \mathcal{H} from \mathbb{R}^d to \mathbb{R} is a universal approximator (of C^{0}(\mathbb{R}^d)) if, for every \varepsilon>0, every compact set K\subseteq \mathbb{R}^d and every f\in C^0(\mathbb{R}^d), there exists g\in\mathcal{H} such that \sup_{x\in K}|f(x)-g(x)|<\varepsilon.

Lemma. Let d\in \mathbb{N} and let \mathcal{H} be a set of functions from \mathbb{R}^d to \mathbb{R}. Then \mathcal{H} is a universal approximator of C^0 if and only if C^0 (\mathbb{R}^d) \subseteq \overline{\mathcal{H}}^{cc}.

(Stone–Weierstrass theorem) Let d\in \mathbb{N}, let K\subseteq\mathbb{R}^d be compact and let \mathcal{H}\subseteq C^0(K,\mathbb{R}) satisfy

  • for every x\in K there exists f\in\mathcal{H} with f(x)\ne 0,
  • for all x\ne y there exists f\in \mathcal{H} with f(x)\ne f(y),
  • \mathcal{H} is an algebra of functions, i.e. \mathcal{H} is closed under addition, multiplication and multiplication by scalars.

Then \mathcal{H} is dense in C^0(K).

(The space of neural networks) Let d,\,m,\,L,\,n\in\mathbb{N} and \sigma:\mathbb{R}\to\mathbb{R}. The set of all functions realized by neural networks with input dimension d, output dimension m, depth at most L, width at most n and activation \sigma is denoted by \mathcal{N}_d^m(\sigma;L,n) := \big\{\phi:\mathbb{R}^d\to \mathbb{R}^m \;\big|\; \phi \text{ a neural network},\ \operatorname{depth}(\phi)\le L,\ \operatorname{width}(\phi)\le n \big\}. Moreover, \mathcal{N}^m_d(\sigma; L) = \bigcup_{n\in \mathbb{N}}\mathcal{N}_d^m(\sigma;L,n).

Theorem (Universal Approximation — Leshno–Lin–Pinkus–Schocken; Petersen–Zech, Thm. 3.9) Let d\in \mathbb{N} and \sigma \in \mathcal{M}. Then \mathcal{N}_d^1(\sigma; 1) is a universal approximator of C^0(\mathbb{R}^d) if and only if \sigma is not a polynomial.

(Admissible set of activations) \mathcal{M} is the set of functions \sigma\in L^\infty_{\mathrm{loc}}(\mathbb{R}) for which there exist a finite M\in\mathbb{N} and intervals I_1,\dots,I_M partitioning \mathbb{R} with \sigma\in C^0(I_j) for every j=1,\dots,M.

Proof idea. The strategy comes down to the three reductions below:

  1. If C^0(\mathbb{R}^1)\subseteq \overline{\mathcal{N}^1_1(\sigma; 1)}^{cc} then C^0(\mathbb{R}^d)\subseteq \overline{\mathcal{N}_d^1(\sigma; 1)}^{cc}.

  2. If \sigma \in C^{\infty}(\mathbb{R}) is not a polynomial, then C^0(\mathbb{R}^1)\subseteq\overline{\mathcal{N}_1^1(\sigma; 1)}^{cc}.

  3. If \sigma \in \mathcal{M} is not a polynomial, then there exists \tilde\sigma\in C^{\infty}(\mathbb{R}) \cap \overline{\mathcal{N}_1^1(\sigma; 1)}^{cc} which is not a polynomial.

  1. If C^0(\mathbb{R}^1)\subseteq \overline{\mathcal{N}^1_1(\sigma; 1)}^{cc} then C^0(\mathbb{R}^d)\subseteq \overline{\mathcal{N}_d^1(\sigma; 1)}^{cc}.

Lemma. Assume that \mathcal{H} is a universal approximator of C^0(\mathbb{R}). Then, for every d\in\mathbb{N}, \operatorname{span} \{x\to g(\textbf{w}\cdot x) \;|\; \textbf{w}\in \mathbb{R}^d, g\in \mathcal{H}\} is a universal approximator of C^0(\mathbb{R}^d).

  1. If \sigma \in C^{\infty}(\mathbb{R}) is not a polynomial, then C^0(\mathbb{R}^1)\subseteq\overline{\mathcal{N}_1^1(\sigma; 1)}^{cc}.

Lemma. If \sigma \in C^\infty(\mathbb{R}) and \sigma is not a polynomial, then \mathcal{N}^1_1(\sigma;1) is dense in C^0(\mathbb{R}).

  1. If \sigma \in \mathcal{M} is not a polynomial, then there exists \tilde\sigma\in C^{\infty}(\mathbb{R}) \cap \overline{\mathcal{N}_1^1(\sigma; 1)}^{cc} which is not a polynomial.

Lemma. Let \sigma\in\mathcal{M}. Then, for every \varphi\in C_{c}^\infty(\mathbb{R}), we have \sigma * \varphi \in \overline{\mathcal{N}_1^1(\sigma;1)}^{cc}.

Lemma. If \sigma \in \mathcal{M} and \sigma * \varphi is a polynomial for every \varphi \in C_{c}^{\infty}(\mathbb{R}), then \sigma is a polynomial.

How do these networks learn?

Given the samples \{(x_i,y_i)\}_{i=1}^{N}, we look for \textbf{w}^* \in \operatorname*{arg\,min}_{\textbf{w} \in \mathcal{W}} \; \frac{1}{N}\sum_{i=1}^{N} \ell\big(y_i, \phi_\textbf{w}(x_i)\big) \;=\; \operatorname*{arg\,min}_{\textbf{w} \in \mathcal{W}} c(\textbf{w}), where \ell(\cdot,\cdot) measures the divergence between the prediction and the expected value.

Gradient descent

To solve \min_{\textbf{w}\in \mathcal{W}} c(\textbf{w}) we follow the direction -\nabla c, iterating \textbf{w}_{k+1}=\textbf{w}_k - \alpha_k \nabla c(\textbf{w}_k), where \alpha_k>0 is the learning rate.

A cost surface in \mathbb{R}^2

Gradient descent in action

Riemannian optimization

Let f:\mathcal{W}\to \mathbb{R}. How do we describe its minimum point?

f:\mathcal{W}\to\mathbb{R} could, for instance, model the cost of the problems we have seen so far.

Riemannian gradient descent

Given a point \textbf{w}_0\in \mathcal{W} and a retraction R on \mathcal{W}, the Riemannian gradient descent algorithm iterates \textbf{w}_{k+1}=R_{\textbf{w}_k}\big(-\alpha_k \operatorname{grad} f(\textbf{w}_k)\big), where \operatorname{grad} f(\textbf{w}_k)\in T_{\textbf{w}_k}\mathcal{W} is the Riemannian gradient.

What is a retraction?

A retraction on a manifold \mathcal{W} is a smooth map R:T\mathcal{W}\to\mathcal{W}, (\textbf{w},v)\mapsto R_\textbf{w}(v), such that, for every \textbf{w}\in\mathcal{W}:

  • R_\textbf{w}(0_\textbf{w})=\textbf{w}, where 0_\textbf{w} is the zero element of T_\textbf{w}\mathcal{W};
  • DR_\textbf{w}(0_\textbf{w})=\operatorname{id}_{T_\textbf{w}\mathcal{W}}.

In other words: R returns a point of the manifold and, to first order, behaves like the linear step we wanted to take in the tangent space. The Riemannian exponential map is one example.

Theorem (Local convergence). Let \mathcal{W} be a complete Riemannian manifold with a retraction R and let f:\mathcal{W}\to \mathbb{R} be smooth. Let w_*\in \mathcal{W} be a critical point of f, that is, \operatorname{grad} f(w_*)=0. Assume that all eigenvalues of \operatorname{Hess} f(w_*) are positive, say 0 < \lambda_1\le \cdots \le \lambda_{\max}<L. Define F:\mathcal{W}\to\mathcal{W} by F(w)=R_w\left(-\tfrac{1}{L} \operatorname{grad} f(w)\right). Then:

  • dF_{w_*}=\operatorname{Id}-\tfrac{1}{L}\operatorname{Hess} f(w_*);
  • there exists a neighborhood U of w_* such that, for every w_0\in U, the sequence w_{k+1}=F(w_k) converges to w_*.

In other words: Riemannian gradient descent converges to the local minimum w_* provided that the initial point w_0 lies in U.

Criticality of accumulation points

Assumption 1 (lower bound). f is bounded below: there exists f_{\text{low}}\in \mathbb{R} with f_{\text{low}}\le f(\textbf{w}) for every \textbf{w} \in \mathcal{W}.

Assumption 2 (regularity). Given S\subseteq T\mathcal{W}, there exists L>0 such that, for every v\in S_\textbf{w}, f\big(R_{\textbf{w}}(v)\big)\le f(\textbf{w})+\big\langle \operatorname{grad} f(\textbf{w}),\, v\big\rangle+\frac{L}{2}\|v\|^2.

This is the Riemannian version of the L-Lipschitz gradient condition: it guarantees that the linear model on T_\textbf{w}\mathcal{W} does not underestimate the function too much along the retraction.

Armijo–Goldstein condition f(\textbf{w})-f\big(R_\textbf{w}(-\alpha \operatorname{grad} f(\textbf{w}))\big) \;\ge\; r\,\alpha \,\|\operatorname{grad} f(\textbf{w})\|^2 for some constant r\in (0,1).

Backtracking line-search algorithm

Pseudocode of the backtracking line-search algorithm

Lemma. Let f be smooth on the Riemannian manifold \mathcal{W}. Fix \textbf{w}\in \mathcal{W}, a retraction R_\textbf{w}, an initial step \bar{\alpha}>0 and constants \tau, r \in (0,1). Assume that Assumption 2 holds for f\circ R_\textbf{w}(s) with constant L, where s=-\alpha\operatorname{grad} f(\textbf{w}), \alpha \in [0, \bar{\alpha}]. Then:

  • there is a first N>0 such that \alpha_N=\tau^{N-1}\bar{\alpha} satisfies the Armijo–Goldstein condition;
  • this \alpha_N is bounded below by \min\left(\bar{\alpha},\, \dfrac{2\tau(1-r)}{L}\right).

Proposition (criticality of accumulation points). Let f be smooth and satisfy Assumption 1 on a Riemannian manifold \mathcal{W}. Fix \tau, r\in(0,1) and S\subseteq T\mathcal{W} on which Assumption 2 holds with the retraction R. Assume:

  • \{\textbf{w}_k\} is the sequence generated by Riemannian gradient descent;
  • the step sizes \alpha_k are obtained by backtracking line-search;
  • the initial steps satisfy \displaystyle\liminf_{k\to \infty} \bar{\alpha}_k\ge \bar{\alpha}>0;
  • -t\operatorname{grad} f(\textbf{w}_k)\in S for every t\in(0,\bar{\alpha}_k).

Then \displaystyle\lim_{k\to\infty}\|\operatorname{grad} f(\textbf{w}_k)\|=0.

Proof idea. Since the learning rate satisfies the Armijo–Goldstein condition, f(\textbf{w}_k) - f\big(R_{\textbf{w}_k}(-\alpha_k \operatorname{grad} f(\textbf{w}_k))\big) \;\ge\; r\,\alpha_k\, \|\operatorname{grad} f(\textbf{w}_k)\|^2.

But the steps are bounded below: \alpha_k\ \ge\ \min\left(\bar{\alpha},\, \frac{2\tau(1-r)}{L}\right)=:d\ >\ 0 .

Hence, with c:=r\,d>0, f(\textbf{w}_k) - f(\textbf{w}_{k+1}) \;\ge\; c\,\|\operatorname{grad} f(\textbf{w}_k)\|^2 , that is, \{\textbf{w}_k\} is sufficiently decreasing.

Sufficiently decreasing. At every iteration the algorithm achieves sufficient decrease: there exists c>0 such that, for every k, f(\textbf{w}_k)-f(\textbf{w}_{k+1})\ \ge\ c\,\|\operatorname{grad} f(\textbf{w}_k)\|^2.

Lemma. Let f be smooth and satisfy Assumption 1 on a Riemannian manifold \mathcal{W}, and let w_0, w_1, w_2,\dots be the sequence of iterates of a sufficiently decreasing algorithm with constant c. Then \lim_{k\to \infty}\|\operatorname{grad} f(w_k)\| =0.

In other words: every accumulation point is critical.

Second-order optimization

Algorithms that use the second derivative

A second-order model function would be m_\textbf{w}(s)= f(\textbf{w})+\big\langle\operatorname{grad} f(\textbf{w}),\,s\big\rangle +\tfrac{1}{2}\big\langle[\operatorname{Hess} f(\textbf{w})]s,\,s\big\rangle .

Newton’s algorithm:

Pseudocode of the Riemannian Newton method

Newton’s method has two drawbacks:

  • it requires solving a linear system at cost O(n^3) at every iteration;
  • if the step lands too far from the expansion point, the Taylor approximation is no longer valid.

Conjugate gradient algorithm:

Pseudocode of the conjugate gradient method

Trust-region algorithm Pseudocode of the trust-region method

Results obtained by applying Riemannian optimization to a machine learning problem

Accuracy across epochs

Final error

Loss function during training

References

  • DRECHSLER, Monica. Uma introdução ao aprendizado profundo e à otimização Riemanniana. São Paulo: Instituto de Matemática, Estatística e Ciência da Computação, Universidade de São Paulo, 2025. MSc dissertation in Mathematics.
  • BOUMAL, Nicolas. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
  • SATO, Hiroyuki. Riemannian optimization and its applications. Springer, 2021.
  • PETERSEN, Philipp; ZECH, Jakob. Mathematical theory of deep learning. arXiv:2407.18384, v4 (Jan. 2024).
  • YE, Jong Chul. Geometry of Deep Learning. Springer Singapore, 2022.
  • ABU-MOSTAFA, Yaser S.; MAGDON-ISMAIL, Malik; LIN, Hsuan-Tien. Learning from data. New York: AMLBook, 2012.
  • CALIN, Ovidiu. Deep learning architectures. New York City: Springer International Publishing, 2020.
  • WATT, Jeremy; BORHANI, Reza; KATSAGGELOS, Aggelos K. Machine learning refined: Foundations, algorithms, and applications. Cambridge University Press, 2020.
  • TOWNSEND, James; KOEP, Niklas; WEICHWALD, Sebastian. Pymanopt: a python toolbox for optimization on manifolds using automatic differentiation. Journal of Machine Learning Research 17.137 (2016), pp. 1–5. URL: http://jmlr.org/papers/v17/16-177.html.

Thank you for your attention

Contact: monicadrechsler@gmail.com