Eigenvalues, SVD, and High-Dimensional Vector Spaces in Machine Learning

Singular Value Decomposition (SVD) forms the mathematical backbone of modern representation learning, latent semantic analysis, and vector retrieval engines.

Spectral Theorem & Low-Rank Approximation

For any arbitrary matrix $A \in \mathbb{R}^{m \times n}$, there exists an orthogonal factorization:

Where $U \in \mathbb{R}^{m \times m}$ and $V \in \mathbb{R}^{n \times n}$ are orthonormal matrices containing the left and right singular vectors, and $\Sigma$ is diagonal with singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0$.

Eckart-Young-Mirsky Theorem

The optimal rank-$k$ approximation $A_k$ minimizing the Frobenius norm error:

import numpy as np

def truncated_svd_embedding(X: np.ndarray, k: int) -> np.ndarray:
    U, S, Vt = np.linalg.svd(X, full_matrices=False)
    # Project high-dimensional token co-occurrence matrix into k-dimensional space
    return U[:, :k] * np.sqrt(S[:k])

Dimensionality Reduction Dynamics

In $d$-dimensional Euclidean space, the volume of a hypersphere of radius $r$ scales according to:

As $d \to \infty$, virtually all mass in a Gaussian distribution concentrates within a thin spherical shell at radius $r \approx \sqrt{d}$, creating the curse of dimensionality in vector search.