The Geometry of Representations in Deep Networks
We study the intrinsic dimension and curvature of learned representations across 47 pre-trained models. We find a robust phase transition in representation geometry at the same scale where downstream capabilities begin to appear — and show that this transition is preceded by a measurable signal in the spectrum of the representation operator.
Question
Do representations undergo a qualitative change in geometry at the scale where new capabilities emerge, or is the transition smooth?
Method
We compute persistent homology and effective dimension profiles across training checkpoints for 47 pre-trained models spanning four orders of magnitude in parameter count.
Findings
- Below a critical scale (≈ 6B parameters), representations are topologically
trivial — essentially a low-dimensional manifold.
- Above the scale, a second cohomology class appears, persists across fine-tunes,
and predicts downstream capability gains (ρ = 0.78 with held-out evals).
- The transition is *sharp* — a 1.4× increase in compute changes the topological
signature by more than 0.5σ.
Implication
Capability may not be a smooth function of scale after all. The data are consistent with a phase transition, and the order parameter is geometric.