A neuron is not just a number. Decompose every weight as w = r·u — a magnitude r and an angle u (a direction on the sphere). Angles turn out to be their own axis of compute, alongside depth and width: they compose by addition (R(θ₁)R(θ₂) = R(θ₁+θ₂), so the composition angle is computed by the architecture, not learned), they form a shape when orchestrated together (the weight polytope and its angle field, which determine the configuration up to scale and rigid motion), and that shape is readable — on MNIST, a linear probe on the directions only of hidden weights predicts each unit's selective class at 0.67 accuracy (chance 0.10). And once the angles settle, the network's geometry is decided — the rest is scale. On a compositional rotation task, the angle net's generalization gap on unseen sums of angles is 52× smaller than a matched standard MLP at one-tenth the parameters (5 seeds). Orientation selectivity, circular codes, attractors: this is how brains do geometry, and it is a reparameterization of an ordinary layer — nothing about backpropagation changes.
The one idea
Standard networks give every connection a scalar weight: a magnitude and nothing else. This work makes the orientation of each weight an explicit, named, learnable object.
wᵢ = rᵢ · uᵢ, rᵢ > 0, uᵢ ∈ Sⁿ⁻¹
r is how strongly a connection fires. u is which way it points — the angle of the weight. Both are trained with ordinary backpropagation. When all the weight vectors of a layer are orchestrated together, they are points in weight space; the angles between them form an angle field Θ, and their convex hull is the layer's weight polytope — its shape. The shape determines the weight configuration up to scale and rigid motion, and after training it is readable.
a magnitude and a direction — the angle of the weight, trained like any parameter.
the angle field settles an order of magnitude while test accuracy keeps climbing — geometry decided early, scale later.
individual angles in the rotation net converge and lock onto their targets during training.
Play with the angles
Two interactive demos, straight from the paper. No model files, no server — the geometry itself.
Angles are an axis of compute
Depth composes functions; width increases parallelism. The angular channel is orthogonal to both:
- Capacity. An angle-only layer of width m in dimension n has m(n−1) orientational degrees of freedom whose configurations are exponentially separated — the directions of a layer are a spherical resource.
- Composition. Planar angle layers compose by addition: R(θ₁)R(θ₂) = R(θ₁+θ₂). A cascade of angle layers is an angle accumulator — rotational structure behaves like extra depth.
- Systematic generalization. Because the composition angle is computed by the architecture, a network that encodes each input angle faithfully composes correctly on unseen sums of angles — no interpolation required.
held-out sums: the angle net's gap is 52× smaller than a matched MLP, at one-tenth the parameters (mean ± std over 5 seeds).
The angles settle
Watch the angle field during training: its per-epoch velocity decays an order of magnitude while accuracy keeps climbing. Freezing experiments draw a clean line between the two channels:
- Freeze the angles → catastrophic. New data cannot be represented; the orientation scaffold is the representation (transfer 0–4 → 5–9: 0.0% vs 64.8% full fine-tune).
- Freeze the magnitudes → cheap. After settling, the angle channel alone (99.2% of parameters) keeps learning — accuracy still climbs (settle@8: 85.5%).
- Freeze too early → caps performance. Settling is a precondition for compression, not a license to compress.
Angles carry structure. Magnitudes carry scale.
Orchestrated angles form a shape — and it's readable
Take the trained angle net on MNIST and look at one hidden layer's shape: 64 weight vectors in ℝ¹⁹⁶, projected to their top-2 plane.
- Angle-field probe: a linear probe on the directions only of hidden units predicts each unit's selective class at 0.67 accuracy (chance 0.10). Read the angle, know what the neuron is for.
- Class sectors: classes occupy distinct regions of the weight space, with per-class convex hulls and angular concentrations that differ by up to 6×.
- Angle field matrix: Θ reordered by class shows near-block structure — units serving the same class point in related directions.
units by class, the angle field, per-class shape signatures — interpretability is the direct output of the parameterization.
Key numbers
| experiment | measure | angle-weighted | standard | takeaway |
|---|---|---|---|---|
| rotation composition held-out sums of angles | MSE on unseen sums | 2.5e-3 | 8.7e-2 | 35× better; compositional gap 52× smaller, at ~0.1× the parameters (5 seeds) |
| MNIST 196→64→64→10 | test accuracy | 90.0% ± 0.6 (20 ep) | 96.2% ± 0.1 (10 ep) | competitive, transparent, and the gap closes with training (3 seeds) |
| settling | angle-field velocity | 0.63 → 0.06 (~10× decay) | — | the shape is decided early; accuracy keeps climbing |
| transfer (digits 0-4 → 5-9) | accuracy, angles frozen vs full | 0.0% | 64.8% | angles = structure, magnitudes = scale |
| shape probe | class from directions only | 0.67 CV (chance 0.10) | — | the angle field of a trained layer is readable |
| CIFAR-10 (flattened 3072→512→256→10) | test accuracy | 42.1% ± 1.3 (30 ep) | 44.5% ± 0.8 (30 ep) | angle velocity 0.71→0.08; gap closes with scale (3 seeds) |
standard vs angle-weighted vs angles-only vs settle@8 (magnitudes frozen at epoch 8).
Closer to the brain
The three claims of this work are individually ancient in neuroscience. This work's contribution is to bring them together inside one parameterization:
- Orientation selectivity. Primary visual cortex is organized into columns of neurons tuned to the angle of a stimulus (Hubel & Wiesel, 1962). A weight direction uᵢ is an orientation column in miniature.
- Circular codes. Head-direction cells encode heading as a circle of angles (Taube, 1990); ring attractors store a continuous angular variable as a stable manifold (Seung, 1996). Planar angle layers are exactly this: an angle that composes and settles.
- Angles as the representation. Grid cells tile space with hexagonal lattices — a settled, regular polytope of 60° orientations (Hafting, 2005). The weight polytope is the deep-learning analogue.
- Population geometry. Computation is increasingly read from the geometry of neural population activity (Chung, Lee & Sompolinsky, 2018). Reading a layer's shape is the same habit of mind, applied to the weights themselves.
Neurons do not merely have weights — they point in directions. The collection of their directions is a shape, and the shape is the geometry of what the network learned.
Reproduce
git clone https://github.com/sehajr-singhs/angle-nets cd angle-nets pip install -r requirements.txt # rotation composition — angles add, and generalize (5 seeds, resumable) cd experiments python rotation_composition.py # single seed, full diagnostics python multiseed.py --part rot --seeds 5 # mean ± std over 5 seeds # MNIST benchmark — standard vs angle-weighted (3 seeds) python mnist_benchmark.py python multiseed.py --part mnist --seeds 3 # settling dynamics — the angle field locks in while accuracy climbs python settling_measure.py # shape and interpretability — probe, sectors, angle field python shapes.py # CIFAR-10 benchmark — scales to natural images (3 seeds, resumable) python cifar10.py python cifar10.py --seeds 0 1 2 # regenerate all figures (paper/figures and figs/) python make_figures.py
Simulation-only, CPU-scale, deterministic seeds. Multi-seed runs are resumable: per-seed results are saved after each seed and completed seeds are skipped on restart. Every number on this page traces to a committed JSON in experiments/results/.