Cracking GPT-6 — A Deep Dive into Looped Transformers

Clickbait title, obviously. Based on some technical speculation that GPT-6 uses a Looped Transformer as its architecture, I spent a long weekend learning about it and running some experiments. This post is a technical write-up about Looped Transformers only. It is not evidence about GPT-6’s actual architecture. Any resemblance is coincidental — Sam, call me.

TL;DR

Everything below was measured on Nanbeige4.2-3B-Base, an openly released Looped Transformer with 22 shared decoder layers and a pretrained loop count of 2.

1. Depth does not extrapolate. Accuracy peaks at exactly the pretrained depth and falls to chance beyond it.
2. The failure mode is representational collapse. The effective rank of the hidden state drops from 131.6 → 14.2 and distinct token positions converge to 0.944 cosine similarity. It is a fixed point, just an uninformative one.
3. The native loop is not reasoning. cos(h¹, h²) = −0.04 — the two loop states are nearly orthogonal, and loop 1’s output distribution is essentially independent of the question.
4. Loop 1 is writable, but not by ordinary supervision. Chain-of-thought SFT leaves the loop-1 probe unchanged, while contrastive representation supervision increases it. Whether a more informative loop 1 buys downstream reasoning remains open.

1. What is a Looped Transformer?

A standard Transformer buys depth by stacking blocks with different parameters. A Looped Transformer instead calls the same block repeatedly:

h⁽ʳ⁺¹⁾ = F_θ(h⁽ʳ⁾)

Parameter sharing creates additional computational depth without new parameters, raising a natural question: does recurrent depth behave like iterative reasoning?

The architecture, precisely

One implementation detail matters throughout. Nanbeige’s recurrence is

h⁽⁰⁾ = E(x)                          (raw embeddings, un-normalised)
h⁽ʳ⁺¹⁾ = RMSNorm(F_θ(h⁽ʳ⁾))

All 22 layers are shared and executed in order on every loop. There is no input injection — E(x) is never re-introduced after the first loop. The final RMSNorm sits inside the loop, which means every loop’s state is already in the LM head’s input format. That makes lm_head(h⁽ʳ⁾) a well-defined early-exit readout, and it is what makes the rest of this analysis possible.

2. The native loop is not iterative reasoning

2.1 Depth does not extrapolate

The cleanest test: run the model at loop counts 1 through 6 and measure task accuracy. Loop count is read per-forward, so this needs no code change — just set config.num_loops.

Performance peaks exactly at the trained depth and collapses immediately beyond it. Loop 3 has already fallen most of the way back to chance; loops 4–6 are chance. The failure is not a gentle decay — which raises the question of mechanism.

2.2 Extra depth collapses the representation

Three measurements on the hidden state, per loop:

• Effective rank: 131.6 at loop 2 → 14.2 at loop 6, an order of magnitude.
• Mean cosine between distinct token positions: 0.750 → 0.944. Different tokens stop having different states.

The recurrence converges toward a low-rank, position-uniform attractor. A readout from such a state cannot be informative; that is the mechanism behind the accuracy collapse. (Intervention experiments and interpretation: Appendix B.)

2.3 Loop 1 is not a draft answer

If the loop were iterative refinement, loop 1 would hold a rough draft that loop 2 sharpens. It does not: loop 1 barely depends on the question. Taking the renormalised option distribution on MMLU-Pro and measuring the L1 distance between different questions (range 0–2):

A 13.5× difference — loop 1’s answer distribution is close to a constant that ignores the input. Consistently, the two loop states are nearly orthogonal: cos(h¹, h²) = −0.04.

The native model is better described as a weight-tied two-stage computation than as iterative refinement. Loop 1 is an intermediate encoding stage whose output the LM head was never trained to read; loop 2 is where the answer is formed.

3. Can Loop 1 be made informative?

3.1 Experimental setup

TEACHER (frozen):  Problem:\n{q}\n\nSolution:\n{reasoning, answer masked}\n<REASON_SUMMARY>
STUDENT (trained): Problem:\n{q}\n\nSolution:\n<REASON_SUMMARY>\n\nAnswer:\n{a}

A frozen teacher sees the question plus a masked chain-of-thought; the student sees the question only. We supervise the student’s loop-1 state toward the teacher representation while keeping ordinary answer supervision at loop 2. A frozen linear probe then measures how much answer-relevant information is decodable from loop 1.

Experiments use 20,000 NuminaMath-CoT examples with integer answers below 100; full dataset construction, splits and training configuration are in Appendices C and A.

3.2 Loop 1 can hold information

The teacher target has to be checked first, because chains-of-thought leak: 87.1% of naively truncated CoTs still contain the answer verbatim, and masking the answer drops the teacher probe from 0.404 to 0.229. Nearly half of the apparent “CoT information” was string leakage. All subsequent experiments therefore use the masked target (full comparison: Appendix D).

The masked chain still carries real information, and loop 1 can receive it: with the masked CoT actually present in the input, the loop-1 probe rises from 0.118 to 0.168. Loop 1 can hold task-relevant information; native pretraining simply leaves it empty. That is what makes training it worth attempting.

3.3 Geometric alignment fails; contrastive supervision works

Cosine alignment substantially increases student–teacher similarity, but leaves the loop-1 probe unchanged relative to answer-only SFT — across every variant we tried (Appendix D). Geometric alignment is not information transfer.

We therefore switched from matching distance to optimising discrimination. InfoNCE moves the loop-1 probe from 0.118 to 0.143 — the only representation-level objective we tested that measurably increases task-relevant information in loop 1.

4. The control that matters

Answer-only SFT and CoT SFT produce essentially identical loop-1 probes (0.132 vs 0.133) despite a 176× difference in supervised tokens. Ordinary token-level CoT supervision does not populate loop 1 — if you want structure in a specific computational slot, you have to supervise that slot directly.

InfoNCE reaches 0.143, but MMLU-Pro remains essentially identical to the answer-SFT control (0.522 vs 0.520). The representation is more informative, but no downstream benefit is demonstrated. (The shared benchmark jump from 0.422 to ~0.52 across all arms deserves its own caution — see the note in Appendix A.)

5. Conclusion

On this model, recurrent depth is not iterative reasoning: performance collapses beyond the trained depth, and loop 1 behaves like an intermediate encoding rather than a draft answer.

That intermediate state is nevertheless writable — and Looped Transformers are interesting precisely because they expose it as an explicit, separately supervisable slot. Ordinary CoT SFT does not meaningfully populate it, while contrastive representation supervision does.

But the resulting representation has not meaningfully beaten plain SFT on the probe, and it has not improved downstream performance. Loop 1 is writable, and we do not yet know what it buys.

Appendix

A. Model and evaluation

• Model: Nanbeige/Nanbeige4.2-3B-Base, 22 shared layers, num_loops=2, hidden 3072, vocab 166,144, bf16. 4.17B parameters.
• Loop count at inference: read per-forward, so config.num_loops = R suffices for R = 1…6.
• Probe: 100-way integer-answer classification, frozen linear (LayerNorm-free), weight-decay grid selected on a held-out validation split, identical splits and recipe across all variants. n_test = 3,000.
• MMLU-Pro: 500 items sampled stratified by category with a fixed seed, 5-shot, answer-letter log-likelihood with renormalisation over the option set, use_cache=False.
• Training: LoRA r=32, α=64, all attention and MLP projections; 1 epoch over the 14,000 training examples, 437 steps, lr 2e-5 cosine with 3% warmup, effective batch 32, seed 42. Arms differ in a 3-line config diff and are otherwise byte-identical, verified by structural diff and by an end-to-end λ=0 identity check.
• Distillation marker: the tokenizer has 166,100 pieces against 166,144 embedding rows, so the <REASON_SUMMARY> marker occupies an unused row. No resize; the existing vocabulary’s logits are bitwise unchanged.
• Compute: a single A800-80GB per run; ~28 minutes per arm (~1.9 h for the CoT-SFT arm, whose sequences are ~4× longer). Peak 16.1 GiB.

A note on the shared MMLU-Pro gain. Every fine-tuned arm improves from 0.422 to 0.520–0.530, and the gain is almost invariant to whether training used 2.5 answer tokens per example, 312 CoT tokens, or representation supervision on top. Training is pure mathematics with integer answers below 100, while MMLU-Pro spans 14 subjects including law, psychology and business. The shared improvement may therefore largely reflect answer-format adaptation — learning to emit a definite choice after Answer: instead of continuing to write text — rather than transferred reasoning ability. If a method gains ten points over a base model, the control worth having is not the base model but a minimally supervised model that has already learned the answer format.

B. Depth diagnostics

Teacher-forced cross-entropy by depth. Against a uniform-distribution CE of 12.02:

Loop 4 is essentially a uniform distribution over a 166k vocabulary.

Convergence. Adjacent-loop cosine similarity rises from 0.078 to 0.814 across loops — the iteration genuinely converges to a fixed point. State RMS stays in the 2–3 range throughout.

Two inference-time interventions, both refuted. Applied via forward pre-hooks, no training:

At α = 2.0 the injected vector has RMS ≈ 1.0 against a state RMS ≈ 3.2 — a 30% perturbation — and it buys 0.04 nats. The collapse is neither a scale-drift problem nor a retrofittable missing-input-anchor problem. It is consistent with the shared block being effectively contractive on states beyond its trained recurrent depth: during training it only ever consumed raw embeddings (loop 1) and h⁽¹⁾ (loop 2), so the h⁽²⁾ it receives at loop 3 lies outside its training distribution.

Loop-1 readout profile. Reading loop 1 through the shared LM head gives 48.4% top-1 accuracy but CE 8.53 (uniform = 12.02): informative in ranking, badly calibrated in mass — a very flat distribution that often ranks the right token first while assigning it almost nothing.

A diagnostic trap. A natural diagnostic for “does looping help?” is the rate at which the correct-answer log-probability decreases from loop 1 to loop 2. On this model that rate is 34%, which looks like a dramatic amount of harmful recurrent updating. It is an artefact: when loop 1 is a near-random predictor, roughly half its log-probabilities will fall by chance. The genuinely destructive transition is loop 2 → 3, at 67.8%. Check whether your baseline loop is informative before interpreting transition rates.

Model-card framing. Nanbeige describes the looping as increasing model capacity without increasing parameters — a claim about capacity, not reasoning depth. The two-stage reading in §2.3 is consistent with the vendor’s own framing.

C. Dataset construction

Source: AI-MO/NuminaMath-CoT, shard 0 (171,899 rows). Every example carries a problem, a worked solution serving as the chain-of-thought, and a final answer in \boxed{}.

The probe needs a fixed label space: a single summary vector cannot do per-example-randomised multiple choice, because the probe never sees the options. The pool is therefore restricted to examples whose boxed answer is an integer below 100, turning the probe into a well-posed 100-way classification. 39,973 examples qualify; 20,000 are kept (a further 640 dropped for exceeding 1,024 tokens), split into three disjoint parts used consistently everywhere:

Source composition of the 20,000:

Grade-school-to-competition mathematics, all of it, with integer answers 0–99. The label distribution is skewed — the answer “2” alone accounts for 7.07% of examples — which is why the majority-class baseline is 0.0787 rather than 0.01, and why balanced accuracy was tracked alongside raw accuracy throughout. Reasoning chains average 377 tokens.

D. Representation experiments in full

What actually carries the answer.

Cosine alignment moves the geometry… With a projection head, alignment cosine −0.0005 → +0.4489; without a head, against a same-space loop-1 target, +0.2795 → +0.4019.

…and not the information. Loop-1 probe accuracy across cosine arms:

The projection head was removed after discovering it actively destroys information: probing P_φ(h¹) scores lower than probing raw h¹ (0.1227 vs 0.1323). Switching to a same-space loop-1 teacher removed the cross-loop confound. Neither helped. A likely explanation: cosine distance treats all 3,072 dimensions symmetrically, so global geometric alignment need not preserve the few directions that encode the task signal. The CoT SFT arm’s alignment cosine to the CoT representation is −0.0027, indistinguishable from zero.

InfoNCE. The objective replaces the distance constraint with a ranking constraint: the student’s loop-1 state must identify its own teacher target among candidates, and is free to sit far from it as long as it is not confusable with anybody else’s. With a micro-batch of 4, in-batch negatives give you 3; instead, 4,096 negatives are sampled from the cached teacher matrix each step (essentially free, since the cache is already resident), using a dedicated RNG so the data order is unchanged relative to the arms without it. The untrained loop-1 state already retrieves its own teacher target out of 4,096 candidates 31.2% of the time (chance: 0.024%) — the discriminative signal was always there. Training raises it to 56.2%.

Full arm comparison.