Between September 14 and 16, 2026, we trained small language models from scratch on one consumer GPU to screen architecture ideas. Retraining the same configuration with the same seed sometimes changed the validation score by up to 0.0018 bits per byte. Changing the seed moved it by up to 0.008. Most of the mechanism effects we were testing were smaller than either number, and the two that looked best on the selection seed did not hold up on fresh seeds.
The setup
All comparison runs used decoder-only Transformers between 26 and 135 million parameters, trained on TinyStories, a small synthetic dataset of children's stories. Each run consumed exactly 512 optimizer updates and 8,388,608 training tokens, with BF16 mixed precision, context length 512 and full causal attention. Hardware and software were fixed: one NVIDIA RTX 5070 Ti (16 GB) on Windows, Python 3.11.15, PyTorch 2.9.1 built for CUDA 12.8.
The score is validation bits per byte (BPB) on a fixed 65,536-token validation set. BPB measures how many bits the model needs, on average, to encode each byte of held-out text. Lower is better, and because it is normalized by bytes rather than tokens it can compare models with different tokenizers. For a sense of scale, the smallest dense model scored about 0.635 and the largest about 0.590.
Three sources of variation
We had repeated runs at three levels, which made it possible to separate where the variation came from.
| What was repeated | Runs | Spread |
|---|---|---|
| Re-scoring saved checkpoints | 83 | < 0.000001 |
| Same seed, dense depth 6 | 8 | 0.000000 |
| Same seed, n-gram variant, depth 6 | 9 | 0.000000 |
| Same seed, dense depth 8 | 8 | 0.000000 |
| Same seed, dense depth 10 | 4 | 0.001777 |
| Same seed, dense depth 12 | 9 | 0.001288 |
| Seeds 42 to 46, dense depth 6 | 5 | 0.008227 |
| Seeds 42 to 44, dense depth 12 (repeat means) | 3 | 0.003997 |
Evaluation was not the problem. The 83 checkpoints, all from one campaign, reproduced their original scores with the captured code to within 0.000000485. The variation came from training.
At the smaller sizes, same-seed retraining was stable to the six recorded decimals. The depth 6 dense model with seed 43 even scored exactly 0.637774 in two separate campaigns. At depth 10 and 12 it was not stable, although initialization hashes and captured source hashes were identical across the repeats. We do not know why the larger configurations drifted and the smaller ones did not.
Same seed is not the same run
Two further records point the same way. In a resume test on a separate, much longer training run with different data, the restored model, optimizer and random-number state were bit-identical to the saved state. The next four training steps still differed from the uninterrupted run, with a maximum loss difference of 0.000025. Independent uninterrupted repeats also differed from each other.
The second case is a pair of seed-42 mixture-of-experts (MoE) controls trained in two different campaigns. All 88 initialization tensors matched, as did the batch order, the learning-rate schedule and the data. Their printed training losses first differed at update 17 (3.994706 versus 3.994785). By the end all 88 weight tensors differed, and the final scores were 0.638883 and 0.637217, a gap of 0.001666. One qualification applies: between the two runs, the training code was refactored to support an optional memory layer. We found no intended change to the MoE computation itself, but this pair is not a clean same-code rerun.
A plausible explanation is that GPU floating-point reductions do not always sum in the same order, and small rounding differences compound over hundreds of updates. That is not proven here. We did not record whether PyTorch's deterministic-algorithm mode was on, which kernels were selected, or the background GPU load. A short replay with deterministic algorithms enforced and per-step weight hashes was proposed to locate the divergence. It has not been run.
What the noise did to our comparisons
Set against that background, here is how the tested changes fared. Seed 42 was used to select candidates; fresh seeds played no part in the choice. Negative differences favor the change.
| Change tested | Seed 42 | Fresh-seed mean | Fresh seeds favoring |
|---|---|---|---|
| Hashed n-gram memory, dense depth 6 | −0.000670 | not run | not run |
| MoE instead of dense, equal tokens | +0.004035 | −0.003030 | 1 of 1 |
| Learning rate 0.03 instead of 0.04, depth 12 | −0.001581 | −0.000909 | 1 of 2 |
| Next Concept Prediction module, depth 6 | −0.002971 | −0.002320 | 2 of 4 |
| Width 384 to 768, depth 6 | not used | −0.032640 | 2 of 2 |
| Width 384 to 768, depth 12 | not used | −0.030223 | 2 of 2 |
| Depth 6 to 12, width 384 | not used | −0.016547 | 2 of 2 |
| Depth 6 to 12, width 768 | not used | −0.014130 | 2 of 2 |
The n-gram difference is smaller than the same-seed drift we saw at depth 10 and 12 and far smaller than the seed spread at depth 6. It came from a single seed, so it tells us little either way. The MoE comparison reversed sign between its two seeds: +0.004035 on seed 42 and −0.003030 on seed 43.
The learning-rate and Next Concept Prediction (NCP) results show the selection trap directly. Both looked better on seed 42. On fresh seeds, the learning-rate change helped on one (−0.002857) and hurt on the other (+0.001038). NCP, a module that predicts compressed codes for upcoming chunks of text, favored the candidate on two of four fresh seeds and failed its predeclared confirmation rule. It also adds 13.88% more parameters, so we trained a plain residual MLP with exactly the same parameter count. That control averaged 0.636494 against 0.637223 for NCP, while training faster.
Width and depth were the only changes whose effects were several times larger than the seed spread and consistent across seeds. They also cost more: doubling depth roughly doubled the time per update. These are equal-token comparisons, not equal-compute ones.
What we would do differently
The campaigns already froze candidates before confirmation and reserved fresh seeds for it, which is why the learning-rate and NCP results were caught. The records point to steps that belonged earlier in the process:
- Measure noise first. Repeat the baseline with the same seed and with several seeds before screening any mechanism, and treat any difference smaller than that spread as unresolved.
- Keep the selection seed out of the evidence. A candidate chosen on seed 42 has to be confirmed on seeds it was never scored on.
- Count same-seed repeats as a measure of execution variability, not as extra samples. Nine runs on one seed are not nine independent observations.
- Add a capacity-matched control whenever a mechanism adds parameters, so that the comparison tests the mechanism rather than the extra size.
- Record determinism settings and kernel choices with every run. Without them, the cause of the drift above cannot be recovered after the fact.
Limits
These are small models, short training runs and one easy dataset on one machine and software stack. The spreads above describe this setup. They are not a general estimate of training variance, and larger models or longer training may behave differently. Repeat counts are small, the scores are validation results rather than a final held-out test, and the cause of the same-seed drift is unexplained. The NCP module was inspired by published work on concept prediction but is not a faithful reproduction of it, so the negative result applies to this implementation only.
Publication disclosure
Implementation, experiments and writing were AI-assisted. All figures come from saved result records of training campaigns run September 14 to 16, 2026. No training or evaluation was rerun for this note. Checkpoints, logs and the experiment repository are private and are not published, so readers cannot reproduce these runs from this note alone.