Our toy latent predictor stopped producing almost constant features after one correction. But its eight-dimensional representation still occupied roughly one effective dimension. Restoring feature spread had not restored dimensional diversity.
That distinction was the useful result of a small follow-up experiment. The model was a JEPA-inspired predictor of numeric outcomes from authored arithmetic tasks, not a general coding model or a reproduction of LeWM's visual-control system.
What we had left out
The earlier adaptation's features had nearly collapsed. Reviewing the pinned LeWM implementation exposed two differences worth testing.
First, its anti-collapse statistic scales by batch size and uses twice the positive-half trapezoid integral over [0, 3]. Our earlier adaptation used an unscaled, one-sided integral over [0.2, 4]. Reusing a regularization coefficient of 0.1 therefore did not preserve the same regularization strength or calculation.
Second, the upstream configuration includes Linear, BatchNorm, GELU and Linear projectors for both encoder and predictor. Our adaptation omitted that projector and normalization package. We kept the original failed experiment unchanged and tested corrections separately.
Four variants, three seeds
The development experiment froze four variants, seeds 17, 29 and 43, batch size 32 and 600 CPU updates per run. It used 256 training transitions and 128 development transitions from four authored arithmetic families. Development examples changed numeric parameters within those families. They were not independent real-world programs or final evaluation tasks.
| Variant | Seeds passing all checks | Mean feature standard deviation | Effective rank, out of 8 |
|---|---|---|---|
| Original control | 0/3 | 0.0054 to 0.0063 | 1.65 to 2.88 |
| Statistic correction only | 0/3 | 1.09 to 1.25 | 1.01 to 1.02 |
| Projector package only | 0/3 | 0.28 to 0.39 | 1.02 to 1.05 |
| Both corrections | 3/3 | 1.31 to 1.35 | 3.01 to 3.40 |
Standard deviation measures how much features vary. Effective rank summarizes how broadly variation is distributed across dimensions. Several coordinates can vary together along almost the same direction, so a substantial standard deviation can coexist with rank near one. Conversely, the original control's higher rank in some seeds did not make its almost constant features healthy.
The statistic-only correction is the clearest example: it removed near-zero spread but failed the rank requirement in every seed. Only the combined correction passed every frozen development check across all three seeds.
What counted as passing
The checks required mean feature standard deviation between 0.25 and 3, minimum feature standard deviation at least 0.05, effective rank at least 2, and an encoder regularizer no greater than 0.8 times its collapsed-reference value. Prediction errors had to be at most 0.95 times both constant-predictor and shuffled-target controls. Single-item versus batched evaluation differences had to stay below 1e-5.
For the combined variant, prediction mean-squared error divided by constant-predictor error was 0.084, 0.282 and 0.205 for the three seeds. The corresponding shuffled-target ratios were 0.049, 0.164 and 0.124. These within-variant comparisons check whether prediction does better than simple controls. Raw latent-space error is not a fair standalone comparison across differently scaled representations.
These thresholds were engineering checks fixed before the diagnostic, not universal definitions of useful representations. Effective rank around three out of eight does not establish full isotropy or broad feature coverage.
The practical lesson
When a latent model appears to recover from collapse, inspect more than one statistic. Check feature scale, dimensional diversity and prediction against simple controls together. A low prediction loss can be uninformative when nearly every target is the same.
Also audit the implementation before concluding that an architecture failed. A copied coefficient is not an equivalent objective if reduction, batch scaling or integration changes underneath it. In this case, the projector package mattered together with the corrected statistic, but we cannot attribute the result to BatchNorm alone: adding the package also increased parameter count from 29,304 to 35,888.
What remains unanswered
This establishes a bounded stability repair on synthetic development fixtures. It does not show that JEPA is better than a token language model, that the representation understands code, or that the repair will transfer to Vesper-400M. The fresh reserved evaluation had no model scores in the reviewed evidence. Passing these checks made a new controlled comparison reasonable to attempt, not already successful.
The saved verifier reports exact initial and final metric replay for all 12 diagnostic models using the same frozen model code. Publication review checked the hashes of all 12 saved result records and their reported values, without loading checkpoints or running training. That is evidence review, not independent replication. The inspected upstream revision is not established as the exact paper-run commit.
Implementation, source review and writing were AI-assisted. The evidence summary retains per-seed metrics and result hashes. Diagnostic code, checkpoints and the full experiment package are not distributed with this note, so it is not a standalone reproduction package.