The Vesper-400M transformer is implemented. It is not a trained language model. As of September 11, we have not started either the planned small-model corpus-training run or target-model training.

The distinction matters when reporting progress. A forward pass can be correct before the model has learned anything useful. A data pipeline can preserve every input byte without establishing that those inputs are suitable for training. Our current work is on those prerequisites.

What the implementation checks establish

The decoder-only PyTorch implementation uses grouped-query attention, rotary position embeddings, RMSNorm, and SwiGLU, with tied input and output embeddings. The first-model tokenizer has 32,768 content IDs plus an explicit end-of-sequence ID.

The target configuration contains 386,060,800 parameters. A separate configuration for the planned smaller training run contains 50,350,848. We checked both counts against modules constructed on PyTorch’s meta device, which represents tensor shapes without allocating the corresponding parameter storage. Neither count establishes that a full training job fits the available hardware.

The September 10 build recorded 17 CPU model tests and 3 CUDA model tests passing. They covered model semantics and numerical behavior, including causal masking, next-token targets, and checkpoint recomputation. The CUDA checks compared BF16 with FP32 references and exercised an 8,192-position forward pass on a tiny model. They did not run a target-sized model or establish trained long-context performance.

One backend limitation remains explicit. Flash Attention was unavailable in the installed Windows PyTorch wheel; forced-Flash checks failed. The passing CUDA suite selected memory-efficient scaled dot-product attention. It would be inaccurate to report this as verified Flash Attention support.

The CPU suite also performed two AdamW updates on tiny synthetic inputs. Those updates checked finite gradients and parameter changes. They were not the planned small-model training experiment. Saved-state tests cover local model-state loading, not a durable training-resume system with optimizer and data-loader state.

The dataset pilot is still small

The offline pipeline preserves source provenance, writes checksummed token shards, and constructs single-document windows with explicit target masks. Padding does not count as a prediction target; a real end-of-sequence token does. Authored fixtures exercise those rules before external text enters the pipeline.

A later Stack-Edu pilot retained five reviewed Python files from five repositories: 8,122 source bytes and 2,226 tokens including end-of-sequence tokens. The repository-separated split contained two training-labelled documents, two validation documents, and one public pipeline-evaluation document. Those labels describe the pipeline test; they do not grant model-training permission.

Copied-source builds reproduced the pilot artifacts byte for byte on the same machine. The files remain pipeline-only, not training-admitted. Private-benchmark exclusion and broader source-rights review are unresolved. We did not execute the downloaded programs, and the pilot’s tensor checks made no optimizer updates.

The latest math-source sampling attempt failed row validation. An offline correction changed how the parser handles crawl-date labels, but the corrected parser has not been tested on a live response. No OpenWebMath candidate rows were retained or admitted. There is no basis yet to describe this as a working mixed-source training corpus.

What must precede training

The next dataset work needs source-specific rights evidence and a qualified sample that can pass the fixed pipeline checks. A training corpus also needs an evaluation-exclusion inventory and enough coverage to support the intended experiment. A permissive label in a dataset catalogue is not sufficient by itself.

Before the planned small-model run, we also need a training and resume path with explicit checkpoint state, a frozen experiment configuration, and measured resource requirements. That run should test the training system before committing to the larger configuration. We are not publishing a completion date or a model-quality estimate from the software tests.

Our tokenizer optimization note covers one completed engineering comparison in detail. The rest of this update reports development status, not a capability benchmark.

Evidence for this update

The publication evidence summary records the milestone counts and their scope. We checked the saved reports and selected hashes for publication; we did not rerun the CPU/CUDA suites, acquire more data, or start training. The implementation snapshots and source corpus remain private, so the summary is not an external reproduction package.