All notable changes to GERMEN are listed here. The format follows Keep a Changelog; versions follow Semantic Versioning.
Closes the limitations of 0.1 that mattered most (tokenizer, resume, clipping, generation speed,
batch size) and makes the training step faster. Measured on the same 2-core AVX-512 virtual
machine as in 0.1, batch 16 × 256: nano 6,040 → 8,261 tokens/s, micro 2,362 → 3,180,
mini 1,051 → 1,260, small 522 → 545.
--tokenizer bpe, --vocab N from 257 to 65,536, --bpe-sample-mb): a
byte-level BPE with a GPT-2-like pre-tokenizer (a space sticks to the following word; merges
never cross a word), deterministic training with incremental pair counts, parallel encoding.
It is learned from the training part of the text only, stored in the checkpoint
(tokenizer.json) and used by generate; every byte sequence round-trips exactly.
--resume insists on the tokenizer the model was trained with.--resume (optimizer.bpk): training continues without the loss bump of a
cold restart. Checkpoints from 0.1 still load; the optimizer then restarts from zero and the
command says so.--grad-clip, default 1, 0 = off) instead of Burn’s
per-tensor clipping; the norm is shown in the training log.--accum): the effective batch is --batch × --accum, the memory
is that of --batch.Gpt::forward_cached, KvCache). 400 tokens from mini:
15.3 s → 1.0 s; 300 bytes from nano: 1.23 s → 0.10 s. When the window is full, generation
continues from its most recent half.germen bench --scaling [LIST]: the benchmark repeated on 1, 2, 4, … threads (or the list
you give), with speed-up, efficiency and the thread count beyond which nothing is gained;
--json includes it. bench --vocab measures a BPE-sized output head.profile_step, profile_mlp and profile_attn (where the time of a
training step goes, and which parts scale with the number of threads).mini), 22 % (micro) and
28 % (nano) faster on two cores, with the same values and gradients as Burn’s attention
(tested to about 1e-5).mini, + 17 % for micro and + 25 % for nano on two cores.tokenizer::Tokenizer (byte or BPE) replaces ByteTokenizer; generate takes a
tokenizer and returns token ids; Event::Start and Event::Step carry more fields; Meta
records the number of accumulated micro-batches.First working version, built to answer one question: how fast can a small language model be trained from scratch on an ordinary CPU, in pure Rust?
flex CPU backend. No Python, CUDA, LLVM or C/C++
toolchain required.nano (0.85 M parameters), micro (2.8 M), mini (10.8 M),
small (25.4 M), base (85.4 M).--resume, clean stop on Ctrl-C.config.json, weights.bpk (Burn burnpack, f32), meta.json; written atomically.germen bench: times real training steps on the current machine and extrapolates the time
needed for 100 M and 1 B tokens; --json saves the measurements.germen demo-corpus / train --demo: a synthetic Romanian corpus generated from a small
grammar, for smoke tests and a first run that finishes in minutes.germen info: detected processors, SIMD level, model sizes and memory for weights.--seq 99999) is refused with a clear message instead of failing inside the allocator.examples/profile_ops.rs: times single tensor operations with a chosen number of threads, to
find which ones do not scale (run it on the target machine).germen bench and share
the numbers.