Status: version 0.2.0 (November 2026). The name is provisional; check it against trademark databases (EUIPO / TMview) before any public launch.
Make “train a language model from scratch” something an ordinary person can do on an ordinary computer: one binary, one text file, no Python, no GPU, no accounts, nothing leaving the machine. The models are small on purpose. The point is to learn how training behaves, to experiment with data, and to have a transparent, auditable base to build on.
GptConfig::train_flops_per_token in config.rs implements the exact estimate.germen bench measures real tokens per second; time = D / (tokens per second). Measure first,
then pick a size. Reference numbers from a 2-core sandbox are in the README.| decision | choice | why |
|---|---|---|
| framework | Burn 0.22.1, pinned (=0.22.1) |
the new device-based API is stable within 0.22; a pin keeps builds reproducible |
| backend | flex |
pure Rust, gemm matmul, run-time SIMD selection, rayon; builds on Windows, macOS, Linux, even WASM, with no native dependency |
| not used | cpu (CubeCL/LLVM) |
needs an LLVM toolchain |
| not used | ndarray |
legacy path in Burn 0.22 |
| not used | LibTorch | deprecated in 0.22, large native dependency |
| not used | Burn’s Learner |
a hand-written loop is ~100 lines, gives full control over events, checkpoints and Ctrl-C |
| tokens | bytes (default) or a trained byte-level BPE (u16 ids, up to 65,536) |
bytes need nothing to train or ship and round-trip any input; BPE gives 3–4× more text per window. The merges are learned from the training split only, so validation stays honest; merges never cross a pre-token, which keeps training and encoding cheap and parallel |
| feed-forward activation | squared ReLU, relu(x)² |
on flex 0.22.1 erf/tanh/exp do not use more than one thread and cost several ns per element: one GELU call took ~13× longer than relu(x)² on the mini shapes (profile_ops), and that time does not shrink with more cores. In profile_mlp the squared-ReLU feed-forward block (forward + backward on the mini shapes) went from 381 to 220 ms on one → two threads (1.73×), a SwiGLU block with the same parameter count from 400–443 to 245–278 ms (≈ 1.6×). On the demo corpus both reached a similar validation loss after 300 steps (0.276 for squared ReLU, 0.285 for GELU; single runs) |
| positions | rotary (own implementation, rotate-half layout, constant tables) | no learned position table, works for any window up to the trained context |
| attention | training: block-wise causal (blocks of 2 × head dimension, scale folded into the queries, additive mask); evaluation: Burn’s fused kernel; generation: key/value cache | under autodiff Burn’s attention falls back to a long chain of ops that also computes the masked upper triangle; the block-wise version skips it (whole steps 8 % faster for mini, 28 % for nano on two cores; same values and gradients, tested). Of block sizes 32/64/128/256, twice the head dimension was best for head dimensions 32 and 64 |
| gradient clipping | global norm over all parameters, default 1 | the standard recipe; Burn’s built-in clipping works per tensor |
| optimizer state | saved next to the weights (optimizer.bpk, keyed by parameter id) |
resuming without it causes a loss bump; a test shows the step after a reload equals the uninterrupted one |
| threads | the whole command runs inside its own rayon pool (sys::run_in_pool) |
+7 % (mini) to +25 % (nano) on two cores compared with driving the global pool from an outside thread, because every parallel loop otherwise needs a hand-off between threads |
| checkpoint | burnpack + JSON | native to Burn, atomic writes, no extra dependency |
| UI language | Romanian messages, English code and docs, READMEs in eight languages | first user is Romanian; moving the messages to a catalogue is a starter issue |
mini, batch 16 × 256)Where the time of one step goes at one thread: forward ≈ 33 %, backward ≈ 66 %, AdamW update
≈ 1.3 %. With two threads the whole step is about 1.5× faster (1.51× in bench --scaling), made up of:
| part | speed-up on 2 threads |
|---|---|
| matrix products in the feed-forward block | 1.7–1.8× |
| attention (before the block-wise version) | 1.24× |
| RMSNorm, rotary embeddings, AdamW update | ≈ 1.0× |
By Amdahl’s law the parts that do not scale decide what many cores can give, which is why
germen bench --scaling exists. What was done about it, and what was not:
mini), + 22 % (micro), + 28 % (nano)
on two threads (three interleaved runs each, against Burn’s attention); + 5 % for mini at one
thread. Blocks of twice the head dimension beat a fixed 128 by 3–6 % for nano and micro.mini), + 17 % (micro), + 25 % (nano); nano
went from about 5.5–6.1 k to 7.1–7.9 k tokens/s over three interleaved runs, which also shows
how much of a small step was thread hand-offs.--batch did not raise tokens per second.mini shapes stay on one
thread; erf/tanh/exp are single-threaded. Candidates: a parallel AdamW update over
parameter tensors, fused RMSNorm and rotary kernels, fewer tensor copies.cargo run --release -p germen-core --example profile_ops -- N times single operations with N
threads; profile_step, profile_mlp and profile_attn do the same for a whole step and for the
feed-forward and attention blocks.
--resume; global-norm clipping; key/value cache;
gradient accumulation; bench --scaling; READMEs in eight languages.wgpu) behind a Cargo feature; community benchmark table.germen bench --scaling --json bench.json produces the curve to paste into the README and to
decide what to parallelise next.base
preset; any real-world corpus; loading a GERMEN model in another engine.