GERMEN

GERMEN — plan and decisions

Status: version 0.2.0 (November 2026). The name is provisional; check it against trademark databases (EUIPO / TMview) before any public launch.

1. Goal

Make “train a language model from scratch” something an ordinary person can do on an ordinary computer: one binary, one text file, no Python, no GPU, no accounts, nothing leaving the machine. The models are small on purpose. The point is to learn how training behaves, to experiment with data, and to have a transparent, auditable base to build on.

2. The cost model — why sizes are what they are

3. Decisions

decision choice why
framework Burn 0.22.1, pinned (=0.22.1) the new device-based API is stable within 0.22; a pin keeps builds reproducible
backend flex pure Rust, gemm matmul, run-time SIMD selection, rayon; builds on Windows, macOS, Linux, even WASM, with no native dependency
not used cpu (CubeCL/LLVM) needs an LLVM toolchain
not used ndarray legacy path in Burn 0.22
not used LibTorch deprecated in 0.22, large native dependency
not used Burn’s Learner a hand-written loop is ~100 lines, gives full control over events, checkpoints and Ctrl-C
tokens bytes (default) or a trained byte-level BPE (u16 ids, up to 65,536) bytes need nothing to train or ship and round-trip any input; BPE gives 3–4× more text per window. The merges are learned from the training split only, so validation stays honest; merges never cross a pre-token, which keeps training and encoding cheap and parallel
feed-forward activation squared ReLU, relu(x)² on flex 0.22.1 erf/tanh/exp do not use more than one thread and cost several ns per element: one GELU call took ~13× longer than relu(x)² on the mini shapes (profile_ops), and that time does not shrink with more cores. In profile_mlp the squared-ReLU feed-forward block (forward + backward on the mini shapes) went from 381 to 220 ms on one → two threads (1.73×), a SwiGLU block with the same parameter count from 400–443 to 245–278 ms (≈ 1.6×). On the demo corpus both reached a similar validation loss after 300 steps (0.276 for squared ReLU, 0.285 for GELU; single runs)
positions rotary (own implementation, rotate-half layout, constant tables) no learned position table, works for any window up to the trained context
attention training: block-wise causal (blocks of 2 × head dimension, scale folded into the queries, additive mask); evaluation: Burn’s fused kernel; generation: key/value cache under autodiff Burn’s attention falls back to a long chain of ops that also computes the masked upper triangle; the block-wise version skips it (whole steps 8 % faster for mini, 28 % for nano on two cores; same values and gradients, tested). Of block sizes 32/64/128/256, twice the head dimension was best for head dimensions 32 and 64
gradient clipping global norm over all parameters, default 1 the standard recipe; Burn’s built-in clipping works per tensor
optimizer state saved next to the weights (optimizer.bpk, keyed by parameter id) resuming without it causes a loss bump; a test shows the step after a reload equals the uninterrupted one
threads the whole command runs inside its own rayon pool (sys::run_in_pool) +7 % (mini) to +25 % (nano) on two cores compared with driving the global pool from an outside thread, because every parallel loop otherwise needs a hand-off between threads
checkpoint burnpack + JSON native to Burn, atomic writes, no extra dependency
UI language Romanian messages, English code and docs, READMEs in eight languages first user is Romanian; moving the messages to a catalogue is a starter issue

4. Performance findings (2-core AVX-512 VM, mini, batch 16 × 256)

Where the time of one step goes at one thread: forward ≈ 33 %, backward ≈ 66 %, AdamW update ≈ 1.3 %. With two threads the whole step is about 1.5× faster (1.51× in bench --scaling), made up of:

part speed-up on 2 threads
matrix products in the feed-forward block 1.7–1.8×
attention (before the block-wise version) 1.24×
RMSNorm, rotary embeddings, AdamW update ≈ 1.0×

By Amdahl’s law the parts that do not scale decide what many cores can give, which is why germen bench --scaling exists. What was done about it, and what was not:

cargo run --release -p germen-core --example profile_ops -- N times single operations with N threads; profile_step, profile_mlp and profile_attn do the same for a whole step and for the feed-forward and attention blocks.

5. Roadmap

6. Open questions

  1. Name — “GERMEN” is provisional; run a trademark search.
  2. Interoperability — colibri’s README lists thirteen engines (GLM, Kimi, DeepSeek, Qwen, OLMoE, …), loads language models from GGUF files as they are, and says checkpoints with the same architecture as a supported one run unchanged; for a new architecture it points to a new C engine file over shared headers, with no step-by-step guide. A GERMEN model (dense, squared-ReLU feed-forward, byte or BPE vocabulary) is not among the listed architectures, so it cannot be loaded there unchanged. Nothing has been tried; a GGUF or safetensors export would be the first step and is not planned before 0.4. Source: the README of https://github.com/JustVugg/colibri, read on 2026-11-17.
  3. Training data — any pre-trained model that is shipped needs a corpus whose licence allows it (public-domain literature, openly licensed Wikipedia text, own writing). Not decided.
  4. Measured speed on the author’s Windows PC (20 cores) — not yet known. germen bench --scaling --json bench.json produces the curve to paste into the README and to decide what to parallelise next.

7. What has been verified so far