Train a small language model from scratch, in pure Rust, on an ordinary CPU.
No Python, no CUDA, no C or C++ toolchain, no account, no network access, no telemetry.
One binary, one text file, one command. Built on Burn 0.22 with its pure-Rust
flex CPU backend.
Languages: English · Română · Français · Italiano · Español · Português · Deutsch · Magyar
GERMEN trains a GPT-style network (RMSNorm, rotary positions, squared-ReLU feed-forward, no biases)
over tokens. By default a token is a byte: the vocabulary is the 256 byte values, so any
text — Romanian diacritics included — works with nothing to prepare. With --tokenizer bpe GERMEN
first learns a byte-level BPE vocabulary (up to 65,536 tokens) from your own text, so the same
context window holds several times more text (typically 3–4× for ordinary prose). The tokenizer is
stored with the model.
It is a workshop for learning and experimenting, and a base for research: you can watch a network go from random noise to spelling, punctuation and the style of your text within minutes. It is not a chatbot factory. A model of 1–25 million parameters trained on a few hundred megabytes learns the form of a text (words, grammar fragments, layout), not world knowledge or conversation. Useful assistants need orders of magnitude more compute than a desktop CPU offers.
You need Rust 1.95 or newer (rustup.rs; on Windows it also asks for the Visual Studio Build Tools). Then:
cargo build --release
./target/release/germen bench # how fast is THIS computer?
./target/release/germen bench --scaling # ...and how does it grow with the number of threads?
./target/release/germen train --demo --preset micro --steps 300
./target/release/germen generate # text from the model you just trained
On Windows use target\release\germen.exe. Pre-built binaries are attached to the releases.
Train on your own text — any file, any language:
germen train --data my-corpus.txt --preset mini --tokenizer bpe --vocab 4096 --steps 5000 --out my-model
germen generate --model my-model --prompt "Era odată " --max-new 400
Press Ctrl-C during training: it finishes the current step, saves, and exits. Continue later
with the same command plus --resume: weights, step counter and AdamW state are restored, so
training goes on without the loss bump of a cold restart (the random text windows are drawn anew).
| command | what it does |
|---|---|
germen info |
detected processors and SIMD level, model sizes and memory for the weights |
germen bench |
times real training steps for each model size and extrapolates the time for 100 M and 1 B tokens; --scaling repeats the measurement with 1, 2, 4, … threads and prints the speed-up curve; --json FILE saves the numbers |
germen demo-corpus |
writes a synthetic Romanian corpus generated from a small grammar (--size-mb) |
germen train |
trains from scratch (--data FILE or --demo), or continues with --resume |
germen generate |
samples text from a trained model, with a key/value cache (--temperature, --top-k, --prompt) |
Training options worth knowing: --tokenizer byte|bpe and --vocab N (the BPE size);
--batch and --accum (gradient accumulation: the effective batch is their product, the memory
is that of --batch alone); --grad-clip (global gradient norm, default 1); --lr, --warmup,
--weight-decay. germen train --help lists them all. Add --threads N to any command to limit
the number of worker threads (the default is every logical processor). The messages of the command
line are currently in Romanian.
| preset | parameters | layers × heads × width | weights (f32) |
|---|---|---|---|
nano |
0.85 M | 4 × 4 × 128 | 3 MiB |
micro |
2.76 M | 6 × 6 × 192 | 11 MiB |
mini |
10.82 M | 6 × 6 × 384 | 41 MiB |
small |
25.44 M | 8 × 8 × 512 | 97 MiB |
base |
85.35 M | 12 × 12 × 768 | 326 MiB |
The sizes are for the 256-token byte vocabulary; a BPE vocabulary of V tokens adds
2·(V − 256)·width parameters (embedding and output head). Training also keeps AdamW’s two moment
estimates (≈ 4× the weights) plus the activations, which grow with --batch and --seq.
The cost of training is about 6 × parameters × tokens floating-point operations (with bytes,
1 MB of text = 1 M tokens). Divide by what your CPU really delivers and you have the time.
germen bench measures exactly that on your machine.
Reference measurement — a small Linux virtual machine with 2 cores (AVX-512), batch 16 × 256, byte vocabulary (one run on a noisy VM, so ± 5 %):
| preset | tokens/s | effective GFLOPS | 100 M tokens | 1 B tokens |
|---|---|---|---|---|
nano |
8,261 | 54 | 3.4 h | 1.4 d |
micro |
3,180 | 63 | 8.7 h | 3.6 d |
mini |
1,260 | 90 | 22.0 h | 9.2 d |
small |
545 | 90 | 2.1 d | 21 d |
Going from 1 to 2 threads made mini 1.51× faster here (825 → 1,247 tokens/s), so do not expect
20 cores to give 20×. Matrix products scale well (≈ 1.8× on two threads); attention scales
less (≈ 1.2×), and RMSNorm, rotary embeddings and the AdamW update hardly at all, so on a machine
with many cores they become the bottleneck (Amdahl’s law). That is what
germen bench --scaling is for: it prints the whole curve and the number of threads beyond which
nothing is gained. Please run germen bench --scaling --json bench.json on your machine — it
is the only way to know. A common rule of thumb for compute-optimal training is about 20 tokens
per parameter; small models are often trained on much more.
Generation uses a key/value cache: 400 tokens from mini take about 1 second instead of 15 on the
same two cores.
crates/germen-core the library: config, model, tokenizer, bpe, data, train, gradients,
generate, checkpoint, bench
crates/germen-cli the `germen` command line (clap)
flex: pure Rust, matrix products through the gemm crate, SIMD kernels
(AVX2/AVX-512/SSE, NEON) picked at run time, multi-threaded with rayon. A generic x86-64
build therefore still uses AVX2 on a CPU that has it.1/√(2·layers)).
Training uses a block-wise causal attention that never computes the part above the diagonal
(whole steps 8–28 % faster on two cores, depending on the model size; same values and
gradients); evaluation uses Burn’s fused kernel, and generation a key/value cache.Event callbacks so a graphical front end
can reuse the same loop.mini to + 25 % for nano on two cores).
Spreading micro-batches over several threads that all drive Burn was tried and rejected; see
docs/PLAN.md.config.json, weights.bpk (Burn’s burnpack, f32),
optimizer.bpk (AdamW moments), tokenizer.json and meta.json. Checkpoints written by
version 0.1 still load.demo corpus is synthetic and tiny: it proves the pipeline learns, nothing more.See CHANGELOG.md and docs/PLAN.md. Contributions are welcome — the open issues are a good start (CONTRIBUTING.md).
MIT — see LICENSE. Copyright © 2026 Alexandru-Ionuț Chiuță (Alexio).