GERMEN

Train a small language model from scratch, in pure Rust, on an ordinary CPU.

No Python, no CUDA, no C or C++ toolchain, no account, no network access, no telemetry. One binary, one text file, one command. Built on Burn 0.22 with its pure-Rust flex CPU backend.

Languages: English · Română · Français · Italiano · Español · Português · Deutsch · Magyar

What it is — and what it is not

GERMEN trains a GPT-style network (RMSNorm, rotary positions, squared-ReLU feed-forward, no biases) over tokens. By default a token is a byte: the vocabulary is the 256 byte values, so any text — Romanian diacritics included — works with nothing to prepare. With --tokenizer bpe GERMEN first learns a byte-level BPE vocabulary (up to 65,536 tokens) from your own text, so the same context window holds several times more text (typically 3–4× for ordinary prose). The tokenizer is stored with the model.

It is a workshop for learning and experimenting, and a base for research: you can watch a network go from random noise to spelling, punctuation and the style of your text within minutes. It is not a chatbot factory. A model of 1–25 million parameters trained on a few hundred megabytes learns the form of a text (words, grammar fragments, layout), not world knowledge or conversation. Useful assistants need orders of magnitude more compute than a desktop CPU offers.

Quick start

You need Rust 1.95 or newer (rustup.rs; on Windows it also asks for the Visual Studio Build Tools). Then:

cargo build --release
./target/release/germen bench                      # how fast is THIS computer?
./target/release/germen bench --scaling            # ...and how does it grow with the number of threads?
./target/release/germen train --demo --preset micro --steps 300
./target/release/germen generate                   # text from the model you just trained

On Windows use target\release\germen.exe. Pre-built binaries are attached to the releases.

Train on your own text — any file, any language:

germen train --data my-corpus.txt --preset mini --tokenizer bpe --vocab 4096 --steps 5000 --out my-model
germen generate --model my-model --prompt "Era odată " --max-new 400

Press Ctrl-C during training: it finishes the current step, saves, and exits. Continue later with the same command plus --resume: weights, step counter and AdamW state are restored, so training goes on without the loss bump of a cold restart (the random text windows are drawn anew).

Commands

command what it does
germen info detected processors and SIMD level, model sizes and memory for the weights
germen bench times real training steps for each model size and extrapolates the time for 100 M and 1 B tokens; --scaling repeats the measurement with 1, 2, 4, … threads and prints the speed-up curve; --json FILE saves the numbers
germen demo-corpus writes a synthetic Romanian corpus generated from a small grammar (--size-mb)
germen train trains from scratch (--data FILE or --demo), or continues with --resume
germen generate samples text from a trained model, with a key/value cache (--temperature, --top-k, --prompt)

Training options worth knowing: --tokenizer byte|bpe and --vocab N (the BPE size); --batch and --accum (gradient accumulation: the effective batch is their product, the memory is that of --batch alone); --grad-clip (global gradient norm, default 1); --lr, --warmup, --weight-decay. germen train --help lists them all. Add --threads N to any command to limit the number of worker threads (the default is every logical processor). The messages of the command line are currently in Romanian.

Model sizes

preset parameters layers × heads × width weights (f32)
nano 0.85 M 4 × 4 × 128 3 MiB
micro 2.76 M 6 × 6 × 192 11 MiB
mini 10.82 M 6 × 6 × 384 41 MiB
small 25.44 M 8 × 8 × 512 97 MiB
base 85.35 M 12 × 12 × 768 326 MiB

The sizes are for the 256-token byte vocabulary; a BPE vocabulary of V tokens adds 2·(V − 256)·width parameters (embedding and output head). Training also keeps AdamW’s two moment estimates (≈ 4× the weights) plus the activations, which grow with --batch and --seq.

How long does it take?

The cost of training is about 6 × parameters × tokens floating-point operations (with bytes, 1 MB of text = 1 M tokens). Divide by what your CPU really delivers and you have the time. germen bench measures exactly that on your machine.

Reference measurement — a small Linux virtual machine with 2 cores (AVX-512), batch 16 × 256, byte vocabulary (one run on a noisy VM, so ± 5 %):

preset tokens/s effective GFLOPS 100 M tokens 1 B tokens
nano 8,261 54 3.4 h 1.4 d
micro 3,180 63 8.7 h 3.6 d
mini 1,260 90 22.0 h 9.2 d
small 545 90 2.1 d 21 d

Going from 1 to 2 threads made mini 1.51× faster here (825 → 1,247 tokens/s), so do not expect 20 cores to give 20×. Matrix products scale well (≈ 1.8× on two threads); attention scales less (≈ 1.2×), and RMSNorm, rotary embeddings and the AdamW update hardly at all, so on a machine with many cores they become the bottleneck (Amdahl’s law). That is what germen bench --scaling is for: it prints the whole curve and the number of threads beyond which nothing is gained. Please run germen bench --scaling --json bench.json on your machine — it is the only way to know. A common rule of thumb for compute-optimal training is about 20 tokens per parameter; small models are often trained on much more.

Generation uses a key/value cache: 400 tokens from mini take about 1 second instead of 15 on the same two cores.

How it is built

crates/germen-core   the library: config, model, tokenizer, bpe, data, train, gradients,
                     generate, checkpoint, bench
crates/germen-cli    the `germen` command line (clap)

Limitations (version 0.2)

See CHANGELOG.md and docs/PLAN.md. Contributions are welcome — the open issues are a good start (CONTRIBUTING.md).

License

MIT — see LICENSE. Copyright © 2026 Alexandru-Ionuț Chiuță (Alexio).