Tokenizers¶

🚧 In progress

This workflow is implemented but has not yet been run end-to-end, so this page is a stub. Detailed usage and results will land here once the pipeline is verified. Until then, treat the entry point and config below as the source of truth.

This stage trains six tokenizers (byte-level BPE, character-level charBPE, BERT WordPiece, SentencePiece Unigram, MorphBPE, and MorphPiece) across a 16k/32k/64k vocab sweep, scores them with six metrics (Fertility, CTC compression, Rényi efficiency, MorphScore, Morph-Edit-Distance, and Morph-Consistency), and exports each as a HuggingFace tokenizer. It consumes the deduplicated corpus (pretrain/05_2_dedup/) for training and two Sloleks-derived golds for the morph metrics: the inflectional gold (tokenization/sloleks.jsonl.gz) and the derivational gold (tokenization/sloleks_relations.jsonl.gz, from the Sloleks 2.0 word relations). The derivational gold adds point-estimate *_deriv metric columns and enriches the morphological backends' morpheme table. Requires the tokenize extra (uv sync --extra tokenize).

An optional scripts/tokenizers/prepare_sample.py step materializes the shared seeded training sample and morpheme lexicon up front, so reruns of the sweep reuse an identical persistent sample.

Entry point: scripts/tokenizers/train.py (also scripts/tokenizers/prepare_sample.py, scripts/tokenizers/analyze.py, and scripts/tokenizers/export.py) Config: configs/tokenizers/tokenizers.yaml