Benchmarks¶
Slovenian evaluation datasets used for downstream IE tasks. Benchmarks are declared in configs/data/download.yaml with role: benchmark (or role: lexicon for tokenizer lexicons such as Sloleks) and a tasks: list, so they share the download pipeline with pretraining corpora. Use --only-benchmarks to fetch just the non-pretraining datasets.
| Dataset | Source | Tasks | Description |
|---|---|---|---|
| SUK 1.1 | CLARIN.SI | POS, LEMMA, DEP, NER, SRL, COREF, WSD, SA | ~1M tokens / 881K words / 2,913 texts manually annotated with MULTEXT-East V6, JOS, and Universal Dependencies. Integrates ssj500k 2.3, Ambiga, ElexisWSD, and SentiCoref subcorpora. License: CC BY-SA 4.0. |
| ssj500k 2.3 | CLARIN.SI | POS, LEMMA, DEP, NER, SRL | ~500K tokens manually annotated with MSD tags, lemmas, UD syntax (UD 2.8), named entities, and semantic role labels. Foundation corpus for SUK 1.1. License: CC BY-NC-SA 4.0. |
| Slovene SuperGLUE | CLARIN.SI | QA, NLI, WSD, COREF, MRC | Slovene translation of SuperGLUE (BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, WSC). Mix of human and Google MT translation. License: CC BY 4.0. Convert to per-task evaluation files with scripts/data/to_superglue.py. |
| SentiNews 1.0 | CLARIN.SI | SA | Slovene news sentiment with three-level annotations (sentence, paragraph, document) and 3-class labels. Directly downloadable. License: CC BY-SA 4.0. Convert to evaluation JSONL with scripts/data/to_sentiment.py. |
| Sloleks 3.1 | CLARIN.SI | TOKENIZER | Slovenian inflectional lexicon (lemmas + word forms with MULTEXT-East V6 / JOS MSDs). Tokenizer / morphology evaluation only — intentionally absent from extract.yaml, never enters the pretraining corpus. Distributed as TEI XML. License: CC BY-SA 4.0. Convert with scripts/data/to_tokenization.py. |
Task abbreviations¶
See Datasets.
Conversion¶
Each benchmark is converted to its evaluation-ready shape via a dedicated pipeline script:
| Benchmark | Conversion script |
|---|---|
| SUK / ssj500k | to_spans.py (NER) |
| Slovene SuperGLUE | to_superglue.py |
| SentiNews | to_sentiment.py |
| Sloleks | to_tokenization.py |
The task converters and evaluation are implemented but not yet run end-to-end — see the 🚧 banners on those pages. Once converted, run Evaluation to score trained models.