Skip to content

Installation

Requirements

  • Python ≥ 3.13 (declared in pyproject.toml and .python-version)
  • uv — recommended package and environment manager
  • Git — for cloning the repository and CLARIN.SI dataset access
  • HuggingFace account — required for gated datasets (e.g., FineWeb-2); see HuggingFace Authentication
  • Disk space — pretraining corpora total tens of GB; plan accordingly
  • GPU (optional) — required for tokenizer/model training; CPU sufficient for data preparation

Clone and install

Clone the repository and create the virtual environment via uv:

git clone https://github.com/eriknovak/SLM4IE.git
cd SLM4IE
uv sync

This creates .venv/ and installs both runtime and dev dependencies pinned in uv.lock. Activate the environment for ad-hoc commands:

source .venv/bin/activate

Or prefix individual commands with uv run to skip activation.

Optional extras

The base uv sync only pulls runtime essentials. Install extras as needed:

Extra Command Purpose
dev uv sync --extra dev Test runner (pytest) and linter (ruff) for contributors
curate uv sync --extra curate datatrove pipeline for the pretraining corpus
tokenize uv sync --extra tokenize Tokenizer training + morphological evaluation (see Tokenizers)
docs uv sync --extra docs MkDocs Material toolchain for building this documentation site
notebook uv sync --extra notebook marimo notebooks + plotly/kaleido for experiment reporting and visualization

Multiple extras can be combined:

uv sync --extra dev --extra curate

Next steps