CI/CD¶
The main pipeline (ci.yml) runs in two stages, the first gating the second:
Static Checks → Integration Tests
Container images are not built here. They are built, scanned and published by the swiss-ai/model-launch-images repository; this repository only consumes the published squashfs files through its env tomls. See Building Container Images.
Workflows¶
| Workflow | Trigger | What it does |
|---|---|---|
ci.yml |
push to main, PRs, dispatch |
The pipeline below |
static.yml |
called by ci.yml |
Lint, format, type checks |
docs.yml |
docs/, mkdocs.yml, pyproject.toml changes |
mkdocs build --strict; deploys Pages from main |
sonar.yml |
push to main, PRs |
Unit tests + coverage → SonarCloud |
model-paths.yml |
daily at 05:30 UTC, dispatch | Checks every models.json and example path still holds a model |
Triggers¶
| Event | Behaviour |
|---|---|
PR to main |
Full pipeline |
Push to main |
Full pipeline |
| Draft PR | Static checks only |
workflow_dispatch |
Runs comprehensive tests |
PRs re-run on opened, reopened, synchronize, labeled, ready_for_review — the labeled trigger is what lets you switch test tiers without a new commit.
Stage 1: static checks¶
Seven parallel jobs: ruff lint/format, mypy, shellcheck, hadolint, markdownlint, taplo (TOML), prettier (JSON/YAML).
All reproducible locally with make static, or individually (make lint, make dockerlint, …). See Development.
Stage 2: integration tests¶
Tests hit a real cluster over FireCREST. At most one tier runs per PR, and none by default: the lightweight tier is temporarily opt-in while the cluster-backed CI is broken.
| Tier | Selected by | Target |
|---|---|---|
| Lightweight | requires-lightweight-tests label |
make _test-lightweight (-n 2) |
| Standard | requires-std-tests label |
make _test-std (-n 13) |
| Comprehensive | requires-comprehensive-tests label, or dispatch |
make _test-comprehensive (-n 28) |
Comprehensive wins over std, which wins over lightweight. A PR that only touches the catalog (models.json, env tomls, mfa_examples/) skips the tiers — there is no launcher code to test.
Locally, use the non-underscore targets (make test-lightweight) — they source .test.sh for credentials. See Development.
Model path checks¶
The paths-marked cases list each weights directory over FireCREST — no SLURM job, seconds for both suites:
| Test | Covers | Resolved from |
|---|---|---|
test_catalog_paths.py |
every models.json entry |
model_path, else <registry>/<vendor>/<model> |
test_example_paths.py |
every --model / --model-path / --tokenizer in examples/clariden/**/*.sh |
the flag value, with literal shell variables expanded |
A case fails if the directory is gone or unreadable, or if it holds none of its marker files — config.json or params.json (Mistral’s native layout) for a model, tokenizer.json or tokenizer_config.json for a tokenizer. An emptied checkpoint directory still exists, so presence alone proves nothing.
Example references are deduplicated by path, so a directory shared by several recipes is listed once and a failure names all of them. Only examples/clariden is covered: the beverin/bristen recipes target clusters CI’s credentials don’t reach.
These carry the lightweight, std and comprehensive marks as well, so every tier runs them: the launch tests only cover the models they launch, and only the comprehensive tier runs the examples at all. model-paths.yml runs the same target (make _test-paths) daily, which is what catches a path that rotted while nobody touched the repo.
Configuration¶
| Name | Kind | Used for |
|---|---|---|
SML_FIRECREST_API_KEY |
secret | FireCREST service-account API key |
SML_SWISSAI_RESEARCH_API_KEY |
secret | Integration tests |
SML_FIRECREST_URL, SML_SYSTEM, SML_PARTITION, SML_RESERVATION |
variable | The test cluster |
SONAR_TOKEN |
secret | SonarCloud (skipped for forked PRs) |
CI authenticates as a service account, not a personal account: the key is sent as an X-API-Key header, so SML_FIRECREST_URL must point at the PAT gateway (e.g. https://f7t-pat.api.svc.cscs.ch/mlp) rather than the Developer Portal endpoint.
When something fails¶
| Symptom | Cause |
|---|---|
| No integration tests ran | A static check failed (reproduce with make static), the PR is a draft, or it only touches the catalog |
path does not exist or is not readable |
A catalog entry’s or recipe’s weights moved or were deleted — repoint it or drop it |
UnexpectedStatusException: last request: 408 … / 500 … in a path test |
FirecREST couldn’t stat the path (command timeout, backend error) even after retries — the path itself is fine; re-run the job |
no model path extracted from: … |
An example’s model flag isn’t in a shape the extractor reads — see tests/example_paths.py |