Apertus Engineer: Evaluations

The engineer will own the evaluation codebase and pipelines that inform training decisions and releases.

Evaluation infrastructure

• Build and maintain the evaluation codebase and pipelines for Apertus models, from checkpoints during training to released models
• Make evaluations run quickly and at scale: parallel execution on Alps, efficient use of inference backends, caching, and result tracking
• Reduce mismatch between evaluations during training and during serving: consistent tokenisation, chat templates, prompting, and sampling across evaluation harnesses and inference engines
• Debug evaluation failures, regressions, and inconsistencies across backends

Benchmark coverage

• Integrate and run the evaluations the project cares about. The design of new evaluations is owned by collaborating researchers and engineers; this role makes them run reliably and at scale
• Cover image and audio evaluations alongside text within the same pipeline
• Integrate new benchmarks as the field evolves, working with our academic collaborators to onboard the benchmarks they create, and validate that metrics and harness implementations are trustworthy

Comparative and third-party evaluation

• Evaluate third-party services and other open and closed models against the same benchmark suite, producing directly comparable and reproducible results
• Provide evaluation results, reports, and dashboards that support training decisions (data mixtures, ablations) and release decisions
• Work closely with the engineers focused on safety, deployment, and community needs, and integrate the evaluations they create into the shared pipeline

Essential

• MSc or PhD in Computer Science, Data Science, Artificial Intelligence, Machine Learning, or a related field. Exceptional
• BSc candidates with strong engineering experience will also be considered
• Strong Python and software engineering skills, including experience building robust data or evaluation pipelines
• Experience with LLM evaluation: established harnesses (e.g. lm-evaluation-harness) or custom benchmark tooling
• Strong collaboration and communication skills and ability to work across research and engineering teams
• Prior hands-on experience in the core domains of this role is required. This can be project or study based experience; formal work experience is preferred
• A high degree of flexibility: priorities, tools, and day-to-day tasks shift with training schedules, releases, and a fast-moving field
• Experience running evaluations at scale on GPU clusters (Slurm or similar) and with inference engines such as vLLM or SGLang
• Familiarity with agentic evaluation and agentic harnesses: tool use, sandboxed execution environments, benchmarks such as SWE-bench or similar
• Experience with multimodal (image or audio) model evaluation

Strongly preferred

• An eye for statistical rigor: variance across runs, prompt sensitivity, significance of differences between models

Nice to have

• Published research in the domains relevant to this role, or familiarity with recently published research on these topics
• Experience with LLM-as-judge pipelines and their calibration
• Familiarity with benchmark contamination detection and decontamination practices
• Experience visualising and communicating evaluation results to research teams