Senior/Staff AI Engineer – Quality & Evaluations

About the role

We are looking for an engineer with strong backend, data, and AI systems experience to build the evaluation and observability foundation for production‑grade LLM agents used in complex audit workflows. This role sits at the intersection of backend engineering, data infrastructure, and AI quality.

Key responsibilities

• Build online and offline evaluation systems for LLM agents, using golden datasets, ground‑truth data, human review workflows, and experiment results.
• Create automated quality gates to test changes to prompts, context, models, or agent logic before production.
• Analyze large volumes of agent traces in BigQuery or ClickHouse to identify failure modes, quality regressions, latency issues, reliability gaps, and cost‑optimization opportunities.
• Develop reliable data retention and replay mechanisms for long‑term analysis of production agent behavior.
• Manage observability tools for tracing, monitoring, debugging, and experiment management of audit agents.
• Collaborate with backend engineers to improve the speed and reliability of retrieval and reasoning agents.

Required profile

• Strong Python and/or backend engineering experience.
• Solid understanding of LLM and agent evaluation methods, including deterministic checks, ground truth, LLM‑as‑judge, human review, and quality metrics.
• Experience deploying and operating cloud systems, preferably on Google Cloud Platform.
• Hands‑on experience building end‑to‑end retrieval or ML pipeline evaluation systems and using tools such as Braintrust, MLflow, Langfuse, or Weights & Biases.
• Comfortable working with analytical databases, data warehouses, columnar stores, and high‑volume event or trace data.
• Knowledge of system design, reliability, observability, monitoring, logging, debugging, and operational trade‑offs.
• Ability to make architectural decisions, communicate trade‑offs, and build extensible systems.
• Comfortable with ambiguity, first‑principles reasoning, and building production AI infrastructure.

Required skills

• Python
• Google Cloud Platform (GCP)
• BigQuery
• ClickHouse
• Braintrust
• MLflow
• Langfuse
• Weights & Biases
• Temporal (optional)

What we offer

• High impact & growth: shape strategy at a scaling AI startup from day one.
• Mission‑driven culture: ambitious team valuing first‑principles thinking and bold ideas.
• Attractive compensation: competitive salary plus significant equity.
• Best tools for the job: generous coding tools budget.
• Startup perks: flexible vacation, team lunches, retreats, central Berlin office.