Product Engineer – AI Agent Evaluation

About the role

This Product Engineer role focuses on an AI agent evaluation product that turns agent traces into reliable, repeatable evaluations. You will work at the intersection of engineering and community, delivering features for agent builders while actively engaging on X and GitHub.

Key responsibilities

• Build and ship product features covering evals, replay, cohorts, and developer experience, emphasizing genuine utility.
• Create public content on X and GitHub, demoing and engaging with the AI engineering community to surface signals that shape the roadmap.
• Adapt weekly to tasks ranging from feature development to marketing motions and customer demos.
• Stay close to AI agent builders, evaluation practitioners, and developers working with agent frameworks and custom runtimes.
• Close the feedback loop between community input and shipped product, treating distribution and engineering as a continuous motion.

Required profile

• 3+ years building or shipping production software, with hands‑on experience building or evaluating agents in real systems.
• Proficiency in Python for production code.
• Hands‑on experience with agent frameworks such as LangGraph, Claude Code, or equivalents.
• Experience with evaluation systems for LLM or agent outputs, trace data, observability, or agent execution debugging.
• Track record shipping developer tools, SDKs, or APIs with strong focus on developer experience.
• Open‑source contributions with measurable adoption by other developers.
• Active, visible participation on X/Twitter or similar platforms discussing AI engineering or developer tooling.
• Comfort operating in an early‑stage, high‑uncertainty environment where wearing multiple hats is the norm.

Required skills

• Python
• LangGraph (or similar agent frameworks)
• Claude Code (or equivalent)
• LLM evaluation systems
• Trace data analysis and observability tools