Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)

▣ About the role

Build realistic enterprise evaluation environments spanning connected applications, representative datasets, user roles, and permissions, with complex, multi-step workflows and reproducible system state. Own evaluation infrastructure, task suites, and automated graders that measure how well agents complete enterprise work, calibrated against human judgment and grounded in end-to-end outcomes.

☰ What you would do

• Build realistic enterprise evaluation environments spanning connected applications, representative datasets, user roles, and permissions, with complex, multi-step workflows and reproducible system state.
• Own evaluation infrastructure, task suites, and automated graders that measure how well agents complete enterprise work, calibrated against human judgment and grounded in end-to-end outcomes.
• Analyze agent trajectories and production failures, identify root causes, and ship improvements across the agent harness, tools, and skills that measurably improve task success and reliability.
• Develop reusable skills and improve tool descriptions, schemas, and result handling so agents can select the right tools, execute workflows correctly, and recover from errors with fewer unnecessary calls.
• Build and evaluate model-selection and routing strategies that match each task to the right model and effort level, balancing quality, latency, reliability, and cost.
• Improve agent efficiency through context management, skill loading, caching, and execution strategies that reduce wasted tokens and cost per successful task without compromising quality or safety.
• Build regression gates and launch-readiness signals for model, connector, tool, skill, and harness changes, and validate that evaluation gains translate into better production outcomes.

◔ What the employer asks for

• + years of software engineering or machine learning engineering experience, with a track record of owning complex production systems from design through deployment and operation.
• Strong system design skills, including building reliable, scalable infrastructure for experimentation, evaluation, or data-intensive applications.
• Strong proficiency in a backend language such as Python, Go, or Rust, and the ability to work effectively across a multi-language production stack.
• Hands-on experience building, evaluating, or improving LLM-based applications or agents, including tool use and multi-step workflows.
• Deep expertise in at least one of the following:
• Evaluation infrastructure, benchmark design, and automated grading.
• Agent harnesses, tool interfaces, and reusable skills for complex workflows.
• Realistic test environments or simulations spanning multiple systems, datasets, permissions, and stateful workflows.
• Model selection, routing, and inference efficiency, including context management and caching.
• Strong experimental and statistical judgment, including choosing representative tasks, defining meaningful success criteria, diagnosing failures, and distinguishing real improvements from noise.

☑ Eligibility and how to apply

• If you’re excited about this role, we encourage you to apply even if you don’t meet every qualification.