Product Designer, Evals & Prompts

About the role

Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. We want AI to be safe and beneficial for our users and for society as a whole. Our team is a quickly growing group of committed researchers, engineers, policy experts, and business leaders working together to build beneficial AI systems.

What you’ll do

• Write and revise the prompts behind Claude's tools, features, and behaviors on a product surface; test the surface, turn findings into prompt fixes, ship them, and confirm the prompt users get is the one intended
• Build the graders that prove a prompt fix and rerun on the next model; turn designers' hand-run rubrics into automated evals, then read transcripts for what the eval missed
• Build visual, low-code eval tools designers can use without an engineer: assemble a comparison set from real transcripts, turn a plain-English rubric into a grader, compare prompt variants across models side by side, and read results in the tool rather than a notebook
• Watch designers use those tools and make them simpler
• Support model releases: test each surface against the new model, write prompt fixes and migrations, and write prompts for features launching with it, so the surface owner's call has numbers behind it
• Stand up and scale the eval harness: build the test environment that exercises our 50 to 100 tools with trustworthy settings, keep evals green across models, and call whether a regression is the harness or the model
• Package what prompting can't fix for training, with the eval attached: graders for crisp behaviors, human-feedback questions and good/bad pairs for fuzzy ones like writing quality

What they’re looking for

Select a requirement to inspect fit, evidence or application context.

• Production-quality Python
• Experience building and maintaining evaluation pipelines for LLM products: graders, rubrics, comparison sets, regression suites, and the plumbing that runs them across models
• Experience building internal tools with a real interface for people who do not write code
• Experience standing up test harnesses, sandboxing tool calls, and pinning the settings that make runs comparable
• Experience shipping prompts, or working closely with people who do, and understanding why a prompt that works on one model fails on the next
• Reads transcripts, not only scores
• A field relevant to the role as demonstrated through coursework, training, or professional experience

Helpful, not always essential

• Has worked inside a model-launch cycle
• A/B testing experience and the ability to connect offline evals to online outcomes
• Front-end or notebook-to-app experience, and opinions about what makes an eval result legible at a glance
• Has turned product rubrics into training signal: graders, human-feedback questions, or preference pairs
• Cares how Claude behaves for the people using it, not only whether the metric moved
• The annual compensation range for this role is listed below.
• For sales roles, the range provided is the role’s On Target Earnings ("OTE") range, meaning that the range includes both the sales commissions/sales bonuses target and annual base salary for the role.
• Annual Salary:
• $305,000—$385,000 USD