The insider playbook for sourcing, screening, and hiring the measurement engineers every AI team is now fighting over.

Anthropic pays a "Research Engineer, Model Evaluations" between $500,000 and $850,000 a year - Anthropic careers. That single number, for a role that barely existed as a titled job two years ago, is the clearest signal yet that evaluation has become the scarce skill inside artificial intelligence. The people who can tell you, reliably and fast, whether a model or a prompt change actually made your product better are now among the most contested hires in technology, and roughly 39.6% of AI-first job postings already list evaluation-related skills as a requirement - JobsByCulture.

Here is the problem for anyone trying to hire one. The AI evals engineer hides under a dozen different titles, rarely advertises the skill in a headline, and gets screened by most recruiters as if they were a generic machine learning engineer, which is exactly how the best ones slip through. Evaluation is not model training, it is not data science, and it is not QA bolted on at the end. It is a distinct discipline with its own literature, its own tools, and its own failure modes, and if you cannot recognize the signal you will either miss the real builders or overpay for people who have only ever quoted a leaderboard.

This guide is the deep, practical version of how to fix that. It covers why the role exploded, what an evals engineer actually builds, the four different kinds you might be hiring, what each tier costs, where the talent concentrates, the sourcing tactics that work when LinkedIn keywords fail, how to screen for real skill instead of buzzwords, how to write the job and close the offer, and how AI agents are quietly rewiring the search itself. The audience is recruiters, talent leaders, and hiring managers who need to actually staff this work in 2026, not just read headlines about it.

Written by Yuma Heymans (@yumahey), who built HeroHunt.ai and its autonomous AI Recruiter to source scarce, technical talent from over a billion online profiles. Finding people whose best work lives in a GitHub commit history rather than a job title is exactly the problem this guide is about.

Contents

1. Why the Evals Engineer Became AI's Most Contested Hire
2. What an AI Evals Engineer Actually Does
3. Not One Role: The Four Kinds of Evals Engineer
4. What They Cost: Compensation and the Two Labor Markets
5. Where the Talent Concentrates: Four Rings
6. The Sourcing Playbook: Hunt by the Evals They Ship
7. Screening and Interviewing: Real Skill vs Buzzwords
8. Writing the Job and Closing the Offer
9. How AI Agents Are Rewiring This Search
10. The 2026 to 2027 Outlook: Regulation, Reliability, and Durable Demand

1. Why the Evals Engineer Became AI's Most Contested Hire

The evals engineer became a contested hire because evaluation, not model choice, now decides how fast an AI product can safely ship. Every team has access to roughly the same frontier models through an API, so the durable advantage is no longer which model you call, it is how quickly and confidently you can tell whether a change helped. That is a measurement problem, and the person who owns it sits closer to the roadmap than almost anyone else on the team. Investors have started saying this out loud: the venture argument that "evals are the new moat" reframes evaluation as the differentiator rather than a cost center, because reliable measurement is what lets you iterate faster than a competitor calling the same model.

The clearest single-line version of the thesis comes from OpenAI president Greg Brockman, who wrote that "evals are surprisingly often all you need" to steer a model toward a useful behavior - Greg Brockman. The reason that lands is arithmetic. If you cannot measure quality, every prompt tweak, model swap, and retrieval change is a coin flip, and you ship blind. If you can measure it, each change becomes an experiment with a signal, and you compound improvements. Teams that build robust evaluation report shipping many more model versions per quarter than teams that do not, because they can trust their own release decisions instead of debating them.

The demand pressure is amplified by how badly ungoverned AI is failing in production. MIT's NANDA initiative found that 95% of enterprise generative-AI pilots delivered no measurable profit-and-loss impact, despite tens of billions in spending, and attributed the failures to brittle workflows and weak validation rather than weak models - Forbes. Gartner reaches the same conclusion from the agent side, predicting that over 40% of agentic AI projects will be cancelled by the end of 2027, driven in part by inadequate risk controls - Gartner. Inadequate risk controls is a polite phrase for the eval gap, and it is the reason evaluation talent has moved from nice-to-have to bottleneck.

The hiring market reflects all of this. LinkedIn has ranked "AI Engineer" as its number-one fastest-growing job in the United States, with 639,000 AI-related postings added between 2023 and 2025 - CBS News. The World Economic Forum's Future of Jobs Report 2025 places AI and machine learning specialists among the top three fastest-growing roles worldwide, with 86% of employers expecting AI to transform their business by 2030 - World Economic Forum. Inside that broad wave, evaluation is the sub-skill that keeps getting singled out, because it is the one that separates a demo from a product.

Timing explains why the role crystallized specifically in late 2025 and 2026 rather than earlier. The frontier-model release cadence compressed from roughly annual to a matter of weeks, and every new model or version invalidates yesterday's assumptions about what your product does well. When a better model arrives twice a quarter, the only way to adopt it safely is to have an evaluation suite that tells you, in an afternoon, whether the swap helped or quietly broke something. Teams without that suite either freeze on an old model out of fear or upgrade blindly and ship regressions, and both outcomes are now visibly expensive. That is why applied-AI companies increasingly hire an evals engineer among their first ten technical staff, treating evaluation velocity as the thing that gates product velocity - JobsByCulture.

There is also a simple structural reason the role is permanent rather than a fad, visible in how quickly benchmarks fall. The chart below, from Stanford's 2025 AI Index, shows one-year jumps on the hardest public benchmarks, including SWE-bench solving rates climbing from single digits to the low seventies. Every time models conquer a test, the test stops discriminating, and someone has to build a harder one.

That treadmill is the job. Public benchmarks saturate, in-house evals become the moat, and the person who keeps inventing measurement as models outgrow the old tests is the evals engineer. For a recruiter, the takeaway is that this is not a spike tied to one product cycle but a structural feature of how AI gets built, which makes evals sourcing a capability worth developing rather than a one-off req to fill.

To ground the role before going deep, it helps to hear the two practitioners who did the most to define it. The conversation below, between the creators of the most popular evals course and one of the most-followed product podcasts, is the single best non-technical explainer of what "evals as a skill" means and why teams now hire for it.

As the discussion makes clear, the discipline is data-first and unglamorous: read your model's outputs, categorize how they fail, and build measurement around the real failures. That framing matters for hiring because it tells you what to screen for, which is judgment about data, not memorized benchmark trivia.

2. What an AI Evals Engineer Actually Does

An AI evals engineer turns fuzzy notions of quality into defensible, reproducible metrics, and then builds the infrastructure to run those metrics continuously. Anthropic's own job description states the goal precisely: the role exists to "turn ambiguous notions of intelligence into clear, defensible metrics" and to build and harden a distributed platform so hundreds of evaluations run reliably during training - Anthropic careers. That one sentence captures both halves of the job. Half of it is measurement science, deciding what to measure and how to know the measurement is trustworthy. The other half is engineering, making the measurement run at scale without flaking.

The most useful mental model for what they build comes from practitioner Hamel Husain, whose widely-cited essay argues that unsuccessful AI products almost always share one root cause, a failure to create robust evaluation systems - Hamel Husain. He organizes the work into three levels that a strong evals engineer moves between fluidly. Level 1 is cheap code-based assertions and unit tests that run constantly. Level 2 is human and model-based evaluation, where you log real traces and use a calibrated LLM-as-judge to grade them. Level 3 is A/B testing against real user outcomes. The craft is knowing which level answers which question, and never trusting a higher level without the discipline of the one below it.

The loop that connects those levels is the actual deliverable, and it looks like the diagram below. It starts not with a metric but with reading data, because the failures you have never looked at are the ones your dashboard will never catch.

Notice that the loop begins and ends with production traces. This is the tell that separates the discipline from ordinary testing: an evals engineer treats real model outputs as the source of truth, systematically reads and categorizes the failures, and only then decides what to measure. The academic backbone for the hardest part, deciding when an automated judge is trustworthy, is Shreya Shankar's paper on aligning LLM-assisted evaluation with human preferences, which names the trap of criteria drift, where the very act of grading outputs changes what you decide the criteria should be - arXiv. A good evals engineer knows their judge must itself be validated against humans, or the whole tower rests on sand.

Beyond the core loop, the role spans several concrete responsibilities that recruiters should be able to name, because each one is a different screening signal. Keeping the list short and then unpacking it in prose is the honest way to represent a job that is mostly nuance.

• Benchmark and dataset design - building test sets that catch real failure modes rather than generic ones
• LLM-as-judge calibration - measuring how well an automated grader agrees with human experts
• Red-teaming and safety evals - probing for adversarial, harmful, or policy-violating behavior
• Agent and tool-use evaluation - scoring multi-step reliability, not just single-answer accuracy
• Hallucination and faithfulness metrics - verifying that outputs are grounded in the retrieved source

Each of those has real depth. Agent evaluation, for example, is its own sub-specialty: Sierra's tau-bench scores multi-step agents against a simulated user for end-to-end task completion and policy adherence, and SWE-bench grades coding agents on 2,294 real GitHub issues using each repository's own test suite - SWE-bench. Faithfulness has named, computable metrics too, such as the RAGAS framework's ratio of answer claims actually supported by the retrieved context - arXiv. An evals engineer does not need to have built all of these, but they should be able to reason about which one fits a given product and why a single accuracy number would be misleading.

Red-teaming and safety evaluation deserve a special mention, because they are the fastest-growing slice of the discipline and the part regulators care about most. This is the work of deliberately probing a model for adversarial, harmful, or policy-violating behavior, and at the frontier it is heavily automated, with labs generating attacks at scale and feeding the results into the safety leaderboards that gate model releases. For an applied team the same instinct shows up as prompt-injection tests, jailbreak suites, and checks that an agent refuses out-of-policy requests. A candidate with genuine red-team experience thinks like an attacker by default, and that adversarial mindset is hard to teach and easy to verify by asking them how they would try to break their own eval before trusting it.

What a rigorous, test-grounded eval task actually looks like is worth seeing concretely, and the anatomy of one SWE-bench instance below makes the point better than prose.

The reason this matters for hiring is that the diagram represents craft, not luck. Building a task like that, with a fail-to-pass test, a held-out solution, and a reproducible harness, is precisely the skill you are paying for. The framework name candidates will use for this discipline is evaluation-driven development, popularized by Chip Huyen in her O'Reilly book, which insists you define how you will evaluate an application before you build it, exactly as test-driven development does for ordinary software - huyenchip.com. When a candidate frames evaluation as first-class infrastructure rather than QA-after-the-fact, that is a strong signal you are talking to a real one.

It is worth stressing that this is not exotic frontier-lab work confined to a handful of companies. Documented production practice at mainstream firms like Asana and GitHub shows the same pattern of offline evaluation sets, model-based judges, and online monitoring feeding back into the product - Evidently AI. A large and underrated part of the job is also human annotation operations: recruiting subject-matter experts, writing grading guidelines, resolving disagreements between annotators, and turning that human judgment into the labeled data that both trains judges and validates them. An evals engineer who has run an annotation effort, argued about a rubric, and watched inter-annotator agreement improve as the guidelines sharpened has a maturity that no amount of framework fluency substitutes for, and it is a background worth actively probing for rather than treating as a soft skill.

3. Not One Role: The Four Kinds of Evals Engineer

The most expensive hiring mistake in this space is treating "evals engineer" as one job, because it is at least four, and they require different backgrounds and command different pay. The clearest evidence is that OpenAI does not run one eval role, it runs a whole job family: Research Engineer for Post-Training Evals, Research Engineer for Frontier Evals and Environments, Engineering Manager for the Evals API, and a Backend Software Engineer for Evals - OpenAI careers. Each of those is a genuinely different person, and if you write one generic job description you will attract none of them well.

The first kind is the research-eval engineer, who invents new ways to measure capability and safety and often sits inside a lab's safety or preparedness org. This person reasons about alignment, robustness, and dangerous capabilities, and their work gates model launches. The second kind is the platform-eval engineer, who builds the distributed infrastructure that runs thousands of evaluations reliably during training. Anthropic's Model Evaluations role is explicit that this half is distributed-systems and data-pipeline reliability work, with on-call responsibility, not just statistics - Anthropic careers. Conflating these two is why some searches stall: a brilliant metric designer who cannot keep a pipeline from flaking is not who the platform team needs, and vice versa.

The third and most common kind, especially outside frontier labs, is the product or applied-eval engineer, the person who makes sure a shipped LLM feature actually works. They live in the Hamel Husain three-level world of unit tests, calibrated judges, and A/B tests, and they are what most startups mean when they post the role. The fourth kind is the safety and red-team engineer, who probes for adversarial and harmful behavior. Google DeepMind staffs this heavily, with research-engineer roles building automated red-teaming tools and benchmarks that power Gemini's safety leaderboards - Google DeepMind. Naming which of the four you actually need, before you write a word of the job description, is the single highest-leverage decision in the whole search.

The cost of getting this wrong is concrete. Picture a Series B startup that posts for an "Evals Engineer," interviews on statistics and metric design, and hires a gifted research-eval type who thinks deeply about measuring capability. Six months later the eval pipeline still flakes under load, dashboards go stale during releases, and nobody can run the suite reliably in CI, because what the team actually needed was a platform-eval engineer who treats evaluation as distributed-systems reliability work. The reverse mis-hire is just as common: a strong infrastructure engineer who keeps the pipeline humming but cannot design a test set that catches the failures that matter, so the suite runs beautifully and measures the wrong thing. Both people are excellent. Neither was the right hire, and the miss traces entirely back to skipping the archetype question at the top of the funnel.

It also helps to draw the boundary against adjacent roles that recruiters routinely confuse with this one. The cleanest framing comes from the essay that named the broader category, Shawn Wang's "The Rise of the AI Engineer," which argues that an entire class of AI work that used to require a research team now sits on top of foundation models via APIs - Latent Space. Using that lens, the distinctions become concrete rather than fuzzy.

• ML engineer - trains and operates the models themselves
• Research scientist - designs experiments and pushes the science forward
• Data scientist - analyzes business and product data for insight
• AI or evals engineer - measures how a model or AI product behaves in the real world

The practical consequence of that boundary is that eval skill hides under all of those titles and rarely surfaces in the headline. Plenty of people doing serious evaluation work call themselves "research engineer," "applied scientist," or just "software engineer," and their eval depth is visible only in what they have built. To see the applied version of the job as it is actually practiced in production, the conference talk below walks through building closed-loop evals for a multimodal agent at scale, which is exactly the day-to-day of the product-eval engineer most teams want to hire.

Watching that work in motion also arms you for the interview, because it shows what a strong answer to "how would you evaluate this feature" sounds like: datasets, judges, and closed loops, not a single accuracy figure. Knowing which of the four archetypes you are hiring, and which adjacent title they might be hiding behind, turns a vague search into a targeted one.

4. What They Cost: Compensation and the Two Labor Markets

Compensation for evaluation work splits into two markets that share almost nothing, and quoting a number without saying which market you mean is how offers get blown up. The first market is full-time eval engineering at frontier labs and well-funded startups, paid at the same elite scale as any machine learning or research engineer. The second is contract and expert-network eval work, paid by the hour and an order of magnitude lower. A recruiter who mixes them will either insult a lab-caliber candidate or terrify a hiring manager expecting startup salaries.

At the top of the full-time market, the eval-specific postings confirm the pay is not a discount for a niche skill. Anthropic's Model Evaluations role posts a total range of $500,000 to $850,000, salary plus equity - Anthropic careers. Those figures sit inside broader compensation bands that are public on aggregators, and the bands are eye-watering. OpenAI software-engineer total compensation runs to a median around $875,000 on Levels.fyi, with senior levels well past a million - Levels.fyi. Anthropic's own software-engineer ladder spans from roughly $367,000 at entry to $1.25 million at staff - Levels.fyi. The chart below shows how steep the gap is between the frontier tier and everyone else.

The spread on that chart carries two lessons for a recruiter. First, a data vendor or applied lab like Scale AI pays roughly half the frontier-lab package at an equivalent level, which is normal and worth explaining to candidates rather than apologizing for - Levels.fyi. Second, most of these packages are equity-heavy, so the headline number is not cash. Above mid-level at OpenAI, Anthropic, or Meta, stock is commonly 60% to 70% of the total, which means the quality of the equity matters as much as the size. Meta pays in publicly-traded, immediately-liquid RSUs, while OpenAI uses profit-participation units and Anthropic private equity, and a savvy candidate will weigh that liquidity difference heavily - Levels.fyi.

Outside the frontier bubble, the numbers come back to earth. Independent career-guide bands put a mid-level evals engineer around $230,000 to $340,000 total, senior around $340,000 to $480,000, and staff around $460,000 to $650,000, with frontier labs paying above that ceiling - JobsByCulture. Geography moves the number sharply: European eval pay lags the United States by roughly 35% to 55% at senior levels, so a $250,000 San Francisco package compares to about €100,000 in Amsterdam or £120,000 in London before equity even enters the picture - Remotely Talents. For a hiring manager on a budget, that gap is not a loophole to exploit cynically but a real lever, since a fully-remote senior evaluator in a lower-cost market can be both well-paid locally and affordable to you.

The second market, contract and expert-network eval work, is where a lot of the actual evaluation labor now happens, and the rates are hourly. This is the supply chain that authors benchmarks and grades model outputs for the labs, and the platforms advertise their ceilings openly. The rates cluster by credential, and the chart below shows the representative advertised ceilings.

Those bars are real advertised numbers, and each has a source a recruiter can check. Scale AI's Outlier advertises up to $150 an hour for machine-learning PhDs training and evaluating models - Outlier. Surge AI pays contract engineers $100 to $150 and up per hour to build coding evals and benchmarks - Surge AI. Mercor publishes an expert tier of $75 to $200 and up, with company-wide average pay reportedly passing $100 an hour as it shifted toward credentialed experts - Mercor. The practical use of this second market for a recruiter is twofold: it is a talent pool of people already doing eval work, and it is a benchmark for what expert judgment costs, which helps you sanity-check both contractor budgets and the aspirations of full-time candidates coming from that world.

Two nuances tend to decide real offers. The first is that the very top of the full-time ceiling is higher than even the eval-specific postings suggest, because many frontier eval roles are titled "research scientist," and that ladder tops out around $1.47 million at senior levels on the aggregators - Levels.fyi. When a candidate has a competing research-scientist offer, you are competing with that ceiling, not the software-engineer one. The second nuance is the pipeline between the two markets: a growing number of full-time evaluators started as contractors on Outlier, Surge, or Mercor, built a portfolio grading real model outputs, and converted that into a salaried role. Treating a strong contractor's hourly history as evidence rather than as a lesser credential is a cheap edge, because most recruiters overlook that pool entirely, and the people in it have often read more real model failures than a typical full-time applicant.

5. Where the Talent Concentrates: Four Rings

Evals talent concentrates in four concentric rings, and each ring has a specific, learnable sourcing handle, which means you can build a target list without a machine learning background. The rings run from the highest-prestige, hardest-to-poach core outward to the broadest, most accessible pool. Picturing them as a map, as in the diagram below, keeps a search organized and stops you from fishing in only the one pond you already know.

Ring one is the frontier labs, where named eval orgs publish and hire openly. OpenAI's Preparedness team builds the frontier-risk evals that gate launches, with leadership comp reaching up to $555,000 - OpenAI. Anthropic runs a Frontier Red Team and an Alignment Stress-Testing team, and in August 2025 it acqui-hired the entire leadership of the eval-tooling company Humanloop, a reminder that labs will buy whole eval teams wholesale - Anthropic. Meta Superintelligence Labs, formed in 2025 under Alexandr Wang, runs a dedicated Evaluations team that was seeded partly by poaching Scale AI's SEAL research group - Built In. These people are the hardest to move, but they are also the most visible, because their names are on the papers.

Ring two, the eval-tooling startups, is the richest hunting ground for most recruiters, because their customer case studies and open-source repositories double as talent maps. Braintrust, an eval-first observability platform that has raised around $121 million, publicly describes how Notion aligned 70 engineers on evals inside its product, and it hires an explicit "Eval Engineer" role you can use as a job-description template - Braintrust. The ring is crowded and well-funded, which is good news for a hunter because it means turnover and options.

• Braintrust - eval-first observability, customers include Notion, Stripe, and Vercel
• Arize AI - observability plus evaluation, its open-source Phoenix library exceeds 2M monthly downloads - Arize
• LangChain (LangSmith) - testing and eval tooling, raised a $125M round at a $1.25B valuation
• Patronus AI - automated eval models, founded by ex-Meta researchers
• Confident AI (DeepEval) - "Pytest for LLMs," a Y Combinator company with heavy open-source usage

The reason this ring rewards attention is that everyone in it is, by definition, an evals specialist who chose to make evaluation their whole career, which is a stronger prior than an eval-curious generalist at a big company. It also spans the price and prestige spectrum, so you can find both the researcher who ships eval foundation models at Galileo and the pragmatic builder who wrote a metric for DeepEval.

The funding signals are worth reading because they tell you where the teams are growing and, therefore, hiring. Galileo raised a Series B and ships proprietary evaluation foundation models, so its team skews toward eval-model researchers - Galileo. Patronus AI was founded by ex-Meta machine learning researchers and builds automated eval and guardrail models, which makes its founder network a specific alumni pool to mine - Patronus AI. And the open-source-first companies are the most accessible of all: Confident AI's DeepEval runs hundreds of thousands of downloads a month with real enterprise usage in continuous-integration pipelines, so anyone who contributed a metric there has demonstrably shipped evaluation into production - Confident AI. A recruiter who tracks these rounds knows which door to knock on this quarter rather than last year's.

Ring three is the expert-data vendors, the companies that employ and aggregate evaluation labor at massive scale, and they are worth knowing both as employers to poach from and as proof of what expert judgment costs. Scale AI's SEAL research team published 15 new benchmarks and more than 450 evaluations across dozens of models in 2025, which makes SEAL a marquee credential line on any resume - Scale AI. Surge AI reached an estimated $1.2 billion in revenue with only around 130 full-time staff plus roughly 50,000 expert contractors serving about a dozen frontier labs - Sacra, while Mercor raised a $350 million round at a $10 billion valuation on the back of 30,000-plus vetted experts - Mercor. The small full-time cores at these vendors, the data scientists and quality engineers who design the pipelines, are a concentrated and often-overlooked target, and the brain drain when a vendor loses a research team, as Scale did to Meta, puts unusually strong people in motion.

Ring four, open source and academia, is the highest-signal and most accessible pool of all, because both the work and the workers are public. Beyond the framework repositories, a cluster of research institutions concentrates the most credentialed evaluators: the UK AI Security Institute behind Inspect, Stanford's CRFM behind HELM, and LMArena, which spun out of UC Berkeley's LMSYS group and raised a $100 million seed to run crowdsourced human-preference evaluation over hundreds of thousands of pairwise votes - LMArena. Feeder programs are the leading indicator worth tracking, because their alumni are eval-native from the first day. METR builds task-horizon autonomy evals, Apollo Research specializes in deception and scheming evaluations and famously surfaced in-context scheming in frontier models - Apollo Research, and Epoch AI built the FrontierMath benchmark with more than 60 mathematicians - Epoch AI. Another accessible ring-four handle is the Hugging Face evaluation team, whose public LLM Evaluation Guidebook, written by the people who run the Open LLM Leaderboard, doubles as both a competency standard and a way to identify who in the community is genuinely eval-focused - GitHub.

Working all four rings, rather than only the LinkedIn-searchable middle, is what separates a full search from a shallow one. Most recruiters camp in ring two because it is the one that shows up in a normal candidate database, but the deepest and cheapest talent, the framework maintainer in ring four and the vendor quality engineer in ring three, is precisely the talent a keyword search never surfaces.

6. The Sourcing Playbook: Hunt by the Evals They Ship

The highest-signal way to source evals engineers is to hunt by the evals they have actually shipped, which means going to the open-source repositories, benchmark papers, and leaderboards where the work is public and every contributor is a named human. This inverts the usual keyword search. Instead of hoping someone wrote "LLM evaluation" in their headline, you start from a repository you know demands real skill and read the list of people who built it. A recruiter needs almost no machine learning background to do this, because GitHub contributor pages, arXiv author lists, and leaderboard credits all resolve to profiles you can then match to LinkedIn or X.

Start with the framework builders, the deepest tier of talent. EleutherAI's lm-evaluation-harness is the de-facto standard evaluation framework and the backend that powers Hugging Face's Open LLM Leaderboard, and its contributor graph is a ready-made shortlist - GitHub. The mechanics are simple: open the repository's contributors page, and each name links to a profile that usually lists a company, a location, and often a personal site. A merged pull request is a far stronger signal than a star, because contributing requires writing a working evaluation, not just bookmarking one. The same move works on the UK AI Security Institute's Inspect framework, whose contributors are effectively government-grade safety engineers - GitHub.

For the product and application layer, mine the repositories that ship real LLM features, because their contributors are the applied evals engineer most startups actually want. These frameworks are where the day-to-day skills live, and one of them, promptfoo, produces the kind of side-by-side evaluation matrix shown below, which is exactly the artifact a strong applied candidate will have built before.

The application-layer repositories worth mining share a trait: contributing to them proves you can build eval pipelines, not just talk about them. Keep the list tight and let the contributor graphs do the work.

• promptfoo - evaluation and red-teaming, used inside OpenAI and Anthropic
• ragas - retrieval-augmented-generation evaluation, widely adopted for faithfulness scoring
• deepeval - 30-plus metrics, the "Pytest for LLMs"
• openai/evals - anyone who merged an eval into the registry designed a real one
• huggingface/lighteval - the Hugging Face team's own harness, 1,000-plus tasks

Once you have candidate names, benchmark papers on arXiv extend the pool into the research-grade tier, because every benchmark publishes a full author list of people who designed a real evaluation. The workflow is to open the abstract page, copy each author, and use Google Scholar to see their other eval work and current affiliation. Papers like MMLU, GPQA, and tau-bench each name a tight cluster of specialists, and first and last authors are the highest-signal targets. Leaderboards do the same job for teams rather than individuals: entries on the SWE-bench leaderboard reveal which companies are building agent evals in production - SWE-bench.

When you do move to LinkedIn, anchor your boolean strings on the tooling rather than the job title, because the tooling is what real practitioners name. A high-precision string is ("lm-evaluation-harness" OR "inspect_ai" OR promptfoo OR ragas OR deepeval) AND (engineer OR contributor), and a broader one is ("LLM evaluation" OR "model evaluation" OR "LLM evals") AND (engineer OR scientist OR researcher) - LinkedIn. Swap the title filter between Engineer, Research Engineer, and Applied Scientist on every pass, because, as the previous chapter established, eval work hides under all three. GitHub's own advanced search closes the loop, letting you find people who wrote eval code but never put the word in their headline - GitHub.

Finally, warm up candidates where they already gather rather than cold-messaging them in research spaces. The practitioner hubs are the AI Engineer conference series, whose talks are free on YouTube and whose speakers self-select as experts - AI Engineer, the MLOps Community with its 90,000-plus members and LLMOps channels, and the EleutherAI Discord where the harness maintainers answer questions in public - EleutherAI. A short, well-curated follow list on X, including Clementine Fourrier, Hamel Husain, Nathan Lambert, and the LMArena team, surfaces the wider community through their reply threads. The discipline of sourcing this way is that you are always verifying skill through artifacts, a commit, a paper, a talk, before you ever evaluate a resume, which is the opposite of keyword-matching and far more reliable for a role this easy to fake on paper.

Before you spend outreach on a name, a two-minute artifact check tells you whether the signal is real. Four things separate a genuine evals engineer from a generalist who once ran a benchmark: merged pull requests to an eval repository, authorship or acknowledgment on a benchmark paper, public writing that explains an eval system rather than just quoting scores, and hands-on annotation or data-labeling experience. Any one of these is a green flag worth an outreach; two or more is a strong candidate you should prioritize. The red flag at the sourcing stage mirrors the one at the interview stage, a profile that only says "trained models" or "improved accuracy" with no visible measurement work, because for a role this easy to claim on paper the absence of artifacts is itself information. Verifying through the work first is also what lets you personalize the outreach, which is the single biggest lever on reply rate for senior technical candidates.

To make this concrete, here is a single sourcing chain run end to end. You open the contributor graph of a product-eval framework like deepeval and pick someone with several merged pull requests touching the metrics folder. Their GitHub profile links a personal site, which mentions a talk they gave at a local meetup on LLM-as-judge calibration, and their bio lists a current employer where they are titled "Software Engineer," a title that would never have surfaced in an evals keyword search. You cross-reference to their LinkedIn to confirm tenure and location, glance at whether they have authored or been acknowledged on a benchmark paper, and only then reach out, referencing the specific metric they contributed rather than a generic pitch. That message lands because it proves you actually looked at their work, and it is the difference between a 5% and a 40% reply rate on a candidate this senior. Kaggle rounds out the funnel for a different profile, the competition and notebook builders who demonstrate applied benchmarking skill in public, searchable by LLM-evaluation tags and ranked by leaderboard standing - Kaggle.

7. Screening and Interviewing: Real Skill vs Buzzwords

Screening for evals skill comes down to one distinction: does the candidate reason from data, or from benchmarks and buzzwords? The strongest single signal, endorsed by essentially every serious practitioner, is disciplined error analysis, the habit of reading real model outputs and categorizing how they fail before choosing any metric. Hamel Husain calls it the most important activity in evals and warns that skipping it leaves everything else built on sand - Hamel Husain. So the first and best interview probe is disarmingly simple: ask the candidate to walk you through the last time they read their own model's outputs. How many did they read? How did they categorize them? A strong candidate names a failure taxonomy and a stopping heuristic. A weak one talks only about aggregate dashboards.

The second discriminator is whether they can reason about LLM-as-judge calibration, because using a model to grade a model is now standard, and doing it naively is a common failure. The reference paper here, on judging LLM-as-a-judge with MT-Bench and Chatbot Arena, documents that automated judges carry systematic biases, position bias, verbosity bias, and self-enhancement bias, even as a strong judge reaches around 80% agreement with humans - arXiv. A sharp interview question is to describe using one model to grade two others and ask the candidate to name three ways that grading is biased and how they would control each. The right answers, randomizing answer order, penalizing length, and never letting a model judge its own family, come quickly to someone who has actually done it. That work is grounded in human-preference evaluation, the crowdsourced blind-voting method pioneered by Chatbot Arena, whose interface is shown below.

A candidate who understands that image understands why raw agreement can lie. If a human labels 88% of items the same class, a judge that agrees 90% of the time may be barely better than chance, which is why the field measures chance-corrected agreement with statistics like Cohen's kappa rather than raw accuracy - Galileo. Statistical literacy in general is a hard filter. Anthropic's own engineer argues in "Adding Error Bars to Evals" that a score without a confidence interval is meaningless, and that comparisons should use paired differences to shrink the standard error - arXiv. A good scenario question is to state that model A scored 71% and model B scored 69% on a 200-item eval and ask whether A is better. The rigorous answer computes a paired standard error and asks about the eval's size and power, rather than declaring a winner.

The calibration discussion also reveals whether a candidate has actually managed human labelers, which is where a lot of eval quality is won or lost. A practical tell is how they handle disagreement between annotators. A mature answer often invokes a single trusted domain expert as the "benevolent dictator" whose judgment is treated as ground truth, precisely to avoid endless committee arguments, and describes running alignment sessions where the expert and the automated judge are compared, the rubric is sharpened, and agreement is re-measured until it holds steady on a held-out set. That iterative, human-in-the-loop rubric refinement is the core of the "who validates the validators" problem, and a candidate who has lived it will describe it in terms of process and disagreement resolution, not just a metric. Someone who has only ever consumed a pre-labeled dataset will not have this texture, and the gap shows immediately.

The third area separates people who can own a benchmark from people who should not, and it is data contamination. If a model saw your test set during training, your benchmark measures memorization, not capability, and studies have found contamination signs on a meaningful fraction of popular benchmarks, with cleaning them dropping some models by double-digit points - arXiv. Ask the candidate how they would detect that a model has seen their benchmark without access to its training data. A strong answer reaches for statistical tests of memorization or membership inference, and for mitigations like held-out splits, freshly-collected data, and canary strings. Related is Goodhart's law awareness: a candidate who celebrates a 98% pass rate without suspicion is a yellow flag, because a saturated eval has usually stopped discriminating, exactly as MMLU did once frontier models passed 86%.

A few domain-specific questions discriminate skill better than any generic one. Ask how they would evaluate a summarizer, and a strong candidate decomposes quality into faithfulness, coverage, and coherence, then proposes a question-answer-generation check rather than reaching for BLEU or ROUGE, which correlate poorly with human judgment. Ask how they would decide whether a customer-support agent is reliable enough to ship, and a strong answer covers repeated-trial variance, not a single pass on clean inputs, because pass at one attempt gives dangerously optimistic estimates and production practice uses k-trial pass rates, perturbation robustness, and trace-level checks of each tool call - Confident AI. Ask how they would build a 200-example golden set, and a senior answer targets specific failure modes and reserves a held-out split so the judge cannot be tuned on the eval set, rather than vaguely gathering "hard examples." One more fast phone-screen filter: ask why binary pass-fail often beats a one-to-five Likert score. The correct answer is that a Likert scale hides uncertainty and clusters at the middle, which is the kind of judgment that only comes from having actually graded outputs at scale.

The best assessment overall is a scoped take-home or an existing portfolio, not a whiteboard puzzle. A well-validated task is to pick a narrow product, build around 50 examples with known correct and incorrect outputs, implement dual grading with both code checks and an LLM judge, run it across several models, and write up which won, how each failed, and how well the judge agreed with human labels. Arize's "Recipe Bot" course homework is a ready-made template for exactly this - Arize. To summarize the signals into a usable rubric, the green and red flags cluster cleanly.

• Green flag - names a failure taxonomy from reading real traces
• Green flag - measures judge-human agreement with kappa, not raw accuracy
• Green flag - reports confidence intervals and reasons about contamination
• Red flag - quotes leaderboard scores but has never built an eval set
• Red flag - trusts a single automated metric and a 100% pass rate

The practical way to apply that rubric is a short loop: a data-processing coding exercise rather than an algorithms puzzle, a rubric-critique where you hand them a flawed grading prompt and ask what is wrong with it, and a portfolio walkthrough where you ask them to name the one metric that turned out to be misleading. A candidate fluent in the modern curriculum, error analysis, then failure taxonomy, then code and judge evaluators, then A/B testing, has the data-first mental model you want, and that curriculum is now codified in Hamel Husain and Shreya Shankar's Maven course, which has trained thousands of engineers and product managers including teams at OpenAI and Anthropic - Maven. Screening against that standard, rather than against a generic ML checklist, is what keeps you from hiring a benchmark-quoter for a job that is really about judgment.

8. Writing the Job and Closing the Offer

The job description is where most evals searches quietly fail, because a generic "machine learning engineer" post attracts generalists and repels the specialists you actually want. The fix is to write for the specific archetype from chapter three and to signal, in the first two sentences, that you understand evaluation is a discipline. Frontier-lab postings are the best templates precisely because they were written by people who do the work. Anthropic's Model Evaluations description leads with turning ambiguous notions of intelligence into defensible metrics and building a distributed eval platform, and it lists the real requirements: strong production Python, distributed-systems reliability, and the communication skill to translate results for non-specialists - Anthropic careers. You can lift that structure directly, then swap in your own product and domain.

Two things belong in the requirements that generic templates omit, and including them acts as a self-selecting filter. The first is a named eval framework, because asking for hands-on experience with Inspect, promptfoo, Braintrust, or LangSmith immediately separates practitioners from the eval-curious. The second is statistical literacy, phrased in plain terms like "comfortable reasoning about confidence intervals and A/B test power," because it screens out anyone who thinks evaluation is just writing prompts. Crucially, avoid demanding a leetcode gauntlet, since the real skill is data-wrangling Python with pandas and async concurrency, not algorithm puzzles, and an algorithm-heavy loop will drive away exactly the applied evaluators you need.

Title strategy deserves as much thought as the description, because, as established earlier, eval talent hides under many labels. Posting only "Evals Engineer" will miss the applied scientist and the research engineer who do the same work under a different name, so run the search under several titles in parallel and let the job description, not the title, carry the specificity. It also pays to be explicit about which of the four archetypes the role is, since a research-eval candidate and a platform-eval candidate read the same vague post and both correctly conclude it is not for them. Ambiguity here does not widen your funnel, it empties it.

The interview loop should mirror the job rather than a generic engineering gauntlet, and designing it well is itself a recruiting advantage because strong evaluators notice. A loop that fits the role is a data-processing coding exercise instead of an algorithms puzzle, a rubric-critique where you hand the candidate a flawed grading prompt and ask what is wrong with it, a short statistics scenario about reading an A/B result, and a portfolio walkthrough of an eval they actually built. Each stage tests something the job requires, so candidates leave feeling the process respected their expertise, which matters when they have competing offers. Keeping the loop tight also serves the second closing lever below, because the same market speed that makes evaluation valuable makes candidates disappear if your process drags. A four-stage loop that concludes within two weeks routinely beats a more prestigious employer whose process runs a month.

Closing the offer for this role turns on three levers beyond money, and the good news is that the non-cash levers are often decisive for the best candidates. Mission and access matter enormously: evaluators want to work on models and problems that matter, with real compute and real data, and a small team offering genuine ownership of the eval stack can beat a bigger name offering a slice of someone else's. Speed is the second lever, because this market moves fast and a two-week-faster process routinely wins a candidate outright. The third is comp realism, which means being honest that a non-frontier company cannot match an $850,000 Anthropic package and instead competing on equity upside, autonomy, and scope. The framing that works is to sell the role a strong evaluator already believes in: that measurement is the moat, and that here they will own it rather than serve it.

A concrete version of that pitch wins more often than a bigger number. Imagine a fifty-person AI company competing against a frontier lab for the same applied-eval engineer. It cannot match the lab's equity, so it does not try. Instead it offers three things the lab cannot: the candidate will personally own the entire eval stack rather than a corner of it, they will report to a founder who treats evaluation as a core strategy rather than a compliance checkbox, and they can ship a new eval to production the week they think of it rather than after a quarter of committee review. For an evaluator who is motivated by seeing their measurement actually change what the company builds, that ownership and speed frequently outweigh a larger but more constrained package. The mistake is to compete only on the axis where you are weakest, comp, instead of the axes where a smaller, faster team is genuinely stronger.

9. How AI Agents Are Rewiring This Search

AI agents are rewiring how you run this search, and the irony is delicious: the people you are hunting are the very people who build the systems that evaluate AI agents. Evaluation is what tells you whether an autonomous system is reliable enough to trust, which is why agent-reliability benchmarks like tau-bench exist, and why Gartner's warning that 40% of agentic projects will be cancelled traces back to the eval gap. That same discipline applies to the AI tools now entering recruiting itself. If you are going to let an agent source and screen candidates, you should evaluate that agent the way an evals engineer would evaluate any other, with real traces, a failure taxonomy, and a calibrated judgment about where it is trustworthy.

For a scarce, technical role like this one, agentic sourcing has a specific advantage over keyword search, and it maps directly onto chapter six. The best evals engineers are discoverable through signals that live outside a LinkedIn text field: a commit history on lm-evaluation-harness, authorship on a benchmark paper, a talk at an AI Engineer conference. A tool that can reason over those signals, rather than only matching strings in a headline, is far better suited to this search than a traditional keyword filter that would rank a self-described "AI enthusiast" above a silent contributor to Inspect. Autonomous AI recruiters are built for exactly that kind of multi-signal, cross-platform search, which is why they help most on precisely the roles where LinkedIn keywords fail.

The broader tool landscape for AI-assisted sourcing has matured, and it helps to know the categories rather than chase logos. Incumbent aggregators like hireEZ and SeekOut built large unified candidate databases with AI ranking on top, natural-language search tools like Juicebox let you describe a candidate in plain English rather than boolean, and LinkedIn Recruiter remains the incumbent everyone still pays for. Each is useful, and none of them, on its own, will surface the silent open-source contributor, which is why the durable edge for this role is combining an AI sourcing tool for reach with the artifact-based hunting that no database indexes well.

There is a neat way to turn this into an advantage: evaluate your own recruiting agent the way your candidate would evaluate any AI system. Read a sample of the shortlists it produces, do error analysis on where it went wrong, and categorize the failures, false positives who looked good on paper but had no real eval work, and false negatives it ranked low because their signal lived in a commit rather than a headline. Calibrate its judgments against a trusted human recruiter the way you would calibrate an LLM judge against a domain expert, and watch for the same biases, an agent that over-weights big-company logos or fashionable keywords is exhibiting exactly the kind of metric-gaming an evals engineer is trained to catch. Doing this does two things at once: it makes your sourcing measurably better, and it gives you fluency in the candidate's own language, which is a quiet but real edge when you reach out.

The deeper point is that recruiting is becoming an eval problem too. When you deploy an agent to screen candidates, you inherit every failure mode an evals engineer studies: bias in the judge, gaming of the metric, and silent quality regressions you will never see without reading the traces. The teams that will source this talent best are the ones that treat their own hiring stack the way the candidate would, measuring it, calibrating it, and staying honest about where the automation helps and where a human still has to read the data. That is not a reason to avoid AI in the search. It is a reason to bring an evaluator's mindset to it.

10. The 2026 to 2027 Outlook: Regulation, Reliability, and Durable Demand

The outlook for evals hiring is durable demand rather than a bubble, and three forces guarantee it: regulation, agent reliability, and a benchmark arms race that never ends. Regulation is the hardest forcing function, because it turns evaluation from a best practice into a legal requirement. The EU AI Act requires high-risk AI systems to complete conformity assessment, technical documentation, logging, and human oversight, with the standalone high-risk obligations now landing on 2 December 2027 after a late-2025 deferral - European Commission. Hiring and employment AI is explicitly named high-risk, which means anyone deploying AI in recruiting will themselves need the evaluation discipline this guide describes.

Regulation is not only European. In the United States, the NIST AI Risk Management Framework institutionalizes a "Measure" function, testing and verifying AI risks across the lifecycle, as a governance baseline that enterprises and procurement teams now cite by default - NIST. Internationally, ISO/IEC 42001, the first AI management system standard, requires ongoing risk assessment and testing, and it is already certifiable, with AWS, Anthropic, and Microsoft among the early holders - ISO. Certifiable standards create durable, auditable demand for people who can actually run AI evaluations and impact assessments, because an auditor will eventually ask to see the evidence.

The second force is the agent-reliability wall, which is why the field keeps inventing harder tests. The past year produced a dense run of high-profile evaluations that made the discipline visible: OpenAI's GDPval measures models on 1,320 real economically-valuable tasks across 44 occupations - arXiv, the ARC Prize put up a $1,000,000 award on a reasoning benchmark where the top team still scored only around 24% - ARC Prize, and Epoch AI's FrontierMath was so hard that leading models initially solved under 2% - Epoch AI. Each of these exists because the previous generation of benchmarks saturated, and each required eval engineers and domain experts to build. The commercial tailwind is quantifiable: the AI evaluation tools market is projected to grow from roughly $1.6 billion in 2026 to $8.7 billion by 2033 - Persistence Market Research.

Where the discipline goes next is toward evaluating ever more autonomous systems, and the frontier of that work is worth hearing from the people building it. The talk below, from a leading observability company, traces the shift from LLM-as-a-judge toward agent-as-a-judge, which is the emerging pattern for scoring multi-step systems that are too complex for a single graded answer.

That trajectory, from grading single answers to grading whole autonomous workflows, is why the role only gets deeper. As models take on longer and more consequential tasks, the measurement problem gets harder, not easier, and the people who can define what "good" means and prove a system meets it become more valuable, not less.

The most important structural prediction for a recruiter is that evaluation is unbundling into its own career track rather than remaining a task every ML engineer does part-time. The pattern that produced dedicated site-reliability engineering out of general operations is repeating here: as the discipline accumulates its own literature, tools, and standards, companies stop expecting a generalist to cover it and start hiring for it explicitly. Certifiable standards accelerate that split, because an ISO 42001 audit or an EU conformity assessment is not something a team improvises, it is evidence someone has to own and produce on a schedule. The knock-on effect is a widening "verifier" economy, where the scarce and well-paid work is not generating outputs, which models increasingly do, but verifying them, which still requires human judgment about what correct even means in a given domain. For talent leaders, the practical implication is to start building the sourcing capability now, while the title is still ambiguous and the best people are still findable through their work rather than through a saturated keyword.

For a recruiter, the whole guide reduces to a short decision framework. First, decide which of the four archetypes you actually need, because a research-eval and a platform-eval hire are different searches. Second, source by artifacts, not keywords, working all four talent rings and reading contributor graphs and author lists rather than only LinkedIn headlines. Third, screen for data-first judgment, error analysis, judge calibration, statistics, and contamination reasoning, not benchmark trivia. Fourth, set comp by market tier, and never quote a frontier number to a startup budget or a contractor rate to a full-time candidate. Do those four things and you will out-hire teams that treat evaluation as a generic ML req.

The uncomfortable truth underneath all of it is that evaluation talent is scarce because the skill is genuinely hard and genuinely new, and no sourcing tool alone will conjure it. What tools can do, from AI recruiters like HeroHunt.ai that scale reach across a billion profiles to the artifact-based hunting this guide describes, is make sure you actually see the few people who have it before a competitor does. In a market where measurement is the moat, the recruiters who learn to recognize a real evaluator will build the teams that ship AI worth trusting.

This guide reflects the AI evaluation and hiring landscape as of August 2026. Compensation figures, job postings, funding rounds, and regulatory deadlines change frequently, so verify current details before making decisions.