AI is outpacing human engineers in technical evaluations—but we’re fighting back. As AI models like Claude Opus 4.5 begin to match or even surpass human performance in complex coding challenges, the traditional methods of assessing technical talent are crumbling. But here’s where it gets controversial: should we ban AI assistance in evaluations, or embrace it as a tool that humans must learn to wield effectively? And this is the part most people miss: the real challenge isn’t just about designing harder tests—it’s about redefining what it means to be a skilled engineer in the age of AI.
Written by Tristan Hume, a lead on Anthropic's performance optimization team, this article dives into the evolving struggle to create AI-resistant technical evaluations. Tristan has been at the forefront of designing—and redesigning—the take-home test that has helped Anthropic hire dozens of top-tier performance engineers. But as AI capabilities grow, what once distinguished the best human candidates is now being matched by machines. The stakes are high: if we can’t find a way to measure human ingenuity alongside AI, we risk losing the very essence of what makes engineers invaluable.
Since early 2024, Anthropic’s performance engineering team has relied on a take-home test where candidates optimize code for a simulated accelerator. Over 1,000 candidates have tackled it, and many now form the backbone of Anthropic’s engineering teams, including those who brought the Trainium cluster to life and shipped models like Claude 3 Opus. But with each new Claude model, the test has had to evolve. When Claude Opus 4.5 matched the performance of even the strongest human candidates, it became clear: the old methods weren’t enough.
Tristan has iterated through three versions of the test, each time learning what makes an evaluation robust to AI assistance—and what doesn’t. This post explores the original design, how Claude models systematically defeated it, and the increasingly unconventional approaches taken to stay ahead. While the work has evolved alongside AI, the need for exceptional engineers remains—but finding them requires ever more creative strategies.
To that end, Anthropic is releasing the original take-home as an open challenge. With unlimited time, the best human performance still surpasses what Claude can achieve. If you can outperform Opus 4.5, Anthropic wants to hear from you—details are at the bottom of this post.
The Birth of the Take-Home Test
In November 2023, as Anthropic prepared to train and launch Claude Opus 3, the company faced a talent crunch. With new TPU and GPU clusters and a massive Trainium cluster on the horizon, the need for performance engineers was greater than ever. Tristan took to Twitter, calling for candidates, and was overwhelmed by the response. But evaluating so many promising candidates through the standard interview pipeline was impractical. A new solution was needed.
Tristan spent two weeks designing a take-home test that would capture the essence of the role and identify the most capable applicants. The goals were clear: create something engaging, representative of real work, and capable of evaluating technical skills with high resolution.
Design Goals and Challenges
Take-home tests often suffer from a bad reputation, filled with generic, uninspiring problems. Tristan aimed higher: the test had to be genuinely engaging, offering candidates a taste of the job while allowing them to showcase their skills. The format also had advantages over live interviews:
- Longer time horizon: Engineers rarely face deadlines of less than an hour. A 4-hour window (later reduced to 2 hours) better reflected the job’s demands, though it had to balance depth with practicality.
- Realistic environment: Candidates worked in their own editors, free from distractions or the pressure of narration.
- Time for comprehension and tooling: Performance optimization requires understanding complex systems and building debugging tools—hard to evaluate in a 50-minute interview.
- Compatibility with AI assistance: Anthropic encourages candidates to use AI tools, as they would on the job, while still demonstrating their own skills.
Beyond these, the test adhered to core principles: it had to be representative of real work, offer high signal, avoid narrow domain knowledge, and be fun. Fast development loops, interesting problems, and room for creativity were key.
The Simulated Machine
Tristan built a Python simulator for a fake accelerator resembling TPUs. Candidates optimized code running on this machine, using a hot-reloading Perfetto trace to visualize every instruction—similar to tools used on Trainium. The machine featured manually managed scratchpad memory, VLIW, SIMD, and multicore capabilities, making optimization both challenging and rewarding.
The task was a parallel tree traversal, chosen to avoid deep learning specifics, as most performance engineers hadn’t worked in that domain yet. Inspired by branchless SIMD decision tree inference, the problem was both classical and novel, with only a few candidates having prior experience.
Candidates started with a fully serial implementation, gradually exploiting the machine’s parallelism. The warmup involved multicore parallelism, followed by SIMD vectorization or VLIW instruction packing. The original version also included a bug, testing candidates’ ability to build debugging tools.
Early Success and AI’s Rise
The initial take-home was a hit. One candidate from the Twitter batch scored significantly higher than the rest and quickly proved their worth by optimizing kernels and solving a launch-blocking compiler bug. Over the next year and a half, about 1,000 candidates completed the test, helping Anthropic hire most of its current performance engineering team. It was particularly valuable for candidates with limited experience on paper but strong practical skills.
Feedback was overwhelmingly positive. Many candidates worked past the time limit, engrossed in the challenge. The strongest submissions included full optimizing mini-compilers and clever optimizations Tristan hadn’t anticipated.
But then Claude Opus 4 arrived. By May 2025, Claude 3.7 Sonnet was already outperforming many candidates. When tested on the take-home, Claude Opus 4 produced a more optimized solution than almost all humans within the 4-hour limit. Tristan had faced this before—in 2023, Claude models had defeated live interview questions he’d designed. But the take-home required a different fix.
Adapting to AI’s Advances
Tristan used Claude Opus 4 to identify where the problem became too challenging, making that the new starting point for version 2. He added new machine features, removed multicore (which Claude had solved), and shortened the time limit to 2 hours for practical scheduling. Version 2 emphasized clever optimization insights over debugging and code volume, serving well—for a time.
But Claude Opus 4.5 defeated that too. Within 2 hours, it solved initial bottlenecks, implemented micro-optimizations, and matched the best human performance. Tristan faced a dilemma: the best strategy on the take-home might soon be delegating to Claude Code.
Exploring New Frontiers
Some suggested banning AI assistance, but Tristan resisted. He believed humans should be able to distinguish themselves even with AI tools, as they would on the job. Others proposed raising the bar to outperform Claude, but this risked making the test too challenging or reducing it to a spectator sport.
Tristan tried two approaches. First, he designed a harder problem based on data transposition and bank conflicts—only for Claude Opus 4.5 to solve it using tricks he hadn’t anticipated. The second attempt involved creating a problem so unusual that human reasoning could outperform Claude’s vast experience base. Inspired by Zachtronics games, he designed puzzles using a heavily constrained instruction set, optimizing for minimal instruction count.
This new take-home, while less realistic, seems to work. Early results show scores correlating well with candidate caliber, and it tests not just coding but also judgment in building debugging tools. Tristan admits it’s not perfect—realism may be a luxury we can no longer afford—but it’s a step forward.
An Open Challenge
Anthropic is releasing the original take-home as an open challenge. With unlimited time, humans still outperform Claude. If you can optimize below 1487 cycles, beating Claude’s best performance, Anthropic wants to hear from you. Details are on GitHub, or you can apply through their typical process, which now uses the Claude-resistant take-home.
The Bigger Question
As AI continues to advance, the challenge isn’t just about designing harder tests—it’s about redefining what it means to be a skilled engineer. Should we focus on tasks AI can’t yet handle, or on how humans and AI can collaborate effectively? The debate is far from over, and your thoughts could shape the future. What do you think? Should AI assistance be banned in evaluations, or is it a tool engineers must master? Let us know in the comments.
