Lessons learned when hiring engineers in the age of AI

A candidate can build an entire app in front of you and still leave you wondering whether you’d trust them to own it.

We recently made a few engineering hires. There’s a lot of debate about whether traditional technical interviews still make sense when engineers use AI all day. I understand the knee-jerk instinct to throw them out. But after our latest hiring spree, I came to the conclusion that that would be a major mistake.

Here’s what we learned, and what our interviews look like now.

1. AI interviews are mostly a screen now

Frontier models are good enough to do much of the thinking in a short app-building exercise. You can inspect the candidate’s prompts and watch them work live, but a working app still tells you surprisingly little about their ability to deliver great software engineering.

We still run an AI round. It’s useful for weeding out people who are effectively using advanced coding agents for the first time in the interview. I know, sad… but it happens. Expect most candidates to pass, or at least for this not to be the main reason you reject them. Beyond that minimum bar, we haven’t found it particularly useful for separating good engineers from great ones.

And no, asking someone to “describe your AI workflow” doesn’t reliably catch this. Primitive and advanced AI users sound surprisingly similar:

• “It’s my main entrypoint.”
• “I manage tickets with AI.”
• “I run many sessions in parallel.”
• “I use a million skills.”
• “I use agents for incident investigation.”
• “My agent tests the UI with playwright.”
• “I created a skill while on the toilet.”

Good and bad AI users will say all of these. You have to watch them actually use it.

2. The ideal AI challenge has a short half-life

Picture the ideal exercise: too much work for a human alone in the allotted time, but doable with AI in an hour or two. Difficult enough that the candidate still has to think, rather than just hand it to the agent and wait.

That window moves with every model release. An exercise that exposes meaningful differences between candidates today might be trivial for the next model.

You can keep making the exercise harder. But now you’re running constant evals on your interview process, across whatever models candidates might use. That’s an expensive interview to maintain.

For us, accepting the AI round as a screen was more practical than trying to keep it calibrated as a test of exceptional engineering ability.

3. Many candidates are stuck with crippling AI budgets

A recurring complaint from candidates was how little their companies let them spend on AI. For many, it was part of why they wanted to leave.

I want to bang my head against the wall every time I hear “$200 per month.” Or worse, $60 or whatever.

The consequence for hiring is that plenty of candidates haven’t had much opportunity to work with top-tier Codex and/or Claude subscriptions. Their workflow reflects those constraints.

That matters when interpreting the interview. Someone might have developed elaborate workarounds because their employer made better tools unavailable. You need to distinguish a lack of exposure from a lack of ability, and account for the ramp-up if you hire them.

4. So how DO you screen for AI usage?

The most useful opening question I’ve found is embarrassingly simple:

What is your go-to AI model?

It’s dumb, but it works. In our candidate pool, if the answer isn’t “Opus” or “Sol,” I’ve learned to expect failure of the AI workflow round.

“Fable for X, Opus for Y,” or equivalent, is fine. “Fable/Opus for planning, a super cheap model for implementation” is more of a question mark. It may just mean their company has draconian AI limits.

Which leads to the second best question:

What is the current AI limit or policy at your company?

What I’m trying to establish is whether they’ve spent enough time with capable models to know what they can delegate. The answer gives me context for the live round; it doesn’t replace it.

In that round, we ask them to build an app from scratch with AI. We look for pragmatism and efficiency.

Funnily enough, candidates who go off the deep end on skills and elaborate workflows often do worse. Frontier models need fewer skills and less direction, people! For these exercises, all that ceremony can make the process slower and more cumbersome for the candidate.

Another signal is excessive defensiveness about what the AI is allowed to do. Caution tied to a specific engineering risk is good judgment. Being nervous about everything the agent does can suggest someone hasn’t spent much time with frontier models.

That’s not necessarily a disqualifier. It may mean they need some ramp-up. I want someone who can give the agent freedom when appropriate and intervene for a reason.

And again, passing this round establishes a floor. It doesn’t tell us they’re great.

5. Take-home is poison

We initially thought a take-home would be critical in the age of AI. Give someone room to work, let them use their tools, and judge what they produce. It can be a great way to assess someone.

But unless you’re hot like Anthropic, you’ll get a lot of process churn. Strong candidates with busy lives can drop out because they don’t want to spend their evenings on your assignment.

You can accidentally build a filter for desperation. The more someone needs the job, the more time they have an incentive to pour into the exercise, and the greater their advantage. That’s a lousy selection effect.

We dropped ours pretty quickly. The extra signal wasn’t worth the extra time we were asking of candidates, especially when the live AI workflow interview covered much of the same ground.

6. Vibe-coding has made the candidate pool hella-noisy

We’re seeing candidates move from somewhat technical, non-engineering roles into full-on coding now that AI makes it possible. Some can sail through an AI workflow interview and still make poor engineering decisions.

I love converts who become true engineers. I’ve hired some amazing people without CS backgrounds. Engineering was the thing they were born to do; they just happened to major in something else.

But a polished AI demo makes it much harder to tell those people apart from someone whose engineering ability ends where the agent’s does. AI has leveled the playing field for interviewing in ways that don’t necessarily carry over to owning a production system.

With non-CS candidates in particular, we’re extra wary of treating AI fluency as evidence of engineering depth. Some of the best people I’ve hired didn’t study CS, so a degree can’t settle this. We need to see the engineering foundations directly.

The cost of getting that wrong is still enormous. Someone has to own the architecture and the consequences of bad decisions, including decisions made by an agent. In this age, true engineering mindset is harder than ever to test compared to whether someone can build an app with AI. But IMO, it’s not impossible.

7. So how DO you test for great?

My dear reader, you may not like this answer: traditional whiteboard, no-AI interview rounds are still king.

BuT vLaD tHeY aRe NoT gOiNg To WrItE cOdE bY hAnD AnYmOrE !!!1111

Let me explain.

We still need to understand how they think when the model isn’t supplying the answer.

We look for three big things:

• Problem-solving on problems they haven’t seen before.
• Good architectural judgment.
• The “spark.”

7.A. Problem-solving without AI

We want smart, adaptable people who can handle the next wave of models and AI practices, along with whatever new engineering problems come with them.

So we give a good old-fashioned problem-solving challenge involving algorithms and first-principles thinking. No AI allowed.

It deliberately doesn’t require exotic algorithms or data structures. That would skew too heavily toward LeetCode enthusiasts. (Plus, we also don’t want to test knowledge primarily, but rather critical thinking). We want to see someone reason through an unfamiliar problem. The algorithm challenge is merely a medium for observing that.

Does it favor CS backgrounds? Yes, a little. Someone without that background who does well really stands out. That can be a spark in itself (see C. below).

About 90% of our candidates fail this round. Even the ones who do well are nervous. I’d be nervous too, and I love being nerd-sniped.

When you’re trying to hire engineers in the top 5-10% a process that rejects the vast majority is by design. (And the vast majority doesn’t like being rejected hence most people hate my opinion.)

7.B. Architectural judgment

We also give a system design question. Again, no AI.

Candidates have to think through the architecture of a specific system, ask good questions, and explain the pros and cons of their choices. Then they have to adapt as the requirements change and get more restrictive. System under load, failure modes, scale, etc. Bonus points if they ask about those things unprompted!

This tests engineering instinct. Can they own their decisions? Do they make pragmatic trade-offs? Do they have that healthy, pessimistic instinct about what might go wrong, and can they adjust without overengineering everything?

Someone who’s skeptical or “pessimistic” about the right things will also know when and how to question their AI’s decisions. They’ll know where to prod the agent and where to give it freedom.

7.C. The spark

And finally, we look for the spark. The spark is something that makes you think:

There’s a chance this person will be the best engineer I’ve ever worked with.

No hiring process is perfect. There’s luck involved. Looking for a spark increases the upside of the whole gamble.

Some examples:

• Fast career progression. They owned critical stuff unusually early at a company known for high-quality engineering.
• Strong pedigree, backed by actual impact. Working at Anthropic or another respected engineering company gets my attention. You still have to check what they actually did there.
• Unusual grit or determination, including outside work. We recently hired someone who is a semi-pro at pickleball with signed sponsors. That kind of hard-won achievement can tell you something about a person, even though it doesn’t prove engineering ability.
• Deep passion for the field. Substantial side projects where it’s clear engineering is what this person does for fun. Especially when their work has significant usage, recognition, or GitHub stars.
• A track record across disciplines. Making de facto product decisions, working closely with customers, or showing unusual creativity in their writing.

This isn’t an exhaustive list. But we do want every hire to have a spark, on top of the problem-solving and judgment. It gives us a reason to bet on exceptional upside once they’ve demonstrated the foundations.

What we’re hiring for

We’re hiring people to own systems, including decisions made by their agents. I want to see them use the best tools available. I also want to know whether they can spot a bad decision when those tools make one.

P.S. We build Earthly Lunar, a guardrails engine that automatically enforces engineering standards in AI and dev workflows.