Companies Are Now Testing Your AI Skills in Job Interviews. Here's What They're Looking For.

In May 2025, Shopify CEO Tobi Lütke sent a company-wide memo that has since been cited in virtually every conversation about the future of hiring. The core message: before any team could request additional headcount, they had to first demonstrate that they had genuinely tried to use AI to do the work themselves. The assumption that a human hire was the default answer was gone.

The memo wasn't a policy about AI in job interviews. But it captured something that has since reshaped them.

Across industries, companies are no longer satisfied with candidates who describe AI as something they "use for research" or "occasionally for drafts." They want to see AI proficiency demonstrated, tested, and stress-tested. The interview, long a place where you recounted what you'd done in the past, is increasingly a place where you're expected to do something with AI right now.

The Shift That Changed Hiring in 2026

For most of 2024 and 2025, AI experience was an additive signal. A bullet point on a resume. A question in the last five minutes of an interview: "How have you used AI in your current role?" The honest answer was almost anything — a ChatGPT prompt, a Notion AI summary — and it was usually taken at face value.

That has changed.

The shift was driven partly by supply: the candidate pool now universally claims AI proficiency, which means claiming it has stopped signaling anything. It was also driven by demand: companies that have reorganized workflows around AI tools need to know that the person they're hiring can actually move faster and produce better work with AI than without it, not just describe a time they tried it once.

The result is a new category of interview format: AI-enabled evaluation. You're given access to tools. You're given a problem. You're expected to use the tools in real time while someone watches how you think.

Canva published a blog post in mid-2025 that crystallized the new expectation: "Yes, you can use AI in our interviews. In fact, we insist." The engineering team explained that the interview wasn't testing whether candidates could memorize syntax — it was testing whether they could make real architectural decisions while using AI as an accelerant. The ability to get AI to generate code, and then critically evaluate, refactor, and improve that code, is a different skill from either hand-coding or blindly accepting output.

Google, Shopify, and Meta have formalized similar formats in various functions. The Bloomberg reporting from July 14 described how these companies' AI-enabled interview approaches specifically test whether candidates understand AI output well enough to catch its errors — because catching errors is part of the actual job.

What They're Actually Testing

The term "AI skills" has become a catch-all that obscures what interviewers are actually looking for. Based on the formats now running at companies that have formalized this, three things are consistently evaluated.

AI judgment. Can you tell a good AI output from a bad one? A significant portion of AI-enabled interviews present the candidate with AI-generated material — a draft, a piece of code, a plan — and ask them to evaluate it. What did the AI get right? What did it get wrong? What's missing? What's subtly misleading? This is harder than it sounds, especially in unfamiliar domains. The evaluation requires domain knowledge, critical thinking, and an understanding of how AI systems tend to fail.

This is also the skill that can't be faked in the moment. A candidate who has genuinely worked alongside AI over hundreds of hours develops an intuition for its failure modes: overconfident assertions in areas where the data is thin, plausible-sounding fabrications in technical domains, structural errors in reasoning that look correct at a glance. A candidate who has described AI use on their resume but not actually built that pattern recognition can't reproduce it under pressure.

Productive collaboration with AI. Can you get AI to do useful things? This sounds like it should be easy, but the gap between "interacting with AI" and "being productive with AI" is substantial. In practice, interviewers are watching how candidates frame problems to AI tools, how they evaluate and refine outputs, how they iterate when the first result isn't right, and how they know when to stop using AI and use their own judgment instead.

The candidate who pastes a prompt, reads the output, and accepts it is demonstrating something different from the candidate who asks the AI to generate an approach, critiques it, asks it to argue the opposite, and then synthesizes across the two responses. Both used AI. The second is demonstrably more capable of doing the job.

Speed and confidence. AI-enabled work environments move faster. The interview simulates that pace. Interviewers at Shopify and similar companies are looking at whether candidates can make decisions, not just generate options. The ability to produce acceptable output quickly and adjust — rather than getting stuck iterating toward perfect — is a job-relevant signal.

What This Looks Like Across Different Roles

The format shifts by function, but the underlying evaluation is consistent.

Engineering. The in-interview coding session now typically provides access to an AI coding assistant. The question isn't "can you write this code" — it's "can you use AI to produce this code and then understand it well enough to explain the tradeoffs, catch the edge cases, and adapt it when the problem changes." Pair programming sessions where candidates use Copilot or Cursor while an interviewer watches them think have become standard at companies with formalized AI hiring policies.

Product management. Candidates are given a product problem and asked to work through it with AI in real time — generating options, evaluating tradeoffs, producing a draft spec or prioritization framework. The evaluation is whether the candidate can direct the AI toward useful output and then make real decisions from it, not just produce a document.

Marketing and content. Rather than asking about past campaigns, interviewers now commonly ask candidates to produce something during the session. Draft a brief. Write a landing page headline for this product with this positioning. The AI can help; the evaluation is whether the output is actually good, whether the candidate can tell the difference, and how long it takes.

Sales. AI proficiency in sales roles is evaluated differently — through scenario-based questions about how the candidate would use AI to prepare for calls, personalize outreach, or research prospects. Some companies have added live exercises where candidates use an AI tool during a mock sales call setup and explain their approach. The question is whether they're thoughtful about when AI helps and when it doesn't.

How to Prepare When the Interview Tests AI in Real Time

Preparation for this kind of interview requires a different approach than memorizing behavioral examples.

Build actual reps, not just familiarity. The gap between "I've used AI" and "I work in AI every day" is visible in about two minutes of watching someone work. If you haven't yet reached the point where reaching for an AI tool for a new problem is instinctive — where you're genuinely faster and better with it than without it — that's the gap to close before you interview at companies with formal AI requirements. The reps compound quickly once you commit to using AI tools as the default first step rather than the occasional supplement.

Practice evaluating AI output, not just producing it. Find examples of AI-generated work in your domain and practice spotting what's wrong with it. What did the AI omit? Where is it overconfident? What would a domain expert catch in 30 seconds that the AI missed? The ability to do this quickly and specifically is the signal that interviewers are most consistently calibrated to look for.

Know which tools, and why. Companies running AI-enabled interviews often don't specify which tool you'll be using. They want to see how you navigate an unfamiliar tool under pressure. Broader exposure to how different AI systems behave — their tendencies, their failure modes, how prompting differs — gives you more flexibility. This is also a signal about genuine investment in the space versus surface-level familiarity.

Prepare to explain your workflow. Even in interview formats that don't have a live component, expect to be asked about your actual AI workflow in real conversations — how you work through a problem, how you handle it when AI produces something wrong, how you've changed your approach as the tools have improved. These questions are harder to answer with a rehearsed talking point than with genuine experience, and interviewers know that.

Use preparation tools that mirror the real thing. One reason live AI-enabled interviews catch candidates off guard is that most interview prep still happens in static formats — studying frameworks, reviewing past questions, practicing answers in your head or out loud. Practicing in the kind of dynamic, live environment the interview will actually create — where you're using AI tools and making real decisions under time pressure — closes the gap faster. Meeting Copilot's interview assistant is built for this: you practice with live AI assistance, building the pattern of using AI effectively under pressure so the interview version feels like something you've already done.

The Interviewer's Perspective

Hiring managers at companies running AI-enabled interviews have described the clearest signal this way: the candidates who perform best are the ones who treat AI as a collaborator they're directing, not a vending machine they're querying.

The vending machine approach — type in a prompt, take the output, present it — produces candidates who seem productive in the interview but fall apart the first time the AI is wrong or the problem doesn't fit the pattern. The collaborator approach — push back, refine, add domain judgment, catch errors, iterate toward something actually good — is harder to fake and more predictive of actual job performance.

That's ultimately what these formats are testing. Not AI use. AI thinking. The ability to move faster and better with AI than without it, while retaining the judgment to know when AI is helping and when it's pulling you toward something wrong.

The Shopify memo put the frame around it: if a team can't show they've genuinely tried to use AI for the work, they haven't earned the right to ask for more headcount. The same principle is showing up in interviews. If you can't demonstrate you've genuinely internalized AI as part of how you work — not as a tool you use sometimes, but as a default mode of thinking — the claim on your resume doesn't change the evaluation.

The candidates who are prepared for this understand that the question isn't whether to use AI. The question is whether you can use it well enough that it shows.

---

Sources: CNBC: Shopify CEO Asks Employees to Justify Tasks That Can't Be Done by AI (May 2025) · Canva Engineering: "Yes, You Can Use AI in Our Interviews. In Fact, We Insist." · The Register: Canva Now Requires Use of AI During Developer Job Interviews · Bloomberg: AI Tools Can Help Job Hunters Cheat on Interviews and Coding Tests (July 14, 2026)