A candidate once pasted an entire assessment brief into Replit. The resulting submission left too much uncertainty about their own ability. Read why I started ProbeHaus.
A useful exercise needs evidence of how someone clarifies, directs, checks and changes the work. The finished application is only part of that evidence.
Start with the work.
List three decisions this engineer will make in their first few months. Keep them specific to your organisation: extending an existing product, integrating a customer system, or deciding how much complexity a feature needs.
Write down the constraints that matter: your stack, team size, security boundaries and how you use agents. Build the assessment around those decisions.
“Can they add a customer-facing feature to our existing application, preserve tenant isolation and leave a change the team can maintain?”
Give them a useful brief.
Provide a small working codebase and enough context to make sensible decisions. State where AI is allowed, which tools are available, what candidates should submit and what you want to learn.
Let a workspace owner invite a colleague.
The app already has users, workspaces and authentication. Add a small invitation flow. A proposed design includes a separate service and message queue; decide whether those are justified.
- Write acceptance criteria and record the assumptions you need to make.
- Implement one useful slice. You may use an AI coding tool.
- Show how you checked access boundaries and failure cases.
- Leave a short note on trade-offs, unfinished work and the next step.
Compare access to tools and context across candidates. Offer a way to ask questions, and make reasonable adjustments to the format where needed.
Set a sensible timebox.
Tell candidates when to stop and what can remain unfinished. A timebox can reveal prioritisation; it should leave room to understand the problem and verify the work.
Clarify & scope
Build a useful slice
Check & simplify
Explain the handover
An illustrative 90-minute structure. Pilot it with someone familiar with the role and adjust the scope before using it with candidates.
Look for decisions.
Define the evidence you want before reviewing submissions. Record what happened, where it appears in the work and what remains uncertain.
| Ability | Evidence worth exploring |
|---|---|
| Clarification | Useful questions, explicit assumptions and testable acceptance criteria. |
| Architecture | A design that fits existing systems and the team’s constraints. |
| Agent direction | Clear instructions, appropriate task boundaries and active review of the output. |
| Verification | Tests or checks that challenge permissions, failure cases and assumptions. |
| Simplification | Recognising unnecessary moving parts and removing them safely. |
| Ownership | An honest account of uncertainty, unfinished work and what should happen next. |
Check a candidate’s explanation against a specific decision or change in their submission.
Explore the evidence together.
Use the submission to guide a conversation. Choose a few consequential decisions and understand the reasoning behind them. Keep the structure consistent while adapting the follow-up to each candidate’s work.
- Reconstruct a decision: explore the constraints they noticed, the options they considered and what made them choose.
- Change one assumption: introduce a realistic requirement and see how they adapt their approach.
- Inspect a claim: walk through the evidence for a behaviour they say works, including a failure case.
Leave time for their questions. The assessment should also help them understand your team and whether the work interests them.
Keep the assessment useful.
Try the exercise with current AI tools before inviting candidates. Can a model produce a convincing submission from the brief alone? Which parts still reveal candidate decisions?
Record the assessment version, validation date and tool setup. Revisit it when models or your team’s work change materially, while preserving a comparable process within a hiring round.
Anthropic describes three iterations of its own performance-engineering take-home as newer models caught up with candidates. Maintaining a useful assessment is ongoing work. Read their account
For your next engineering hire
Have ProbeHaus handle it.
Get an assessment, candidate coordination and interview guidance built around your team.
Get a tailored assessmentResearch
Research reviewed September 2026. These sources provide context; they do not establish that every traditional assessment is ineffective.
- JetBrains: AI coding agent adoption, 2026. A weighted survey of 15,000+ professional developers: 90% reported weekly agent use and 68% daily use.
- HackerRank: AI use in assessments. Reports that 56% used AI in coding assessments in its 2025 survey. AI use is not necessarily a rule violation.
- CodeSignal: assessment fraud research, 2025. Reports a 35% flag rate covering multiple kinds of suspected misconduct. This is vendor detection data, not an industry-wide AI cheating rate.
- Anthropic: designing AI-resistant technical evaluations. An account of the effort involved in maintaining its take-home.