AI agents will tell you everything works. Here is how to validate AI output when the mistakes are ones no human would ever make.

Peter and Dave compare notes from building software, training material, and presentations with AI. The problems they keep running into are strange ones: the wrong database ID dropped into a system, 236 passing tests that test nothing, and five layers of abstraction added to a solution that needed none of them. Their answer borrows from good agile practice. Define what validation means before the agent starts, the same way a team agrees on its definition of done. Keep checking that the context you gave the AI is still relevant as the work moves. And keep someone in the loop who knows the problem well enough to look at the result and say, "Actually, you got this wrong."

This week's takeaways:
- Decide what "done" and "valid" mean before you hand work to an AI agent, and use a separate agent to grade what comes back.
- A test suite that pings components or mocks everything out can report hundreds of passes and prove nothing, so ask for synthetic transactions that run through the whole system.
- AI will happily over-engineer a small problem, so someone needs enough knowledge to ask whether the solution fits the size and scale of what you are actually trying to solve.

Listen to the full episode at definitelymaybeagile.com
Subscribe so you never miss an episode.
Have a question or topic you'd like us to cover? Reach out at feedback@definitelymaybeagile.com