Programming · Engineering Blog · 🇦🇺 Australia · via agentusecases.ai
Lab tests an agent for a month: most tasks failed
Answer.AI gave Devin real tasks and documented that few of them worked out.
- Profession
- Researcher
- Tools
- Devin
- Source
- answer.ai ↗
- Country
- 🇦🇺 Australia
- Found via
- agentusecases.ai (Alex Finn) ↗
- Report date
- January 8, 2025
- Verified on
- October 25, 2026
What they did
Answer.AI, founded by Jeremy Howard, used Devin for a month. Among the tasks was extracting an author's 25 most recent papers from Google Scholar, getting around paywalls.
How it went
Of the tasks tested, most failed or required a lot of rework; the write-up is one of the most cited critical evaluations.
Good to know
Worth reading before you commit to expensive agents: it separates promise from results.
The prompt
I want to test an AI tool on [task] before paying for it. Build an evaluation plan with 5 real tasks, the success criterion for each and how to log the time spent fixing the output.
Replace what is in [brackets]. Prompt written by us from the account, not a literal copy of the source. Do not paste other people’s personal data (privacy laws such as LGPD and GDPR apply).
Summary in our own words. Results as stated in the original account; we have not tested or audited them. Read the source ↗ Case found via agentusecases.ai (Alex Finn); text and prompt rewritten by us. Found a mistake?
Similar cases
01Asking AI for a specific algorithm for a research project
A developer skeptical of LLMs used Gemini to generate random quantum circuits in the QCOP project.
02Letting the agent deploy and read production errors on its own
The founder of a deployment platform explains how they designed an MCP server to take the human out of "copy and paste the log".
03Launching a habit app solo and doing the marketing with AI
A dev created Kada, a habit app with no account and no ads, and used AI for marketing.
04Building an "internet TV" in 6 days with a team of agents
A dev created Crom TV, with 24-hour channels of YouTube videos, coordinating several AI agents.
05Teaching an agent to use your bank's API
A developer created an open source skill so Codex and Claude Code can integrate with the Banco Inter API.
06Building an entire platform without being a programmer
A construction inspection entrepreneur in Kazakhstan used only Claude Code for ~6 months to create more than 30 modules.
07Set up a 100% local AI coding environment
A junior developer integrated local models into VS Code with Ollama and Cline, with no paid APIs.
08Build a World Cup tracking website with AI help
An engineer built Central da Copa 2026 and explains why it wasn't "just asking the AI to do it."