Home / Methodology
How we test
Same jobs, same words, same accounts. Here's exactly how every test is run and scored.
The rules
- Same prompt, word for word. Every agent gets the identical prompt for each task. You can read them all on the task pages.
- Fresh start. Each test starts in a brand new session or chat, with no memory of earlier tests.
- Same test accounts. Every agent uses the same test logins (Gmail, Woolworths, SEEK and so on) with the same seeded data.
- Australian location. Tests run from the Gold Coast, Queensland, using Australian sites and prices.
- We never press the final button. No payment, booking, application or message is ever actually submitted. Agents are told to stop at the final screen, and we stop them ourselves if they don't.
- Logins don't count as help. We type passwords and 2FA codes for the agent when asked. Anything else we say counts as a nudge.
Scoring
| Pass | Reached the correct final screen with no help beyond login or 2FA. |
| Partial | Got there with 1 or 2 nudges, or finished with a minor error. |
| Fail | Got stuck, did the wrong thing, or claimed it was done when it wasn't. Saying "done" when it isn't is always a fail. |
| Not available in AU | The agent (or the feature needed) can't be used in Australia. |
Each task page lists the exact pass, partial and fail criteria for that job.
Works in AU?
Each agent also gets an overall flag. Yes means it is available here and handled Aussie sites fine. Partly means it is available but struggled with Australian sites, blocked them, or used US stores, prices or formats. No means Australians can't use it yet.
What we record
- A screen recording and screenshots of every run
- Time taken, start to finish
- Number of nudges, and what they were
- Credits or money used
Fair go
Agents change fast, so each round is a snapshot of the month it was run. If an agent maker thinks we got something wrong, they can email pj@thataibloke.com and we'll re-check the evidence and publish any correction.