AI test case generation: building a generator QA engineers actually trust
Most AI test case generation demos fall apart on real apps. The four-stage approach we use to generate test cases from requirements and a live application, and the verification step that makes them trustworthy.
AI test case generation looks solved in a demo. The naive version of our generator took an afternoon: fetch a page, dump the HTML into a model, ask for test cases. It produced fifty cases that read beautifully and tested nothing. This is the rewrite, and what it taught us about generating test cases a QA engineer will sign.
What is AI test case generation?
It is using an AI model to write the test cases for an application: the steps, the starting conditions and the expected result for each thing worth checking. The input can be requirements, user stories, Jira tickets, the running application, or all of them.
Done well, it removes the slowest part of starting a QA project. Done badly, it produces a long, confident document that nobody can run.
Why the naive version fails
A single page's HTML tells you what elements exist. It does not tell you:
- which of those elements are reachable from where,
- what state the app must be in first,
- which flows actually carry business risk.
Test cases written without that context describe the DOM, not the product.
The same failure shows up when you generate test cases from requirements alone. A requirement says what should happen. It does not say what the application actually does today, so the cases cannot be checked against anything real.
What a good test case contains
Before judging a generator, agree on what it must produce. Every case we accept has:
- A precondition: the state the application must be in first.
- Steps a person or a script can follow without guessing.
- One expected result that is clearly right or wrong.
- A risk rating, so the cases that protect revenue and data come first.
- A link to its source: the requirement, ticket or journey it came from.
That last point is what makes a suite maintainable. When a requirement changes, you can find every case it touches.
The four-stage approach
The pipeline that works has four stages, and only one of them is a model call:
- Explore. Playwright walks the app, records the reachable route graph, and captures the accessibility tree rather than raw HTML. The a11y tree is already a semantic summary: roles, names, states.
- Reduce. Collapse the graph into flows. A flow is a path from an entry point to a state change.
- Generate. For each flow, ask the model for cases with an explicit risk rating and preconditions.
- Verify. A second pass checks every generated case against the captured tree. A case referencing a selector that was never observed gets dropped.
That last stage is the one that buys trust. It is also the one every demo skips.
Generating test cases from requirements and Jira tickets
Requirements and the running application answer different questions, so we use both:
- Requirements, user stories and Jira tickets say what the product is meant to do, including rules the interface does not show, such as who is allowed to do what.
- The live application says what is actually there to test.
Where the two disagree, that is not noise. It is usually a bug or an out-of-date ticket, and it is one of the most useful things the process produces.
Why a QA engineer still reviews every case
A model can write a plausible case for a rule that does not exist. So nothing is automated until a QA engineer has read it. In our own project, 78% of generated cases survived that review. The rest were removed by the reviewer, which is the point of having one. Typical reasons to reject a case are that it duplicates another, checks something of little value, or rests on a wrong assumption.
Review is also where risk gets corrected. The model rates risk from what it can see. A person knows that the discount code is the part finance cares about.
From test cases to automated tests
A reviewed, risk-rated case is most of the way to an automated test. Because each case already has a precondition, steps and an expected result, it maps directly to a script in Playwright, Cypress or Selenium:
- Smoke tests come from the highest-risk cases and run on every change.
- Regression tests cover the rest and run before each release.
- Sanity tests are the short set you run after a fix.
The scripts then run in your CI/CD pipeline, in GitHub Actions, Jenkins or Azure DevOps. We describe how an AI agent can also write those scripts in AI test automation with Playwright.
What we would do differently
Start with the accessibility tree, not the HTML. We lost two weeks to token budgets before we made that change.
The second lesson is to build the verify stage first. It is tempting to tune prompts until the output looks good. Looking good was never the problem.
Frequently asked questions
- What is AI test case generation?
- AI test case generation uses an AI model to write test cases for an application, including the steps, the starting conditions and the expected result. The input can be requirements, user stories, Jira tickets or the running application itself.
- Can AI generate test cases from requirements or Jira tickets?
- Yes, and it works best when requirements are combined with the running application. Requirements say what the product should do, and the live application shows what is actually there to test. Differences between the two often reveal bugs.
- Are AI-generated test cases accurate?
- Only with verification and review. A model can write a convincing case for behaviour that does not exist, so each case should be checked against what was actually observed in the application and then reviewed by a QA engineer before it is automated.
- Do AI-generated test cases replace a QA engineer?
- No. Generation removes the slow first draft, but a QA engineer still decides which cases matter, corrects the risk ratings and rejects cases built on a wrong assumption.
- Can generated test cases be turned into automated tests?
- Yes. A case with a precondition, clear steps and one expected result maps directly to an automated script in Playwright, Cypress or Selenium, which can then run in a CI/CD pipeline.
Want test cases and working tests for your own application? Our free test run puts our AutoQA Agent on your staging site and gives you the results in 48 hours, or you can talk to us first.