AI-Assisted Test Automation That Passes Code Review

AI test automation passes code review when the assistant is given your team's rules, checks every locator against the running application instead of guessing, and is blocked from committing until a self-audit comes back clean. Without those safeguards, an assistant produces tests that compile and still fail review.

This post explains why AI-assisted QA so often stalls at the review stage and shows a setup that fixes it, using Claude Code for test automation. The setup comes from a real engagement on the test automation repository of a multi-tenant commerce operations platform.

Why AI test automation fails code review

An assistant that has not been told your conventions will guess. It guesses locators from markup, adds sleeps to make a test pass, and builds test data inline. The code compiles and often runs, and then a reviewer leaves the same comments they left on the last merge request, and the one before that.

The root problem is that the conventions live in the wrong place. In many teams, page object rules, banned waits and shared fixtures that must never be mutated exist mostly in reviewers' heads and in a long checklist. A human learns them over time. An assistant starting a fresh session never does, unless you put them where it can read them.

Another problem is context. When a product spans several repositories that behave as one system, answering what a component actually renders means switching repositories by hand or clicking through a test environment. An assistant needs that access built in.

Put the rules in the repository, not in the prompt

The decision that mattered most was this: the rules live in the repository and the skill only points at them. A rule embedded in a prompt is invisible to the team and goes stale silently. A rule in a file is reviewable, versioned, and improves for everyone when one person learns something.

This is also how agent skills stay maintainable. A skill is a packaged set of instructions an assistant loads for a kind of task. Keep it thin, let it reference the rule file, and give it a self-update protocol. When review feedback or a failure root cause reveals a missing rule, the skill proposes an edit to the rule file, so the lesson is captured once instead of re-explained in every review.

What belongs in the rule file? Anything a reviewer has ever had to say twice: how waits are written, how page objects are structured, which fixtures are read-only, and how test data is created. Write each rule so that it can be checked. The same file then serves the human reviewer, the assistant and the self-review, and onboarding a new engineer becomes the same job as onboarding a new assistant session.

Verify locators with Playwright MCP before you commit

The Model Context Protocol lets an assistant use tools against the real world. With a browser tool in the loop, such as Playwright MCP, the agent can open the running application and look at it. That changes what it can claim.

The pattern is simple. Before a locator or assertion is committed, the agent opens the application, takes an accessibility snapshot, and confirms the actual role, accessible name or alert text. It no longer infers them from the markup. A locator confirmed against the live page is far less likely to be the guessed locator a reviewer rejects.

The accessibility snapshot is the right thing to check because role-based locators are matched against it. Markup can contain wrappers, hidden elements and generated names that never reach the user, so a locator that looks right in the source can still miss on the page.

Pair that with a pre-commit self-review layer. The assistant audits its own change against the rule file, reports every violation in a table, and the commit stays blocked until the table is empty. Review then starts from a clean baseline, and humans spend their attention on design instead of repeating style comments. The aim is not to remove review. It is to move review up a level, so that people ask whether a test checks the right behavior rather than whether it follows the conventions.

What this looked like on a real engagement

The automation repository had strict, fast-changing conventions: page-object rules, banned waits, shared fixtures that must never be mutated, and a migration from REST to GraphQL waits. Those rules lived mostly in reviewers' heads and in a long checklist. We built the following.

• A custom agent skill. It is 467 lines across three files: a role and guardrail file, nine task playbooks, and a self-update protocol. The playbooks cover authoring, model tests, refactoring, pre-push self-review, review replies, failure triage, local runs and bug filing.
• A workspace configuration. It treats the three repositories as one system, with the test repository as the write target and the others as read-only references.
• A pre-commit self-review layer. It is a 475-line rule file with four enforcement triggers. Every violation is reported in a table and the commit is blocked until that table is empty.
• Live verification through MCP. Before a locator or assertion is committed, the agent opens the running application and confirms what is really there. Over seven months, 796 accessibility snapshots and 33 screenshots were captured this way.
• Persistent memory. It holds 89 single-fact entries covering non-obvious knowledge about component traps, error semantics and environment quirks.

The result is a workflow where AI-assisted work arrives at review already conforming to the team's conventions, with locators verified against the running application rather than inferred. This is the foundation of the quality tooling Kaycore is building as product. You can read the full case study or browse all case studies.

A checklist for AI-assisted QA

• Conventions are written down in files in the repository, and the assistant skill points at them.
• The skill has playbooks for the tasks the team repeats, including review replies and failure triage.
• The skill can propose edits to the rule file when a review or a failure reveals a missing rule.
• Every locator and assertion is confirmed against the running application before it is committed.
• A pre-commit self-review reports violations in a table and blocks the commit until it is empty.
• Related repositories are visible to the assistant, with only the test repository writable.
• Hard-won facts about the product are saved as small, single-fact memory entries.

The rules this setup enforces start with how a suite waits, which we cover in how to fix flaky Playwright tests. If your product itself was built with AI, see a sample quality audit of an AI-built app, or read about our services. For the wider discipline, start with what AI Quality Engineering is.