What to let an automated software testing agent own
Creating cases from a change. It has the context at the moment the code is written, which is when the details are freshest.
Running and re-running. Mechanical, and a human in this loop adds only latency.
Triaging obvious failures. Environment down, dependency missing, target unreachable. These do not need judgement.
Writing the reproduction for a bug it hit. The case gets written while the bug is in hand, rather than in a ticket three days later.
What to keep
What correct means
An ambiguous requirement is a product decision. An agent will pick something plausible and encode it forever.
What blocks a release
Which failures matter today needs context that is not in any document.
Whether the check is honest
Read the plan. An agent asked for green may deliver green by the shortest route.
The boundary in one sentence
Let it decide how to check, keep deciding what should be true. That splits cleanly, it maps to where each side is actually better, and it survives contact with a real team.
The guardrail worth having
Every check should name a specific observable, and you should have seen it fail at least once. A vague check is an invitation for the agent to satisfy it rather than fix the product, and that failure mode is quiet by nature.
What goes wrong in month two
The first weeks of working this way go well, and there is a predictable dip afterwards that is worth anticipating.
The agent has created a hundred cases. Most are reasonable, nobody has read them all, and two or three assert something subtly wrong. Those two or three now fail whenever the product changes correctly, so people start treating red as ambiguous. From there it is the familiar slide: rerun, then skip, then ignore.
The intervention is cheap if you make it on schedule rather than in response to a crisis. Once a month, read the assertions of every case added since the last pass. Not the steps, only the assertions. It takes under an hour at this volume and it catches exactly the cases that would otherwise erode trust in the whole suite.
Setting it up
Terminal
npm install -g @testsprite/testsprite-cli
testsprite setup
The same setup is available in the TestSprite dashboard if you would rather not install anything locally. The rest of the CLI surface is in the CLI repository.
Connect the repository from the dashboard and runs start from the deployment you already produce, or add a step to your own workflow instead.
What TestSprite owns and what stays yours
It owns the how: creating cases from a change, running them against the deployed application, re-running, and triaging the mechanical failures. Setup installs the verification skill into your coding agent so all of that happens where the work is.
You keep the what. Reading the plan, correcting the assertions that would be true of a broken product, and deciding which failures block a release. That is an hour a week at normal volume and it is where your knowledge of the product enters the suite.
The benefit of the split is that reproduction cases get written while the bug is in hand rather than three days later in a ticket, and regression coverage grows from what actually went wrong rather than from a planning exercise.
Can the agent run without supervision?
For running and mechanical triage, yes. For deciding what should be true and what blocks a release, supervision is the point rather than an overhead.
How do we stop test sprawl?
Review created cases like code and delete duplicates. Volume without judgement is the main failure mode.
What if it writes a check that cannot fail?
Common enough to plan for. Break the thing deliberately and confirm the check goes red before you rely on it.
Does it need repository access?
It verifies the deployed application through its interface. Your coding agent already has the code.
How do we measure whether this works?
Escaped defects and time from failure to fix. Test counts rise immediately and mean very little.
It decides how. You decide what.
An automated software testing agent should own creating, running and triaging. Keep what correct means, what blocks a release, and whether the check is honest. Name a specific observable in every check and confirm it can fail.