Why generative AI testing breaks the usual assertion
Assert that the response equals a fixed string and the test fails on the second run for no reason related to your product. Loosen it to a substring and it passes on output that is subtly wrong. Neither is useful, and iterating between them is how teams conclude that generative features cannot be tested.
What to assert instead
| Properties | Valid JSON, required fields present, within length, cites a source that exists. Checkable and stable across runs. |
| Refusals | It declines what it should decline, and does so consistently. Often the highest-value check in the suite. |
| Grounding | Every claim traces to something in the provided context. Catches fabrication without judging style. |
Rates, not results
One run tells you nothing. Run the same input many times and track how often the property holds. Your unit of measurement is a rate, and your signal is a change in that rate rather than a single failure.
This also changes what a regression means. The question is never "is it perfect", it is "did this change make it worse than last week".
Choosing the properties to assert
The hard part of this is not the mechanics, it is deciding what to check when you cannot check the answer. A useful way in is to ask what would make the output unusable rather than what would make it good.
Unusable is concrete: malformed output the caller cannot parse, a citation to something that does not exist, a refusal that should have been an answer, an answer that should have been a refusal, content that exceeds a limit downstream. Each of those is binary and checkable, and each corresponds to a real incident rather than to a quality opinion.
Good, by contrast, is a judgement that changes between reviewers and is not worth encoding as a gate. Track it if you like, with human review on a sample, and keep it out of the pipeline where it will produce arguments rather than decisions.
The part that tests normally
Most of a product containing a generative feature is ordinary deterministic software: authentication, storage, rate limiting, the interface around the feature. All of that should be tested conventionally, and treating the whole product as non-deterministic because one part is loses you a lot of cheap coverage.
Draw the boundary explicitly. Inside it, properties and rates. Outside it, normal assertions against the running application.
Terminal
npm install -g @testsprite/testsprite-cli
testsprite setup
If you would rather not install anything, the dashboard does the same thing. Everything else the command line can do is in the CLI repository.
Two routes, and the choice is really about who owns the pipeline. Connecting the GitHub App from the dashboard needs no change to your repository at all, because it reacts to the deployment your build already emits. Adding a GitHub Actions step puts the check in the repo where it gets reviewed like the rest of the build.
What TestSprite covers around the feature
The deterministic majority of the product, which for most teams is the larger share by far. Authentication, storage, rate limiting, the interface around the feature and every flow a user actually takes are ordinary software and test conventionally. Auto-Authentication keeps sessions alive across a run, Dynamic Variables carry values between calls, Dependency Chains derive execution order, and Auto-Cleanup removes exactly what the run created.
Coverage is generated from your product and refined in plain language, and it runs against the deployed application on every change, so the parts of the system that can be checked exactly are checked exactly.
The generative feature itself needs properties and rates, which is a separate harness with its own notion of passing. Drawing that boundary explicitly is what stops a team either applying statistical methods to deterministic code or giving up on assertions entirely because exact matches kept failing.
Can we use an AI to grade the output?
Common and workable, with a caveat: the grader has its own failure modes and needs its own calibration. Check it against human judgement on a sample before trusting it.
How many runs make a meaningful rate?
Enough that the number stops moving when you add more. Start with tens rather than hundreds and watch for stability.
Should generative tests block a merge?
A large drop in a refusal rate, yes. Small variation in quality, no, or the pipeline becomes noise.
What about cost?
Repeated runs cost money. Sample rather than running the full set on every change, and run the full set on a schedule.
Is prompt testing the same thing?
Related. Prompt changes are the most common cause of rate changes, which is a good argument for treating prompts as versioned artifacts.
Properties and rates, not equality.
Generative AI testing replaces exact assertions with properties, refusals and grounding, measured as rates across many runs. Test the deterministic majority of the product normally and draw the boundary explicitly.