LLM Testing
How to Test an LLM Before Production: A Readiness Guide
Testing a large language model means measuring how reliably it produces safe, accurate, and consistent output across the inputs your real users will send, not confirming that it works on a handful of happy-path prompts. The difference between those two activities is the difference between a demo and a product.
This guide covers what to test, how to measure it, and the thresholds that separate a system that is ready from one that only looks ready.
Step 1: Cover three categories of input
A credible test suite exercises the model across three distinct input types. Skipping any one of them leaves a blind spot that users will find for you.
- Normal cases. The requests users send most of the time. These set your baseline quality.
- Edge cases. Rare but valid scenarios, empty or malformed input, very long context, mixed languages, and ambiguous requests.
- Adversarial inputs. Jailbreak attempts, prompt injection, and scope violation, where a user tries to push the system outside its intended job.
Step 2: Measure the metrics that predict real failures
Accuracy alone is not enough for a probabilistic system. These are the measures that correlate with production incidents.
| Metric | What it answers |
|---|---|
| Hallucination rate | How often the model states something false or unsupported. |
| Groundedness or faithfulness | Whether the answer actually reflects the retrieved context in a RAG system. |
| Consistency | Whether repeated runs of the same prompt give stable answers. |
| Adversarial resistance | How often jailbreak and injection attempts succeed. |
| Drift | Whether quality changes over time or after a model update. |
| Latency and cost | Whether the system meets its performance and budget targets under load. |
Step 3: Set thresholds before you look at results
Decide what passing means in advance, so you are not tempted to move the goalposts after seeing the numbers. Widely cited guidance for language systems:
- Hallucination rate: about 5 percent or lower for general use, about 1 percent or lower for high-stakes domains.
- Evaluation set size: at least 50 examples for an early read, 500 or more for a production-grade result.
- Adversarial pass rate: define an acceptable ceiling for successful attacks, then treat any regression above it as a release blocker.
Treat these as starting points, not universal law. The correct threshold is a business decision about how much a wrong answer costs you.
Step 4: Choose evaluation methods that hold up
Golden datasets
A golden dataset is a curated set of inputs with known good outputs or scoring rubrics. It is the backbone of regression testing for AI. When a prompt or model changes, you rerun the set and watch the scores.
LLM-as-a-judge, used carefully
Using a second model to grade outputs scales well, but it has real failure modes. Judges can be biased toward longer answers, inconsistent between runs, and fooled by confident wrong answers. The pattern that works is to calibrate the judge against human labels on a sample, then audit it periodically. Do not treat an unaudited judge as ground truth.
Human in the loop
For the highest-risk categories, keep expert humans reviewing a sample of outputs. Automated metrics are fast and human review is accurate, and the combination is stronger than either alone.
Continuous integration gates
The evaluation should run automatically on every change, and the pipeline should block a merge when scores fall below threshold. A quality check that a human has to remember to run is a quality check that will eventually be skipped.
A short readiness checklist
- You have written down the three most costly ways the system can fail.
- You have a golden dataset of at least 50 examples, growing toward 500.
- You test normal, edge, and adversarial inputs.
- You track hallucination rate, groundedness, and consistency, with thresholds set in advance.
- Your evaluation runs in CI and blocks releases that regress.
- You monitor the live system for drift after launch.
If you cannot check every box yet, that is the roadmap. You can also get the full version as a free checklist. For the wider discipline this sits inside, see what AI Quality Engineering is, and if you want a partner to build this with you, read how to choose an AI testing partner.
