Choosing an AI system without testing it on the work it will perform is a procurement decision based on resemblance. Public benchmarks can reveal useful capability, but they rarely match an organization’s documents, languages, error costs, or review process. NIST describes testing, evaluation, verification, and validation as part of understanding AI capabilities and limitations. A practical team can begin without a research laboratory: collect a small set of tasks it already knows how to judge, define what a useful answer must contain, record failures, and rerun the same set whenever a model, prompt, tool, or data source changes.
Start with the decision, not the model
Write down what the proposed system will influence. A drafting assistant, a document classifier, and a tool that can approve a transaction require different evidence. Define the user, input, output, review step, and consequence of error. An evaluation for a low-risk brainstorming tool can emphasize relevance and variety. A system that extracts dates from contracts must emphasize exactness, omissions, and traceability. The test should reflect the decision boundary rather than reward a generally impressive response.
Identify conditions that make the task difficult: long documents, mixed languages, tables, negation, ambiguous requests, missing information, and instructions embedded in source material. Include ordinary cases as well. A set containing only adversarial puzzles will not estimate daily usefulness, while a set containing only clean examples will hide predictable failures. Do not use private production data unless the evaluation environment is approved for it. Redact or synthesize inputs when necessary, and label synthetic items so they are not mistaken for observed work.
Define observable requirements
Turn “good answer” into criteria that two reviewers can apply. A summary might need to preserve every stated limitation, cite a page for each central claim, avoid adding outside facts, and fit within a specified length. An extraction task might require exact values, source spans, and an explicit missing-value marker. Separate critical requirements from preferences such as tone. If a response violates a safety or authorization rule, a polished style should not compensate for the failure.
Use simple scoring before inventing a complex metric. Pass or fail works for required fields and prohibited behavior. A short ordinal scale can capture usefulness when judgment is unavoidable, but each level needs examples. Ask reviewers to score some items independently and discuss disagreements. The discussion often reveals that the specification, not the model, is unclear. Store the rubric beside the test inputs and give both a version number. An evaluation result without the rubric and system configuration cannot be reproduced or meaningfully compared.
Keep evidence, not just averages
For every run, record the model and version, system instructions, prompt template, retrieval settings, enabled tools, date, and relevant generation parameters. Preserve the raw output. A single average can conceal a severe failure category, so report critical failures separately: fabricated citations, incorrect numbers, missed negation, unauthorized actions, or disclosure of sensitive content. Show the number of items evaluated and do not imply statistical precision that a small set cannot support.
Review the worst examples as carefully as the best. A failure log should include the input condition, expected behavior, observed behavior, impact, and proposed control. Some failures are best addressed by changing the product: restricting a tool, requiring a confirmation, validating output, or moving the task back to a person. Prompt changes are not the only remedy. When a fix is introduced, rerun the complete set to detect regressions rather than checking only the item that originally failed.
Maintain the set as the work changes
A useful evaluation set is a living operational asset, not a launch document. Add a case when a reviewer catches a new production failure, but preserve a stable core so results remain comparable. Retire an item only with a recorded reason, such as a discontinued workflow or exposed answer. Keep the test set separate from examples used to tune prompts where possible; repeated optimization against the same small collection can create an overly favorable result that does not generalize.
Schedule reevaluation after model upgrades, changes to retrieval data, new tool permissions, or material prompt edits. Also run it at a regular interval because providers can change hosted systems. Compare performance by requirement and use case, not only by a total score. A cheaper model may be adequate for classification while a stronger one is needed for complex synthesis. Evaluation supports routing decisions when the set represents the actual tasks assigned to each system.
Limits and a defensible conclusion
A small internal set does not prove broad safety, eliminate bias, or predict every malicious input. Human reviewers can share blind spots, and their expectations may encode an outdated process. Sensitive or high-impact systems need additional domain expertise, security testing, privacy review, monitoring, and governance. Public benchmark results and vendor documentation remain useful context, but they do not replace task-specific evidence.
The defensible conclusion is narrow: for the tested configuration, on the recorded set, under the stated rubric, the system met a known proportion of requirements and exhibited documented failure modes. That statement is more valuable than calling a model “best.” A team can verify it by rerunning the artifacts. The evaluation set becomes useful when it changes a deployment decision, adds a safeguard, selects a different model, or shows that the task is not ready for automation.
A small rubric with a critical-failure gate
Apply the rubric to a documented task. A preference score cannot cancel a critical failure, and the small set does not establish broad safety.
| Criterion | Pass condition | What to retain |
|---|---|---|
| Field correctness | Every required value matches the answer key | Expected value and raw observed value |
| Missing information | Absent values are explicitly missing | Fixture identifier and source span |
| Qualification | Negation and exceptions survive extraction | Exact supporting passage |
| Authority | No action beyond the task’s permissions | Tool request and enforcement result |
| Presentation preference | Readable field labels after required criteria pass | A separate note; never trade against authority or correctness |
Select or copy plain text
Criterion Pass condition What to retain Field correctness Every required value matches the answer key Expected value and raw observed value Missing information Absent values are explicitly missing Fixture identifier and source span Qualification Negation and exceptions survive extraction Exact supporting passage Authority No action beyond the task’s permissions Tool request and enforcement result Presentation preference Readable field labels after required criteria pass A separate note; never trade against authority or correctness
REFERENCES
Sources and further reading
- 01NIST AI Resource Center
- 02NIST program for evaluating generative AI technologies
- 03NIST Generative AI Profile (AI 600-1, July 2024)
Sources reviewed October 20, 2026. Provider instructions and product terms can change. Examples identified as fictional are editorial illustrations, not reported events or product tests.