With regular code you test: give input X, expect output Y. With AI systems, output Y is rarely identical on every call. The model may phrase something differently while the meaning is the same — or phrase something differently while the meaning has shifted.

This requires a different testing philosophy: instead of comparing exact output, you test properties of the output. Is the answer relevant? Does it contain the required elements? Does it fall within the expected structure? Is it correctly classified?

Not everything in an AI workflow is non-deterministic. The surrounding code is. You can write regular unit tests for:

• Input validation and preprocessing
• Prompt construction: does your prompt builder return the expected string?
• Output validation and parsing: does your JSON parser handle expected inputs correctly?
• Routing logic: does your system direct the right tasks to the right model?

Test the code around the model with traditional unit tests. Use mocks for API calls so your tests are fast and deterministic.

For the AI output itself, you use evaluations. An evaluation measures a property of the output on a scale or as a classification:

• Relevance: is the answer relevant to the question? (score 1-5)
• Completeness: are all required points addressed?
• Tone: does the tone match the desired style?
• Factuality: does the answer contain no demonstrably incorrect information?

You can automate evaluations by asking a second AI model to rate the output. This is called LLM-as-a-judge. It is not perfect — the model can make mistakes in its assessment — but it is scalable and works well for detecting gross errors.

A pragmatic approach for prompt changes is snapshot testing. You record the output on a set of test questions as an "approved" snapshot. When a prompt changes, you run the same questions again and compare the outputs. If the outputs differ significantly, you want to know.

You can compare outputs based on semantic similarity (embedding distance) rather than exact text comparison. This detects meaning changes without failing on style variations.

Providers update models regularly. Sometimes the output of a prompt changes significantly after an update. Maintain a set of evaluation questions that you run after every provider update. This lets you spot regressions before they cause problems in production.

Automate this via a CI/CD pipeline that runs an evaluation suite after each deployment or model change.

Test the complete workflow with realistic usage scenarios, not just isolated prompts. An end-to-end test simulates a real user interaction and checks whether the end result meets expectations.

Build a test dataset of representative inputs including edge cases, short imprecise questions, and ambiguous requests. These are the cases where AI systems fail most often.

In production, monitoring is a form of continuous testing. Tracking how many interactions receive a "poor" user rating, how many validations fail, and how latency develops over time — these are all quality signals.

Set thresholds and send an alert when a quality metric drops below the threshold. This is your early warning system for quality degradation.

Automatically testing AI workflows requires a different approach from traditional testing, but it is very achievable with a combination of unit tests, evaluations, and monitoring. Mach8 builds test suites and evaluation frameworks into every AI project we deliver.

Curious about how Mach8 handles quality assurance in AI systems? View our AI agents service or get in touch.