Testing AI Features: LLM Outputs and Prompts

The AI Testing Challenge

AI features present testing challenges that don't exist for deterministic code:

• LLM outputs are non-deterministic
• Quality is subjective
• Calling real AI APIs in tests is slow and expensive
• Streaming responses require special handling

Four strategies address these challenges: mock, contract, behavioral, and evaluation testing.

Strategy 1: Mock the AI

For unit and integration tests, replace the LLM call with a deterministic mock. This tests your application code without calling the real AI API — fast, free, and deterministic.

Strategy 2: Contract Testing

Verify your code correctly handles the expected response format, regardless of content. Mock the response with a known structure and assert your parsing code extracts the right fields.

Strategy 3: Behavioral Testing

Test that your system behaves correctly given any plausible AI response:

• Handle very long responses without breaking layout
• Show error message when AI API fails
• Handle empty responses gracefully
• Handle malformed JSON responses

Strategy 4: Evaluation Testing (Prompt Regression)

For testing actual AI quality, maintain a test set of prompts with expected output characteristics. Mark these as skip by default and run them separately since they call real APIs.

Testing Streaming Responses

Create a ReadableStream with test chunks encoding SSE format, mock fetch to return it, call the streaming function, and assert the assembled result matches expected output.

Key Takeaways

• Mock the AI for unit and integration tests — fast, free, deterministic
• Contract tests verify your code handles the response format correctly regardless of content
• Behavioral tests verify your application handles any plausible AI response gracefully
• Evaluation tests check actual AI quality — run these separately from your fast test suite
• Test your guardrails explicitly: verify harmful content is blocked, PII is not exposed