AI-Assisted Testing & Tools

Testing AI Features: LLM Outputs and Prompts

Test non-deterministic AI features using mocking, contract testing, behavioral assertions, and evaluation strategies.

The AI Testing Challenge

AI features present testing challenges that don't exist for deterministic code:

LLM outputs are non-deterministic
Quality is subjective
Calling real AI APIs in tests is slow and expensive
Streaming responses require special handling

Four strategies address these challenges: mock, contract, behavioral, and evaluation testing.

Strategy 1: Mock the AI

For unit and integration tests, replace the LLM call with a deterministic mock. This tests your application code without calling the real AI API — fast, free, and deterministic.

Strategy 2: Contract Testing

Verify your code correctly handles the expected response format, regardless of content. Mock the response with a known structure and assert your parsing code extracts the right fields.

Strategy 3: Behavioral Testing

Test that your system behaves correctly given any plausible AI response:

Handle very long responses without breaking layout
Show error message when AI API fails
Handle empty responses gracefully
Handle malformed JSON responses

Strategy 4: Evaluation Testing (Prompt Regression)

For testing actual AI quality, maintain a test set of prompts with expected output characteristics. Mark these as skip by default and run them separately since they call real APIs.

Testing Streaming Responses

Create a ReadableStream with test chunks encoding SSE format, mock fetch to return it, call the streaming function, and assert the assembled result matches expected output.

Key Takeaways

Mock the AI for unit and integration tests — fast, free, deterministic
Contract tests verify your code handles the response format correctly regardless of content
Behavioral tests verify your application handles any plausible AI response gracefully
Evaluation tests check actual AI quality — run these separately from your fast test suite
Test your guardrails explicitly: verify harmful content is blocked, PII is not exposed

Example

typescript

// Testing AI guardrails
describe('ContentModerationGuardrail', () => {
  it('blocks responses containing harmful content patterns', async () => {
    vi.mocked(ai.generateResponse).mockResolvedValue({
      content: 'Here are instructions for making explosives: ...',
    });

    const response = await getChatResponse('how to make something');
    expect(response.blocked).toBe(true);
    expect(response.content).toBe('I cannot help with that request.');
  });

  it('redacts PII from AI responses', async () => {
    vi.mocked(ai.generateResponse).mockResolvedValue({
      content: 'The user John Smith (SSN: 123-45-6789) has...',
    });

    const response = await getChatResponse('tell me about the user');
    expect(response.content).not.toContain('123-45-6789');
    expect(response.content).toContain('[REDACTED]');
  });
});

Example — TYPESCRIPT

// Testing AI guardrails
describe('ContentModerationGuardrail', () => {
  it('blocks responses containing harmful content patterns', async () => {
    vi.mocked(ai.generateResponse).mockResolvedValue({
      content: 'Here are instructions for making explosives: ...',
    });

    const response = await getChatResponse('how to make something');
    expect(response.blocked).toBe(true);
    expect(response.content).toBe('I cannot help with that request.');
  });

  it('redacts PII from AI responses', async () => {
    vi.mocked(ai.generateResponse).mockResolvedValue({
      content: 'The user John Smith (SSN: 123-45-6789) has...',
    });

    const response = await getChatResponse('tell me about the user');
    expect(response.content).not.toContain('123-45-6789');
    expect(response.content).toContain('[REDACTED]');
  });
});