feat(ai): PR 7 — E2E Tests + Fixtures + Config

[ianwhitedeveloper]

Context

Part of the PR #394 consolidation effort. Targets ai-testing-framework-implementation-consolidation.

Dependency order: Foundation → Utilities → Parsers → Config + Validation → Core Runner → CLI + Output (#420, merged) → E2E (this PR) → outputFormat + riteway ai init (#423)

Next: Draft PR #423 (pr/agent-output-format) — outputFormat serialization strategy + riteway ai init eject command — is queued behind this PR.

---

What's in this PR

End-to-end test suite + fixture files + dedicated Vitest config for the AI testing framework, plus post-consolidation fixups to test-extractor.js.

source/e2e.test.js

Full E2E test coverage using describe.skipIf(!isClaudeAuthenticated) to gate all tests on real Claude CLI auth:

• Full workflow — runAITests + recordTestOutput against sum-function-test.sudo; asserts assertion count, pass/fail, run counts, TAP file content and filename format
• --agent-config file flow — loads claude-agent-config.json via loadAgentConfig, verifies config shape, runs the same fixture
• Validation error tests (extraction-only, faster than full workflow):
  • MISSING_PROMPT_UNDER_TEST — fixture with no import statement
  • MISSING_USER_PROMPT — fixture with no userPrompt field
  • NO_ASSERTIONS_FOUND — fixture with no assertion lines
• SudoLang userPrompt — happy-path test proving the framework handles SudoLang syntax in userPrompt

All error-path tests use Try (not try/catch) and assert the full error.cause object — name, message, code, and testFile.

Fixtures (source/fixtures/)

File	Purpose
sum-function-spec.mdc	Self-contained prompt-under-test spec (no project-specific ai/ dependency)
sum-function-test.sudo	Primary happy-path fixture (3 assertions)
sudolang-prompt-test.sudo	Happy-path fixture with SudoLang userPrompt
claude-agent-config.json	Valid agent config for --agent-config file-loading flow
no-prompt-under-test.sudo	Triggers MISSING_PROMPT_UNDER_TEST
missing-user-prompt.sudo	Triggers MISSING_USER_PROMPT
no-assertions.sudo	Triggers NO_ASSERTIONS_FOUND

Config + scripts

• vitest.config.e2e.js — dedicated config (include: ['source/e2e.test.js'], testTimeout: 300000)
• vitest.config.js — updated comment on e2e.test.js exclusion
• package.json — "test:e2e": "vitest run --config vitest.config.e2e.js"

Post-consolidation fixups (source/test-extractor.js)

• Non-inferring extraction prompt — buildExtractionPrompt now includes explicit EXTRACTION RULES instructing the agent to return "" / [] for missing fields rather than synthesizing a userPrompt or extracting assertions from imported file contents. This makes MISSING_USER_PROMPT and NO_ASSERTIONS_FOUND reliably testable end-to-end.
• Agent-attributed error messages — MISSING_USER_PROMPT and NO_ASSERTIONS_FOUND messages now correctly say "Extraction agent returned…" rather than "Test file does not…", accurately pointing at the source of truth.
• Validation reorder (A1) — userPrompt and assertions checks now fire before resolveImportPaths() IO, so structural errors surface before any filesystem reads.
• buildJudgePrompt guard (A2) — CONTEXT (Prompt Under Test) section is now conditionally omitted when promptUnderTest is empty, matching buildResultPrompt pattern.
• Mock consistency (A3) — TAP YAML embed in ai-runner.test.js mock uses JSON.stringify() for consistency with extractionResult and resultText.

---

⚠️ E2E tests: local-only, team decision needed

npm run test:e2e requires an authenticated Claude CLI and must be run locally. These tests do not run in CI by default (no claude binary available in the CI environment).

A team decision is needed on CI strategy, for example:

• Skip e2e in CI permanently (unit coverage is sufficient for most paths)
• Run e2e in CI with a secrets-injected Claude API key on a scheduled basis
• Gate e2e on a manual workflow trigger

Until a decision is made, contributors should run npm run test:e2e locally before merging PRs that touch the extraction prompt, fixtures, or agent config.

---

Why no deterministic failure fixture?

Deterministic E2E failure tests are not viable with capable LLMs as both result and judge agents — the result agent satisfies requirements from first principles regardless of bad prompt context, and the judge scores the actual output rather than the prompt quality. The failure detection path is fully covered by unit tests with mock agents in ai-runner.test.js. See source/fixtures/README.md for the full rationale.

---

Test results

npm test         → 190 tests passing
npm run lint     → Lint complete.
npm run ts       → TypeScript check complete.
npm run test:e2e → 6/6 passing (requires Claude auth)