Gate a RAG change with a fixed evaluation set

Job to be done

Decide whether a RAG change is safe to ship using reproducible evidence instead of a demo impression.

Evidence state: Editorial reference protocol; no community outcome is claimed.

Baseline

1. Freeze a representative evaluation set
2. Record current configuration and dependencies
3. Define pass, latency, cost, and rollback thresholds

Method

1. Run both versions on identical cases
2. Blind qualitative review where feasible
3. Record retrieval and answer failures separately
4. Stop if any critical gate fails

Evaluation

• Pass rate
• Citation support
• P50/P95 latency
• Cost per case
• Critical failure count

Limitations

• A fixed set can miss production distribution shifts.
• Cost and latency depend on provider, region, cache state, and date.