Case study, vendor-security-questionnaire-rag
Twenty CAIQ questions, zero fabricated answers.
A drafting system for vendor security questionnaires, measured on a published harness rather than described in adjectives.
Every one of the 20 questions, by outcome
The bar has no third colour because nothing was invented. Grounding is checked twice, independently, and an answer that fails the second check is replaced rather than softened.
The problem
A vendor security questionnaire is a few hundred questions whose answers already exist, scattered across a company's security policies. Filling one in is retrieval work done by an expensive human under deal pressure.
A language model drafts it in minutes. The catch is that a plausible sentence about key rotation is indistinguishable from a true one, and the answer is a written representation to a prospective customer.
The approach
Retrieval runs against the company's own policy corpus. Grounding is then enforced twice, independently: the drafting pass must cite, and a separate verification pass checks that every cited sentence actually appears in the retrieved evidence.
When verification fails, the answer is not softened or hedged. It is replaced with "not found in provided documents" and flagged for a human, with the reason recorded.
The result
On 20 questions from CAIQ v4.0.2, 13 answers were usable as drafted and 7 were correctly abstained. None were fabricated. The seven blanks are the outcome worth paying for: they tell a reviewer exactly where the policy base is silent.
| Outcome | Count | What it means for review |
|---|---|---|
| Cited | 13 | Spot-check against the named source |
| Abstained | 7 | Answer by hand, or fix the policy gap |
| Fabricated | 0 | Nothing to catch |
What isn't proven
Twenty questions is a sample, not a certification, and the corpus was one company's policy set. Scaling the eval and running it against a second corpus is the next piece of work, not a finished claim.
The harness and the methodology are in the repository, so you can disagree with the numbers using the same tooling that produced them.