Can a RAG system prove where its answers come from?
We indexed 3 public MAS documents on AI risk, wrote a gold set of 32 questions with answers tied to a specific passage and page, and measured whether retrieval finds that passage. This is the same method we use in a RAG Accuracy Health Check, on documents anyone can check.
What this shows
- Finding a relevant passage somewhere in the top 5 is not the same as citing the right one first. The best English recall@5 is 0.84, but the top-1 result lands on the gold page at most 0.41 of the time. A system that answers from its top result would often cite the wrong paragraph.
- Model choice changes the answer. Four open CPU models, same chunks and questions: recall@5 ranges from 0.53 to 0.84.
- Multilingual questions break English-only setups. Asking 10 of the questions in Chinese or Malay against the English corpus: English-only models score 0.00 recall@5; the best multilingual small model reaches 0.40. Small sample — a signal, not a benchmark.
- Latency is not the bottleneck here. Exact vector search over 292 chunks takes well under 1 ms; the query embedding dominates. Accuracy is where the work is.
Results — English gold set (32 questions)
| Embedding model | recall@5 | recall@1 | Citation hit (top-1 doc+page) | MRR@5 | Query p50 / p95 ms embed + search | Search-only p50 / p95 ms |
|---|---|---|---|---|---|---|
| all-MiniLM-L6-v2 | 0.84 | 0.31 | 0.31 | 0.52 | 3.8 / 29.6 | 0.039 / 0.052 |
| bge-small-en-v1.5 | 0.78 | 0.38 | 0.34 | 0.53 | 8.6 / 67.9 | 0.039 / 0.074 |
| paraphrase-multilingual-MiniLM-L12-v2 | 0.53 | 0.25 | 0.34 | 0.37 | 7.3 / 76.2 | 0.040 / 0.067 |
| multilingual-e5-small | 0.78 | 0.41 | 0.41 | 0.57 | 7.9 / 122.5 | 0.039 / 0.093 |
Multilingual probe (10 questions: 5 Chinese, 5 Malay)
| Embedding model | recall@5 | recall@1 | Citation hit (top-1 doc+page) | MRR@5 | Query p50 / p95 ms embed + search | Search-only p50 / p95 ms |
|---|---|---|---|---|---|---|
| all-MiniLM-L6-v2 | 0.00 | 0.00 | 0.00 | 0.00 | 4.6 / 13.7 | 0.038 / 0.043 |
| bge-small-en-v1.5 | 0.00 | 0.00 | 0.00 | 0.00 | 8.2 / 82.4 | 0.039 / 0.067 |
| paraphrase-multilingual-MiniLM-L12-v2 | 0.20 | 0.00 | 0.10 | 0.07 | 7.4 / 89.4 | 0.044 / 0.069 |
| multilingual-e5-small | 0.40 | 0.10 | 0.20 | 0.16 | 9.8 / 101.9 | 0.042 / 0.084 |
p95 latency on a shared, small CPU box is noisy; treat tails as indicative. Each query was run 5 times.
Failure analysis (all-MiniLM-L6-v2: questions missed in top 5)
| ID | Question | Gold | Top-1 cited |
|---|---|---|---|
| q02 | By when must FIs meet the AI life cycle controls and capability expectations? | guidelines p.5 | guidelines p.16 |
| q08 | When can a financial institution apply only basic AI governance policies? | guidelines p.8 | guidelines p.3 |
| q18 | What contingency plans are needed for high-risk AI, including kill switches? | guidelines p.17 | guidelines p.17 |
| q27 | When did the consultation on the AI risk management guidelines close? | consultation p.3 | consultation p.10 |
| q31 | Can customers appeal or request review of AI-driven decisions under FEAT? | feat p.6 | guidelines p.20 |
In a client engagement this list is the starting point of the fix plan: chunking around numbered paragraphs, hybrid keyword + vector search, re-ranking, and query rewriting are then re-measured against the same gold set.
Method
- Corpus:
- Chunking: 120-word windows with 30-word overlap (292 chunks). Index: faiss IndexFlatIP (exact, cosine), CPU only.
- Gold set: each question has a short answer and an exact evidence quote; its position in the PDF gives the gold page.
- recall@5: a top-5 chunk overlaps the evidence quote. Citation hit: top-1 chunk comes from the gold document and page. MRR@5: mean reciprocal rank of the first relevant chunk.
- Generation (LLM answer writing) is out of scope here: this measures the retrieval and citation layer that answers depend on.
- Environment: Python 3.13.5, 8 CPU threads. Code, gold set and raw results.json are published alongside this page.
Want the same numbers on your own documents? See the RAG Accuracy Health Check or email qcrcemark@gmail.com.
Documents © Monetary Authority of Singapore, used for demonstration. QCRCE is not affiliated with MAS. This page is not legal or compliance advice.