Testing hallucination checks on medical RAG answers

Why I did this

I work on a RAG tool that answers medical questions from the literature and cites a source for every claim. Its weak spot so far has been verification, because it checks that a cited paper exists but not whether the paper actually supports the claim, and closing that gap means adding a verifier, a model that reads the answer next to its sources and says whether something in it isn't supported.

The usual options are small classifiers trained for this job, like HHEM or MiniCheck, a large LLM asked to act as a judge, or the ready-made grounding check of the cloud the system already runs on. I also wanted to try Jev, which TypeSafe released in September 2026 and which answers with a label and a probability, so that its output can go straight into code. Before choosing I wanted to get a sense of how these approaches compare on medical answers, and of whether their probabilities are good enough to act on without someone looking, and since I couldn't find a comparison on this kind of data I put one together. Some of what came out will probably sound familiar to people who work on evaluation, but seeing it on medical answers still changed how my tool handles the claims it flags.

What counts as a hallucination

Here an answer counts as hallucinated when it says something the retrieved documents don't say, even if it happens to be true. Whether the documents themselves are right is a different question, and this test leaves it aside.

Data

I used three public datasets, 5,246 answers in total.

RAGBench is the closest to what a real RAG system produces, so it's the one I give most weight to. In each dataset 30% of the questions went into choosing thresholds and the other 70% into measuring, with every answer to the same question kept on the same side, and the dataset versions are pinned, so anyone rerunning the code should get the same rows.

What I tested

Every system returns a probability that the answer is hallucinated, or a score that can be read as one, but they arrive at that number in different ways, and the charts are coloured by how. The groups follow what each model was made for according to its makers, which doesn't necessarily say how well calibrated it turned out to be here.

Qwen is there mostly as a reference. Since Jev's number is computed from the model rather than written out by it, I wanted a large general-purpose model whose number comes out the same way, to get an idea of how much of Jev's result depends on Jev itself. I also ran it a second time asking for a written verdict and confidence like the other judges, on the same provider and with the same question, so that for this one model the way the number is read is the only thing that changes. All the judges got the same question with the same definitions, and the two with reasoning on are there because the obvious objection to LLM judges is that they do better when allowed to think first. Gemini 3.8 Flash costs about thirty times as much as Jev per answer, so I ran it on RAGBench only.

The two cloud checks are used the way their documentation describes, with default settings. AWS gets the documents, the question and the answer, while Google's API has no place for the question and gets only the documents and the answer, with longer documents cut into pieces of up to 10,000 characters as it requires. When Google finds nothing in an answer worth checking, a bare "True" for example, it returns no score and flags nothing, so I counted those answers as not flagged, which is what would happen to them in production.

How I measured

The main measure is AUROC, which describes how well a system ranks hallucinated answers above correct ones, from 0.5 for chance to 1 for a perfect ranking, and I report it as the average over the five datasets. A surface feature can point either way, since MedHallu's wrong answers are shorter than its correct ones while RAGBench's are longer, so the two trivial references take their direction from the threshold questions, dataset by dataset, the same way every checker takes its threshold from them.

I also looked at how often a "90% or more" turns out to be right. The datasets are about 45% hallucinated, which is far more than a working system should produce, so I recalculated this for rates of 5, 10 and 20%.

Each system is compared with Jev on the same answers, with intervals from 20,000 resamples and a Holm correction because there are 106 comparisons in all. For Jev and Qwen I also repeated the whole run and reworded their question, once for each of Jev's two forms and twice for Qwen, and I measured speed one request at a time and cost in dollars per 1,000 answers.

Results

Ranking

Qwen came first at 0.925, followed by Jev Choice at 0.917, Jev Noul at 0.916 and Claude Haiku 5.5 at 0.905. The gap between Jev and Qwen is significant only on MedHallu, where Qwen is 0.02 ahead, and my guess is that part of it comes from the dataset, since MedHallu's correct answers often say more than their passages do and Jev flags half of them against Qwen's 8%, with each of those flags counted as a mistake. I haven't verified this, though. On RAGBench alone the three end up within 0.002 of each other.

The average gives each dataset the same weight, so RAGBench's CovidQA part, with only 26 hallucinated answers in the test part, counts as much as MedHallu's 1,462 answers. Weighting each dataset by its size, or leaving that part out, keeps Qwen, Jev Choice and Jev Noul first, second and third, at 0.936, 0.929 and 0.928 or at 0.927, 0.926 and 0.924. Further down the order shifts by a place or three, and BioDecision rises to fourth with the weighted average, mostly because MedHallu, which it was trained on, is the largest dataset. Both averages are in the table under the chart.

Jev is significantly ahead of Gemini 2.5 Flash on 4 of the 5 datasets, of GPT-6 Luna on 3 and of most open models on 4 or 5. BioDecision is an exception, with Jev significantly ahead on 2 datasets but behind on MedHallu, which BioDecision was trained on, while GPT-5.6 Luna and Claude Haiku 5.5 stay close and are significantly behind only on MedHallu. Somewhat oddly, GPT-5.6 Luna also did better than the newer GPT-6 Luna on average, although GPT-6 Luna was ahead on RAGBench, and I didn't test the difference between the two.

Asked to write its confidence, Qwen dropped from 0.925 to 0.874, level with GPT-6 Luna at 0.875, and on RAGBench from 0.847 to 0.784. It wrote 0.05 or 0.95 for 80% of the answers and used only 12 different values in all, so most answers end up tied and can't be ranked against each other, while its token probabilities spread them out. Jev is significantly ahead of this version on 3 of the 5 datasets. For this model, at least, most of its advantage over the other judges came from the way the number was read rather than from the model.

Switching reasoning on didn't change the picture much. Haiku became the best of the judges that write their answer, but it stayed behind Jev and Qwen on the overall average and on RAGBench, and Gemini 3.8 Flash, which I only ran on RAGBench, scored 0.811 there against Jev's 0.848, significantly behind on the PubMedQA part. Haiku also declined to judge 9 of the 5,246 answers, most of them about viruses and infections, where its provider's content filter stopped the request, so its average covers 99.8% of the answers.

The cloud checks came out well below the LLM judges, Google's at 0.746 and AWS's at 0.727, about where AlignScore (0.747) and MiniCheck are, and Jev is significantly ahead of both on 4 and 5 datasets. AWS did well only on HaluBench's CovidQA part, at 0.956, where the wrong answers are visible rewrites of the passage. Google found nothing to check in 110 answers, 99 of them in that same CovidQA part, and 51 of the 110 were hallucinated. Counting those answers as not flagged puts them below every answer Google did score, and that choice costs it something, since leaving them out would raise its average to 0.761 and scoring them at 0.5 to 0.757, just above AlignScore, while its result on the other four datasets barely moves. These services are general-purpose and the answers here are all medical, which is probably part of the gap, and both can change under the same name without notice.

Answer length alone reached 0.568 and the share of words missing from the documents 0.612. Every checker with a full average beats both except Laya, at 0.559, which on these answers seems to pick up little beyond the surface, and on RAGBench RAGAS (0.558) also falls below the share of missing words (0.634).

Real answers vs synthetic ones

Almost every system does better on HaluBench, where the wrong answers were written on purpose, than on RAGBench, and the two models trained on related data drop the most, Lynx from 0.95 to 0.64 and BioDecision from 0.96, and 0.99 on MedHallu, to 0.75 on RAGBench. LettuceDetect is the only one that did better on RAGBench, RAGAS (0.56) stays close to chance there with TruLens (0.65) not far above, and the AWS check drops from 0.81 to 0.66.

The drop probably has more than one cause, because RAGBench differs from HaluBench both in its answers, which are real, and in its labels, which come from GPT-4 and are likely noisier. Even so, it suggests that a good score on a synthetic benchmark doesn't tell you much about how a checker will do on the answers your own system writes.

Can you trust a high score?

How often a system is right when it says an answer is at least 90% likely to be hallucinated depends a lot on how many hallucinated answers there are to begin with. The datasets have around 45%, and a working system should have far fewer, so I recalculated it for 5, 10 and 20%.

At 10%, Qwen is right 75% of the time, Jev Noul 66%, Claude Haiku 57% and Jev Choice 44%, while the judges that write their confidence without reasoning do worse, with Gemini 2.5 at 38%, GPT-5.6 Luna at 24% and GPT-6 Luna at 17%, partly because GPT-6 Luna says "90% or more" on 72% of all answers. The two cloud checks are right 21% (Google) and 25% (AWS) of the time. BioDecision reaches 96%, but that figure includes datasets it was trained on. On RAGBench alone the numbers are lower and the margins wide, with only 863 answers there, at about 51% for Qwen and Haiku, 40% for Jev Choice and 25% for Gemini 3.8 Flash, while Jev Noul flags so rarely there that its 70% comes with an interval running from 47% to 100%.

Qwen shows the same thing here. Writing its confidence, it was right 49% of the time at 10%, against 75% when read through its token probabilities, and 38% against 51% on RAGBench alone. The other written judges are different models, so this doesn't show that computed numbers always beat written ones, only that for the one model I could read both ways they did.

Low scores turned out to be much more dependable, at least on the mixed data, where an answer scored at 10% or less by one of the top systems was fine 98.5–99.7% of the time. It's a bit like screening for a rare condition, where, if most answers are correct, even a small share of false alarms ends up being a large part of all alarms, so a high score is better treated as a reason to go and check than as a reason to delete.

On RAGBench alone this holds for Jev and Haiku but much less for Qwen, which gave 10% or less to 54% of the hallucinated answers there, against 3% for Jev Choice, 4% for Haiku and none for Jev Noul. With 10% of answers hallucinated, a low score from Qwen would be right about 94% of the time, which is only a little better than the 90% you'd get with no check at all.

Robustness

When I reworded the question, Jev moved by 0.009 at most. Qwen didn't move with a simpler wording, but when I asked "does it contain anything unsupported?" in place of "is everything supported?" it dropped by 0.114, and the two versions agreed on only 60% of answers.

A second full run of both changed the average by 0.003 at most and flipped about 1% of verdicts, and the same three systems stay on top when I leave out MedHallu or HaluBench, or keep only RAGBench.

Time and cost

Jev took 0.24 s per check and cost $0.07 per 1,000 answers, against 0.73 s and $0.28 for Qwen (0.93 s and $0.32 when it writes its confidence), about 1 s for Gemini 2.5 Flash and about 2.5 s for the Luna models. Reasoning costs time, with Claude Haiku at 4.7 s and $0.32 and Gemini 3.8 Flash at 4.3 s and $2.07. The cloud checks answered in about 0.2 to 0.35 s even under load, with AWS charging $0.54 per 1,000 answers on these documents and Google well under a cent. That matters mostly if every answer is checked before the user sees it, since the time then adds to their wait.

Single sentences

Jev can also check each sentence on its own, and on RAGBench's sentence labels it scored 0.746 on PubMedQA, ahead of Bespoke-MiniCheck (0.670) and of MiniCheck and HHEM (around 0.6), while on CovidQA Bespoke-MiniCheck did better, with 0.860 against 0.817. The comparison isn't entirely even, because Jev sees the question and the whole answer, while the others see only the sentence.

Notes on the setup

In RAGBench's PubMedQA part each question has two answers, one from GPT-3.5 and one from Claude 3 Haiku, under the same ID, so the generating model is part of each answer's ID, and the two answers to a question always share a side of the split. HaluBench has no question ID at all, and in its CovidQA part about half the answers share their question with another answer, usually the right answer and a rewritten one, so there the split follows the text of the question. The Qwen requests all go to a single provider, because the models' router otherwise spreads them across providers running different versions of the model.

On 26 answers Granite Guardian gave "yes" and "no" the same weight, to the precision the model runs at, so those sit at 0.5. I looked at its token probabilities for some of them and the tie is in the model's own output, not in how its answer is read.

I also tried ExpertQA, where medical professionals rated whether each claim was supported by its source, but every system scored about 0.5 on it, much as every model does on the LLM-AggreFact leaderboard. The stored sources are short fragments of web pages, so I looked at 16 claims the experts had marked as supported and Jev had not, and found the full 2023 page for 7 of them on the Internet Archive. With the full page Jev still judged all 7 unsupported, and in the one case I checked by hand the page indeed didn't say it. It's a small sample, and it leans partly on one of the systems being tested, but it suggests the experts were judging whether the claim was correct more than whether the source said it, so I left the dataset out.

On the hardware side, one verifier filled 16 GB of RAM on long documents, a 40 GB GPU ran out of memory on 9,000-token passages, and the Colab disk could only hold three 8B models at a time.

Limits

• RAGBench labels come from GPT-4 and not from people, which may favour judges that work like GPT-4, and none of the labels here come from people.
• Lynx was trained on HaluBench's sources and BioDecision on MedHallu data, so their scores there are inflated, and I can't know whether the closed models saw these public datasets in training.
• MedHallu's correct answers sometimes go beyond their passages, which tends to favour lenient judges, and its wrong answers are shorter than the correct ones, enough for answer length alone to reach an AUROC of 0.745 there.
• Only Qwen was read both ways, so whether token probabilities would help GPT, Gemini or Claude as much is untested, and not all of their APIs return them.
• On the core test slice (716 answers) Jev Choice and Jev Noul had calibration errors of 0.148 and 0.152, close to Claude Haiku (0.147) but higher than Qwen (0.097), Qwen writing its confidence (0.084) and Gemini 2.5 Flash (0.115), so for Jev being built for scoring didn't translate into the best-calibrated probabilities here. The lowest error of all was BioDecision's (0.048), partly on data it was trained on. With 716 answers these errors are noisy, so differences of a few hundredths shouldn't be read as a ranking.
• I wrote one prompt per system, apart from the wording test on Jev and Qwen, and Laya and BioDecision got Jev's wording.
• The largest judges, such as Claude Opus or the top GPT models with reasoning on, aren't here, and neither are Azure's groundedness detection, Cleanlab's TLM or claim-level checkers like RefChecker.
• The cloud checks and the API models can change under the same name, so a rerun may not reproduce their numbers exactly, which is one reason every score is published.
• The datasets overlap in places, with 200 questions in both HaluBench-PubMedQA and MedHallu and 76 in both CovidQA sets.
• RAGBench-CovidQA has only 26 hallucinated answers in the test part but counts for a fifth of the average. The top three keep their order with the other two averages, but its own AUROC moves a lot from one answer to the next.
• Everything is in English, and RAGBench's answers were written in 2024 by GPT-3.5 and Claude 3 Haiku, so the test shows how today's checkers catch the mistakes of those models, which are probably cruder than the ones current models make.
• RAGAS and TruLens are the scores the RAGBench authors computed in 2024 with the judge models they used then, not runs of mine, so their low results describe that setup and not how either tool would do today with a current judge. They are the least even comparison here.
• Being supported by the sources doesn't mean being medically correct.

What I changed in my tool

The tool I mentioned at the start is EBM Lens, an open-source project that searches the biomedical literature and writes answers in which every claim cites a source, and it can now use Jev to check each claim against the sources it cites. When one of those sources contradicts the claim, or none of them seems to address it, the claim stays in the answer with its citations, marked as weak and flagged for the reader to check. Deleting claims of the second kind would have looked reasonable before these numbers, but a confident flag seems to be wrong often enough that deleting would throw away good claims along with the bad ones.

If you're adding a similar check to your own system, the thing I'd take from this is to set thresholds on answers your system actually produced and to treat a high score as a request for a second look. A low score from Jev looked like a fairly safe green light on these data, but that didn't hold for every checker, so it's something to verify on your own answers, and if a general LLM is the judge it also helps to read its token probabilities rather than a confidence it writes, where the API returns them, to pin the provider and to try more than one wording of the question, since all three changed my results.

Rerun it

The code, the notebook and every score are on GitHub. The datasets are downloaded from their original sources at the pinned versions, the scores of every checker on every answer are attached to the release, and one command rebuilds all the numbers on this page from them without calling any model. Rerunning the checkers themselves needs a GPU for the open models and keys for the APIs, and the whole study cost about $20 in API and cloud calls, false starts included.