On Fact-checking RAG Outputs

Thu, 26 Dec 2024 18:36:34 +0000

Overview

In this article, we explore the challenges and methods of fact-checking RAG

(Retrieval-Augmented-Generation) output.

We begin with a philosophical rant about what exactly a fact is. And why I think knowledge is better represented in an “index” (encyclopedia) rather than in a “knowledge graph”.

Following that, we go into practical experiments, including:

Implementing a fact extractor using “GPT-4o-mini”.
Utilizing “DeBERTa” for detecting contradictions between two sentences.
Building a fact-checker pipeline.
Finally, we test these tools in a controlled environment using a single-document Retrieval-Augmented Generation (RAG) setup to detect if the RAG system has given us false information.

The code accompanying this article can be found here on Github.

Fact-Checking on The Smoking Duck Blog

On Fact-checking RAG Outputs

Overview