Icon

Evaluation of RAG Workflows with Giskard RAGET

<p>The <strong>Giskard RAGET Test Set Generator</strong> (<strong>R</strong>etrieval <strong>A</strong>ugmented <strong>G</strong>eneration <strong>E</strong>valuation <strong>T</strong>oolkit) is designed to create a diverse set of questions to evaluate RAG systems. The generated questions will vary in complexity, including simple, complex, distracting, situational and double types to comprehensively test the system's capabilities. In the provided example, we populate our knowledge base with the update pages from the 5.2 and 5.3 releases and generate embeddings for each paragraph. After that we generate questions related to this knowledge base that can be used to test against a RAG system.<br>For more information about RAGET and Giskard, visit: https://docs.giskard.ai/en/stable/index.html</p>

URL: RAGET Test Set Generation https://docs.giskard.ai/en/stable/open_source/testset_generation/testset_generation/index.html
URL: RAGET Evaluation https://docs.giskard.ai/en/stable/open_source/testset_generation/rag_evaluation/index.html

Model Requirements

The generation of the test set requires a chat model for text generation and an embedding model to represent documents in the knowledge base.
Important: The embedding model used to generate the test set and to create the knowledge base must be the same.

Knowledge Base Requirements

Provide the text documents for your knowledge base and generate embeddings for the content.

Test Set Generation

Configure the Giskard RAGET Test Set Generator to produce an appropriate number of test questions for your knowledge base. Be aware that generating a larger number of questions will increase the cost of the process. If you're using a proprietary model, it's advisable to check the pricing before execution.

Test and evaluate a RAG system with RAGET

The Giskard RAGET Test Set Generator (Retrieval Augmented Generation Evaluation Toolkit) is designed to create a diverse set of questions to evaluate RAG systems. The generated questions will vary in complexity, including simple, complex, distracting, situational and double types to comprehensively test the system's capabilities. In the provided example, we populate our knowledge base with the update pages from the 5.2 and 5.3 releases and generate embeddings for each paragraph. After that we generate questions related to this knowledge base that can be used to test against a RAG system.
For more information about RAGET and Giskard, visit: https://docs.giskard.ai/en/stable/index.html

(Optional) Split the Documents

When creating a knowledge base, particularly from
extensive sources, it may be beneficial to further
divide the text into smaller sections. This approach
ensures that the content remains precise and
comprehensive, reducing the risk of overlooking
important information.

If the documents are split, they need to be connected to the Text Embedder as well as the FAISS Vector Store Creator

RAGET Evaluation

The Giskard RAGET Evaluator takes an existing RAG workflow and evaluates the generated test set against it.

During testing, the node will query the model several times, depending on the test set size. If you're using a proprietary model, be aware of the costs of this operation.

The RAG System

The RAG System will take a user query such as "What can the Giskard nodes do?"
and search for matching documents in the knowledge base to support the LLM
answering the question with related content from the vector store (knowledge base).

OpenAI Authenticator
Giskard RAGET Evaluator
Model used by theRAG system
OpenAI Chat Model Connector
Generate Test Set
Giskard RAGET Test Set Generator
text-embedding-3-small
OpenAI Embeddings Connector
Capture workflow with input:User Query
Capture Workflow Start
Generate embeddingsfor each document in theknowledge base
Text Embedder
Capture the used nodeswith inputs and output
Capture Workflow End
Model usedfor evaluation
OpenAI Chat Model Connector
Combine retrieved docs withuser query for final answer
String Manipulation
Retrieve Docs
Vector Store Retriever
Generate answer
LLM Prompter
Create Knowledge Base
FAISS Vector Store Creator
Split the documents intosmaller chunks
Text Chunker
Whats new pages for5.2 and 5.3
Retrieve Data
User query
Table Creator
Povide an OpenAI API key
Credentials Configuration
Table View
Extract question type
JSON Path

Nodes

Extensions

Links