Icon

Evaluate a RAG system

<p><strong>Evaluate a RAG system for medication prescriptions</strong></p><ul><li><p>This workflow tests and evaluates an RAG system that provides medication prescriptions to patients, based on their symptoms and diagnosed disease.</p><p></p></li><li><p>It features a test set generator that creates a test set of questions with varying complexity (e.g., simple, complex, distracting, situational and double types) to comprehensively test the RAG system's capabilities.</p><p></p></li><li><p>It features an evaluator that assesses the correctness of the RAG system's answers by comparing them to the reference answers contained in the automatically generated test set of questions. The main purpose of this evaluation is to help identify the weakest components in a RAG system.</p></li></ul>

URL: KNIME for Generative AI https://hub.knime.com/c/D4ckx2q_J5FPBQXu
URL: Giskard RAGET Testset Generation https://docs.giskard.ai/en/stable/open_source/testset_generation/testset_generation/index.html#raget-testset-generation
URL: Giskard RAGET Evaluation https://docs.giskard.ai/en/stable/open_source/testset_generation/rag_evaluation/index.html

Evaluate a RAG system for medication prescriptions

  • This workflow tests and evaluates an RAG system that provides medication prescriptions to patients, based on their symptoms and diagnosed disease.

  • It features a test set generator that creates a test set of questions with varying complexity (e.g., simple, complex, distracting, situational and double types) to comprehensively test the RAG system's capabilities.

  • It features an evaluator that assesses the correctness of the RAG system's answers by comparing them to the reference answers contained in the automatically generated test set of questions. The main purpose of this evaluation is to help identify the weakest components in a RAG system.

Model selection

Knowledge base

Test set generation

RAG system evaluation

Capture RAG system

Model used by the RAG system:gpt-4
OpenAI Chat Model Connector
Augmented promptengineering with context
Expression
Retrievemedication sheet
Vector Store Retriever
Generate embeddingsfor each medication sheet
Text Embedder
OpenAIAPI key
Credentials Configuration
Tabular view
View medication prescription
Model:text-embedding-3-small
OpenAI Embeddings Connector
Vector store query:Symptoms + diagnosis
Table Reader
OpenAI Authenticator
Medications.table
Table Reader
Tabular view
View test set & metadata
Create vector store frommedications dataset
FAISS Vector Store Creator
Extract retrieveddocuments
Ungroup
Model used for evaluation:gpt-4
OpenAI Chat Model Connector
Get communicationfor the patient
LLM Prompter
Generatetest set
Giskard RAGET Test Set Generator
Capture Workflow Start
Extract question typeand topic
JSON Path
Capture Workflow End
Mind: this is not deterministicand results may vary at each execution
Giskard RAGET Evaluator

Nodes

Extensions

Links