Giskard RAGET Test Set Generator

This node utilizes the provided knowledge base and the task description of the RAG system to automatically generate a diverse set of test questions that are designed to assess the performance of various components of the RAG system, such as the retriever, generator, and knowledge base quality. The questions target specific aspects of the RAG system, helping to identify potential weaknesses and areas for improvement.

The different types of questions include

• Simple questions: Simple questions generated from an excerpt of the knowledge base.
• Complex questions: More complex questions that use paraphrasing.
• Distracting questions: Questions containing distracting information to test the retrieval part of the RAG system.
• Situational questions: Questions that include context information to evaluate if the system can produce answers relevant to the context.
• Double questions: Questions consisting of two separate parts.

This node does not support Giskard's conversational questions yet.

For more details and examples refer to the Giskard documentation

The output table has the columns

• Question: The generated question.
• Reference Answer: A reference answer to the question.
• Reference Context: The context used to create the reference answer.
• Metadata: Additional information specific to the type of question.