H2O Eval Studio Test Suite Library

This is H2O Eval Studio test suite library for LLM, RAG and agent evaluation.

Test suites can be used for question answering, privacy, fairness, security, summarization and classification evaluation. In addition to that test suites can be combined, sampled, perturbed and customized for specific evaluation needs.

Test suites are provided normalized in H2O Eval Studio JSON format - see also details:

Test Suites

• advglue
  • Adversarial GLUE Benchmark (AdvGLUE) is a comprehensive robustness evaluation benchmark that focuses on the adversarial robustness evaluation of language models.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 721
• Alan Greenspan Globalization
  • Alan Greenspan globalization speech RAG evaluation test suite.
  • Evaluates: LLM, RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 5
• Analogical Similarity
  • To measure the model’s ability in discriminating between different degrees of analogical similarity in two given episodes
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 323
• Annual Report Singtel
  • Test suite Annual Report Singtel for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 160
• Annual Report Singtel (multi choice)
  • Test suite Annual Report Singtel (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 74
• Annual Report Singtel (question type)
  • Test suite Annual Report Singtel (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 153
• ARC-Easy
  • A dataset of genuine grade-school level, multiple-choice science questions in advanced Q&A. This is the challenge set.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 2590
• ARC-Easy
  • A dataset of genuine grade-school level, multiple-choice science questions in advanced Q&A. This is the easy set.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 5197
• Bank Teller
  • Bank tellers are responsible for handling customer transactions like deposits, withdrawals, transfers, money orders, and checking. They are also responsible for counting cash, answering customer questions, and keeping records of money and negotiable instruments involved in financial transactions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 6
• Banking Act
  • Test suite Banking Act for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 148
• Banking Act (multi choice)
  • Test suite Banking Act (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 69
• BBQ-lite on age - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on age, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1840
• BBQ-lite on age - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on age, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1840
• BBQ-lite on disability-status - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on disability-status, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 778
• BBQ-lite on disability-status - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on disability-status, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 778
• BBQ-lite on gender - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on gender, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 2836
• BBQ-lite on gender - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on gender, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 2836
• BBQ-lite on nationality - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on nationality, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1540
• BBQ-lite on nationality - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on nationality, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1540
• BBQ-lite on physical-appearance - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on physical-appearance, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 788
• BBQ-lite on physical-appearance - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on physical-appearance, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 788
• BBQ-lite on race-ethnicity - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on race-ethnicity, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 3440
• BBQ-lite on race-ethnicity - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on race-ethnicity, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 3440
• BBQ-lite on race-x-gender - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on race-x-gender, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 7980
• BBQ-lite on race-x-gender - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on race-x-gender, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 7980
• BBQ-lite on race-x-ses - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on race-x-ses, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 5580
• BBQ-lite on race-x-ses - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on race-x-ses, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 5580
• BBQ-lite on religion - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on religion, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 600
• BBQ-lite on religion - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on religion, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 600
• BBQ-lite on ses - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on ses, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 3432
• BBQ-lite on ses - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on ses, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 3432
• BBQ-lite on sexual-orientation - Ambiguous Questions
  • This dataset is an excerpt from Bias Benchmark from QA on sexual-orientation, containing only the ambiguous questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 432
• BBQ-lite on sexual-orientation - Disambiguated Questions
  • This dataset is an excerpt from Bias Benchmark from QA on sexual-orientation, containing only the disambiguated questions.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 432
• Ben Bernanke Economic Outlook
  • Ben Bernanke, The Economic Outlook, May 2013 RAG evaluation test suite.
  • Evaluates: RAG, LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 5
• Broker Agreement
  • Test suite Broker Agreement for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 59
• Broker Agreement (multi choice)
  • Test suite Broker Agreement (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 19
• Broker Agreement (question type)
  • Test suite Broker Agreement (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 58
• cause_and_effect
  • This recipe measures if the model is able to distinguish and understand cause and effect. The higher the grade, the better the system can perform this capability.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 102
• CBA AnnualReport 2023
  • Test suite CBA AnnualReport 2023 for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 33
• CBA AnnualReport 2023 (multi choice)
  • Test suite CBA AnnualReport 2023 (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 106
• ChallengingToxicityPrompts
  • ChallengingToxicityPrompts was created by the authors of DecodingTrust, with the aim to measure the model’s ability in identify toxic sentences.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1000
• ChallengingToxicityPrompts
  • ChallengingToxicityPrompts was created by the authors of DecodingTrust, with the aim to measure the model’s ability in identify toxic sentences.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 2392
• ChallengingToxicityPrompts
  • ChallengingToxicityPrompts was created by the authors of DecodingTrust, with the aim to measure the model’s ability in identify toxic sentences.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1000
• Chinese Linguistics & Cognition Challenge (CLCC)
  • This dataset is a subjective benchmark created by BAAI FlagEval group.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 760
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1392
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 980
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 586
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 11988
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1534
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 13528
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 2176
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 8700
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 2176
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1534
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 8700
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 7400
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 3960
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 13528
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 3960
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1856
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 560
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1856
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1588
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 586
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 13528
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 11988
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1588
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 7400
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 980
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1392
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 13528
• Chinese Version - Bias Benchmark for QA
  • This aims to measure social biases across 9 different categories in Chinese language.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 560
• Constitution of the Republic of Singapore
  • Test suite Constitution of the Republic of Singapore for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 29
• Constitution of the Republic of Singaporet (multi choice)
  • Test suite Constitution of the Republic of Singaporet (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 54
• Contextual Parametric Knowledge Conflicts
  • To measure the model/’s ability in making informed predictions
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 17528
• Cyber Security Policy
  • Test suite Cyber Security Policy for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 131
• Cyber Security Policy (multi choice)
  • Test suite Cyber Security Policy (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 80
• Cyber Security Policy (question type)
  • Test suite Cyber Security Policy (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 125
• CyberSecEval Prompt Injection
  • Check if llm is susceptible to prompt injection
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 251
• Defense Management
  • Test suite Defense Management for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 136
• Defense Management (multi choice)
  • Test suite Defense Management (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 74
• Defense Management (question type)
  • Test suite Defense Management (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 137
• Digital Health Guidelines
  • Test suite Digital Health Guidelines for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 141
• Digital Health Guidelines (multi choice)
  • Test suite Digital Health Guidelines (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 64
• Digital Health Guidelines (question type)
  • Test suite Digital Health Guidelines (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 127
• Employment Contract
  • Test suite Employment Contract for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 67
• Employment Contract (multi choice)
  • Test suite Employment Contract (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 40
• Employment Contract (question type)
  • Test suite Employment Contract (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 60
• enronemail
  • The Enron email dataset is aimed at evaluating if there is leakage of private information. The dataset was used in DecodingTrust as part of its privacy evaluation.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 166418
• Ethics-Commonsense-Hard
  • This dataset contains prompts that embedded the body of moral standards and principles that most people intuitively accept.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1000
• Ethics-Commonsense-Hard
  • This dataset contains prompts that embedded the body of moral standards and principles that most people intuitively accept.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1000
• Ethics-Commonsense-Hard
  • This dataset contains prompts that embedded the body of moral standards and principles that most people intuitively accept.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1000
• Ethics-Commonsense-Hard
  • This dataset contains prompts that embedded the body of moral standards and principles that most people intuitively accept.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1000
• EU AI Act
  • Test suite EU AI Act for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 26
• EU AI Act (multi choice)
  • Test suite EU AI Act (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 48
• EU AI Act (question type)
  • Test suite EU AI Act (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 144
• Facts about Asia pacific in True and False in 4 languages (Chinese, Malay, Tamil and English)
  • Contain prompts that contain facts about Asia Pacific and Singapore, in True/False format
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 22
• Facts about Singapore in True and False
  • Contain prompts that contains facts about Singapore, in True/False format
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 50
• FBIAgentGPT
  • Evaluates: LLM, RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 20
• Financial Records Management
  • Test suite Financial Records Management for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 138
• Financial Records Management (multi choice)
  • Test suite Financial Records Management (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 89
• Financial Statements Alphabet Tesla
  • Test suite Financial Statements Alphabet Tesla for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 77
• Financial Statements Alphabet Tesla (multi choice)
  • Test suite Financial Statements Alphabet Tesla (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 79
• Food in Singapore
  • Contain prompts that test model’s udnerstanding in Food, in True/False format
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 100
• Frank Summarization (small)
  • Test suite for the summarization evaluation.
  • Evaluates: LLM
  • Purposes: summarization
  • Tests: 1
  • Test Cases: 7
• Frank Summarization (small)
  • Test suite for the summarization evaluation.
  • Evaluates: LLM
  • Purposes: summarization
  • Tests: 1
  • Test Cases: 499
• GAIA
  • GAIA - A Benchmark for General AI Assistants. GAIA is a benchmark for General AI Assistants. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs. This test suite has tasks of all levels, and both tasks which use and do not use documents.
  • Evaluates: LLM, RAG, agent
  • Purposes: Q&A
  • Tests: 39
  • Test Cases: 203
• GAIA (tasks w/o documents)
  • GAIA - A Benchmark for General AI Assistants. GAIA is a benchmark for General AI Assistants. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs. This test suite has GAIA tasks which do not use documents (corpus) to evaluate the agentic system/LLM/RAG.
  • Evaluates: RAG, LLM, agent
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 127
• Gender Occupational Bias
  • The gender occupational bias is a set of gender neutral text-to-image prompts that are likely to result in models favouring the generation of one gender over the other. The occupations included were proposed in the paper: https://arxiv.org/abs/2211.03759
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 13
• Gender Occupational Bias
  • The gender occupational bias is a set of gender neutral text-to-image prompts that are likely to result in models favouring the generation of one gender over the other. The occupations included were proposed in the paper: https://arxiv.org/abs/2211.03759
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 13
• gre_reading_comprehension
  • To measure the model/’s ability to summarize text, infer and deduce knowledge from context, and match the context.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 32
• GSM8K
  • This dataset consists of high quality grade school math problems created by human. These problems take between 2 and 8 steps to solve, and usually requires a sequence of elementary calculations.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 8792
• H2O.ai Eval GPT
  • H2O.ai Eval GPT test suite with prompts from diverse areas like math, planning, writing or troubleshooting.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 60
• h2oGPTe Benchmark (2023-11-15)
  • h2oGPTe benchmark test suite based on GitHub repository test data from 2023/11/15.
  • Evaluates: RAG, LLM
  • Purposes: Q&A
  • Tests: 34
  • Test Cases: 122
• h2oGPTe Benchmark (2024-08-26)
  • h2oGPTe benchmark data import (2024-08-26_16h56m00s).
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 63
  • Test Cases: 154
• h2oGPTe Benchmark (2024-10-01)
  • h2oGPTe benchmark data import (2024-10-01_15h37m17s).
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 64
  • Test Cases: 155
• Health Service Standards
  • Test suite Health Service Standards for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 34
• Health Service Standards (multi choice)
  • Test suite Health Service Standards (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 71
• Health Service Standards (question type)
  • Test suite Health Service Standards (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 82
• HellaSwag
  • This dataset is used to evaluate commonsense with questions that are trivial for humans but difficult for state-of-the-art models.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 49947
• Home Affairs
  • Test suite Home Affairs for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 109
• Home Affairs (multi choice)
  • Test suite Home Affairs (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 61
• Home Affairs (question type)
  • Test suite Home Affairs (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 115
• HR Policy
  • Test suite HR Policy for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 149
• HR Policy (multi choice)
  • Test suite HR Policy (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 86
• HR Policy (question type)
  • Test suite HR Policy (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 122
• HR Policy Procedures (multi choice)
  • Test suite HR Policy Procedures (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 118
• HR Policy Procedures (question type)
  • Test suite HR Policy Procedures (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 148
• HSBC Annual Report
  • Test suite HSBC Annual Report for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 126
• HSBC Annual Report (multi choice)
  • Test suite HSBC Annual Report (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 84
• HSBC Annual Report (question type)
  • Test suite HSBC Annual Report (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 121
• Iconic Places in Singapore
  • Contain questions about Singapore’s iconic places.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 16
• IMF Financial Statements
  • Test suite IMF Financial Statements for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 130
• IMF Financial Statements (multi choice)
  • Test suite IMF Financial Statements (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 68
• Immigration in Singapore
  • Test suite Immigration in Singapore for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 128
• Immigration in Singapore (multi choice)
  • Test suite Immigration in Singapore (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 67
• Immigration in Singapore (question type)
  • Test suite Immigration in Singapore (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 122
• In-the-wild Jailbreak Dataset
  • This dataset contains jailbreak prompts that were collected from various places (e.g., Discord).
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 22
• Inappropriate Image Prompts (I2P)
  • The I2P benchmark contains real user prompts for generative text2image prompts that are unproportionately likely to produce inappropriate images. I2P was introduced in the 2023 CVPR paper Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models. This benchmark is not specific to any approach or model, but was designed to evaluate mitigating measures against inappropriate degeneration in Stable Diffusion.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 4703
• Information Security
  • Test suite Information Security for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 136
• Information Security (multi choice)
  • Test suite Information Security (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 74
• Information Security (question type)
  • Test suite Information Security (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 125
• Information Security Policy
  • Test suite Information Security Policy for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 156
• Information Security Policy (multi choice)
  • Test suite Information Security Policy (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 67
• Information Security Policy (question type)
  • Test suite Information Security Policy (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 146
• Inherent Risk Assessment
  • Test suite Inherent Risk Assessment for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 122
• Inherent Risk Assessment (multi choice)
  • Test suite Inherent Risk Assessment (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 59
• Inherent Risk Assessment (question type)
  • Test suite Inherent Risk Assessment (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 116
• IRS Document 1
  • Test suite IRS Document 1 for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 41
• IRS Document 1 (multi choice)
  • Test suite IRS Document 1 (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 62
• IRS Document 1 (question type)
  • Test suite IRS Document 1 (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 146
• IRS Document 2
  • Test suite IRS Document 2 for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 31
• IRS Document 2 (multi choice)
  • Test suite IRS Document 2 (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 47
• IRS Document 2 (question type)
  • Test suite IRS Document 2 (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 127
• IRS Strategic Operating Plan
  • Test suite IRS Strategic Operating Plan for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 134
• IRS Strategic Operating Plan (multi choice)
  • Test suite IRS Strategic Operating Plan (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 45
• IRS Strategic Operating Plan (question type)
  • Test suite IRS Strategic Operating Plan (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 128
• Kaggle: LLM Science Exam competition
  • Test suite which is based on Kaggle’s LLM Science Exam competition.
  • Evaluates: RAG, LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 200
• Kaggle: LLM Science Exam competition (small)
  • Test suite which is based on Kaggle’s LLM Science Exam competition.
  • Evaluates: RAG, LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 25
• List of Occupations
  • This dataset contains a list of occupations collected from various websites (e.g., MoM of Singapore). The dataset can be used to test job-related applications.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 7270
• m3exam-indonesian
  • This dataset contains local exam questions for testing Indonesian capability.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1580
• m3exam-thai
  • This dataset contains local exam questions for testing Thai capability.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 2126
• m3exam-vietnamese
  • This dataset contains local exam questions for testing Vietnamese capability.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1745
• Maternal Child Healthcare
  • Test suite Maternal Child Healthcare for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 156
• Maternal Child Healthcare (multi choice)
  • Test suite Maternal Child Healthcare (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 112
• Maternal Child Healthcare (question type)
  • Test suite Maternal Child Healthcare (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 147
• MedMCQA
  • MedMCQ is a large-scale, multiple-choice question answering dataset designed to address realworld medical entrance exam questions. It contains more than 194k high-quality AIIMS & NEET PG entrnace exam MCQs covering 2.4k healthcare topics and 21 medical subjects.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 182822
• MedQA (US)
  • MedQA is a free-form multiple-choice OpenQA dataset for solving medical problems. These are collected from the professional medical board exams. We extracted the list of MCQ questions from the US in this dataset.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 10178
• MLC Baseline - Chemical, Biological, Radiological, Nuclear, and Explosives
  • Responses that enable, encourage, or endorse the creation of weapons of mass destruction.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 720
• MLC Baseline - Chemical, Biological, Radiological, Nuclear, and Explosives
  • Responses that enable, encourage, or endorse the creation of weapons of mass destruction.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1570
• MLC Baseline - Child Abuse and Exploitation
  • Responses that contain, describe, enable, encourage, or endorse the sexual abuse of children.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 720
• MLC Baseline - Child Abuse and Exploitation
  • Responses that contain, describe, enable, encourage, or endorse the sexual abuse of children.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1330
• MLC Baseline - Hate
  • Responses that demean or dehumanize people on the basis of their sensitive, personal characteristics.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 26530
• MLC Baseline - Hate
  • Responses that demean or dehumanize people on the basis of their sensitive, personal characteristics.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 720
• MLC Baseline - Non-Violent Crimes
  • Responses that enable, encourage, or endorse the commission of non-violent crimes.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1810
• MLC Baseline - Non-Violent Crimes
  • Responses that enable, encourage, or endorse the commission of non-violent crimes.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 720
• MLC Baseline - Sex-Related Crimes
  • Responses that enable, encourage, or endorse the commission of sex-related crimes.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1330
• MLC Baseline - Sex-Related Crimes
  • Responses that enable, encourage, or endorse the commission of sex-related crimes.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 720
• MLC Baseline - Suicide & Self-Harm
  • Responses that enable, encourage, or endorse acts of intentional self-harm.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 720
• MLC Baseline - Suicide & Self-Harm
  • Responses that enable, encourage, or endorse acts of intentional self-harm.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1090
• MLC Baseline - Violent Crimes
  • Responses that enable, encourage, or endorse the commission of violent crimes.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 720
• MLC Baseline - Violent Crimes
  • Responses that enable, encourage, or endorse the commission of violent crimes.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 4390
• MMLU
  • This dataset covers 57 tasks including elementary mathemathics, US history, computer science, law, and more.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 17487
• MMLU (Arabic)
  • The MMLU (Massive Multitask Language Understanding) benchmark is a comprehensive test for evaluating the knowledge and reasoning abilities of language models across diverse subjects. It consists of multiple-choice questions spanning 57 tasks, from elementary to professional levels, covering various fields like STEM, humanities, and social sciences. MMLU is designed to assess a model’s ability to generalize knowledge and reason across different domains, making it a challenging and realistic evaluation. This is the Arabic version of the MMLU benchmark, translated using machine translation.
    • Source: https://gitlab.com/tiiuae/alghafa/-/blob/main/arabic-eval/mmlu_okapi_ar/eval/mmlu_ar_okapi_test.csv
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 12928
• MMLU (Arabic) 50 prompts
  • The MMLU (Massive Multitask Language Understanding) benchmark is a comprehensive test for evaluating the knowledge and reasoning abilities of language models across diverse subjects. It consists of multiple-choice questions spanning 57 tasks, from elementary to professional levels, covering various fields like STEM, humanities, and social sciences. MMLU is designed to assess a model’s ability to generalize knowledge and reason across different domains, making it a challenging and realistic evaluation. This is the Arabic version of the MMLU benchmark, translated using machine translation.
    • Source: https://gitlab.com/tiiuae/alghafa/-/blob/main/arabic-eval/mmlu_okapi_ar/eval/mmlu_ar_okapi_test.csv
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 50
• MMLU Anatomy
  • MMLU measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This particular dataset contains questions on MMLU Anatomy
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 135
• MMLU Clinical Knowledge
  • MMLU measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This particular dataset contains questions on MMLU Clinical Knowledge
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 265
• MMLU College Biology
  • MMLU measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This particular dataset contains questions on MMLU College Biology
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 144
• MMLU College Medicine
  • MMLU measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This particular dataset contains questions on MMLU College Medicine
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 173
• MMLU Medical Genetics
  • MMLU measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This particular dataset contains questions on MMLU Medical Genetics
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 100
• MMLU Professional Medicine
  • MMLU measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This particular dataset contains questions on MMLU Professional Medicine.json
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 272
• NIST AI 600-1
  • NIST AI 600-1 document questions and answers with constraints for RAG evaluation.
  • Evaluates: RAG, LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 528
• NIST AI 600-1 (small)
  • NIST AI 600-1 document questions and answers with constraints for RAG evaluation.
  • Evaluates: LLM, RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 30
• OIG ARRA
  • OIG ARRA stands for Office of Inspector General (OIG) American Recovery and Reinvestment Act (ARRA). The ARRA was a stimulus package passed by the United States Congress in 2009 in response to the Great Recession. It provided funding for various programs and initiatives aimed at stimulating the economy and creating jobs.
  • Evaluates: LLM, RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 5
• Personally Identifiable Information leakage
  • This evaluation set tests large language model leakages of personally identifiable information.
  • Evaluates: LLM
  • Purposes: privacy, security
  • Tests: 1
  • Test Cases: 8
• Places in Singapore
  • Contain prompts that test model’s udnerstanding places in Singapore, in True/False format
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 50
• Policy Document
  • Test suite Policy Document for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 141
• Policy Document (multi choice)
  • Test suite Policy Document (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 110
• Product Disclosure Statement
  • Test suite Product Disclosure Statement for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 130
• Product Disclosure Statement (multi choice)
  • Test suite Product Disclosure Statement (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 71
• PubMedQA
  • PubMedQA is a large-scale, multiple-choice question answering dataset designed to address realworld medical entrance exam questions. It contains more than 194k high-quality AIIMS & NEET PG entrnace exam MCQs covering 2.4k healthcare topics and 21 medical subjects.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1000
• Question type Financial Statements Alphabet Tesla
  • Test suite Question type Financial Statements Alphabet Tesla for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 365
• RealtimeQA
  • RealTime QA is a dynamic question answering (QA) platform that inquires about the present.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 50
• RealToxicityPrompts
  • RealToxicityPrompts aims to measure the model’s ability in identify toxic sentences.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 198884
• Red Teaming (Giskard AI)
  • This evaluation set tests large language models ability to adhere to guardrails - not leaking any secure information or system prompt for example.
  • Evaluates: LLM, RAG
  • Purposes: security
  • Tests: 1
  • Test Cases: 9
• Risk Management Guidelines
  • Test suite Risk Management Guidelines for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 140
• Risk Management Guidelines (multi choice)
  • Test suite Risk Management Guidelines (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 91
• Risk Management Guidelines (question type)
  • Test suite Risk Management Guidelines (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 123
• Risk Management Policy
  • Test suite Risk Management Policy for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 37
• risk management policy (multi choice)
  • Test suite risk management policy (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 62
• risk management policy (question type)
  • Test suite risk management policy (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 96
• Risk Management Techniques Tool
  • Test suite Risk Management Techniques Tool for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 133
• Risk Management Techniques Tool (multi choice)
  • Test suite Risk Management Techniques Tool (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 93
• Risk Management Techniques Tool (question type)
  • Test suite Risk Management Techniques Tool (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 120
• SA Home Affairs
  • Test suite SA Home Affairs for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 148
• SA Home Affairs (multi choice)
  • Test suite SA Home Affairs (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 74
• SA Home Affairs (question type)
  • Test suite SA Home Affairs (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 140
• Safety Benchmark (Singapore Context)
  • Contain prompts that test safety in Singapore-context
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 59
• Samsum Summarization
  • Test suite for the summarization evaluation.
  • Evaluates: LLM
  • Purposes: summarization
  • Tests: 1
  • Test Cases: 46
• Sensitive data leakage
  • This evaluation set tests large language model leakages of sensitive data like activation keys, certificates and API keys.
  • Evaluates: LLM, RAG
  • Purposes: privacy
  • Tests: 1
  • Test Cases: 8
• sg-legal-glossary
  • A list of singapore legal terms extracted from SICC and Judiciary websites.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 425
• sg-university-tutorial-questions-legal
  • Contain tutorial questions ans answers from Singapore’s Universities to test model’s ability in understanding legal context in Singapore.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 32
• Singapore Cyber Landscape
  • Test suite Singapore Cyber Landscape for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 110
• Singapore Cyber Landscape (multi choice)
  • Test suite Singapore Cyber Landscape (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 64
• Singapore Cyber Landscape (question type)
  • Test suite Singapore Cyber Landscape (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 104
• Singapore Labour Force
  • Test suite Singapore Labour Force for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 120
• Singapore Labour Force (multi choice)
  • Test suite Singapore Labour Force (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 61
• Singapore Labour Force (question type)
  • Test suite Singapore Labour Force (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 113
• Singapore Polical History
  • Contain questions about Singapore’s key historical events in political.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 21
• Singapore Transport System
  • Contain questions about Singapore’s public housing system
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 12
• Singapore Transport System
  • Contain questions about Singapore’s transport system
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 27
• squad-shifts-tnf
  • Zero-shot reading comprehension on paragraphs and questions from squadshifts
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 48201
• SR 11-7
  • SR 11-7 MRM document questions and answers for RAG evaluation.
  • Evaluates: RAG, LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 171
• SR 11-7 (small)
  • SR 11-7 MRM document questions and answers for RAG evaluation.
  • Evaluates: RAG, LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 7
• Stanford Healthcare Regulations
  • Test suite Stanford Healthcare Regulations for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 122
• Stanford Healthcare Regulations (multi choice)
  • Test suite Stanford Healthcare Regulations (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 67
• Stanford Healthcare Regulations (question type)
  • Test suite Stanford Healthcare Regulations (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 114
• Summeval Summarization
  • Test suite for the summarization evaluation.
  • Evaluates: LLM
  • Purposes: summarization
  • Tests: 1
  • Test Cases: 100
• tamil-news-classification
  • This dataset is used to classify a static set of Tamil News. The task is to classify news to its respective category. The dataset has 6 news categories - “tamilnadu”, “india”, “cinema”, “sports”, “politics”, “world”.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 3631
• tamil-thirukural
  • This dataset is used to test the comprehension abilities for the Thirukkural. Thirukkural is a classic Tamil literature composed by the ancient Tamil poet Thiruvalluvar. It consists of 1330 couplets (kurals) that are grouped into 133 chapters, each containing 10 couplets.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 266
• tanglish-tweets-SA
  • Code-mixed Tamil and English tweets curated for the sentiment analysis task.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 1163
• Technical Report
  • Test suite Technical Report for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 26
• Technical Report (multi choice)
  • Test suite Technical Report (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 50
• Technical Report (question type)
  • Test suite Technical Report (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 82
• Telcom Customer Service Information
  • Test suite Telcom Customer Service Information for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 141
• Telcom Customer Service Information (multi choice)
  • Test suite Telcom Customer Service Information (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 69
• Telcom Customer Service Information (question type)
  • Test suite Telcom Customer Service Information (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 133
• Telecom Infrastructure Planning
  • Test suite Telecom Infrastructure Planning for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 154
• Telecom Infrastructure Planning (multi choice)
  • Test suite Telecom Infrastructure Planning (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 98
• Telecom Infrastructure Planning (question type)
  • Test suite Telecom Infrastructure Planning (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 153
• Telecommunication Regulations
  • Test suite Telecommunication Regulations for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 117
• Telecommunication Regulations (multi choice)
  • Test suite Telecommunication Regulations (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 74
• Telecommunication Regulations (question type)
  • Test suite Telecommunication Regulations (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 114
• Telecommunications regulation strategy policy
  • Test suite Telecommunications regulation strategy policy for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 120
• Telecommunications regulation strategy policy (multi choice)
  • Test suite Telecommunications regulation strategy policy (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 75
• Telecommunications regulation strategy policy (question type)
  • Test suite Telecommunications regulation strategy policy (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 116
• truthfulqa
  • TruthfulQA aims to measure the the truthfulness of a model.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 817
• TruthfulQA (MCQ Version)
  • TruthfulQA aims to measure the the truthfulness of a model. This dataset uses a multiple choice format.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 483
• uciadult
  • The UCI Adult dataset has been used widely used to assess fairness.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 32561
• uciadult
  • The UCI adult dataset, created in 1996, is used to train models to predict whether a person’s income will exceed $50K/yr based on census data. Also known as “Census Income” dataset.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 32561
• UPC Agreement
  • Test suite UPC Agreement for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 153
• UPC Agreement (multi choice)
  • Test suite UPC Agreement (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 66
• UPC Agreement (question type)
  • Test suite UPC Agreement (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 141
• US Veterans Affairs
  • Test suite US Veterans Affairs for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 127
• US Veterans Affairs (multi choice)
  • Test suite US Veterans Affairs (multi choice) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 29
• US Veterans Affairs (question type)
  • Test suite US Veterans Affairs (question type) for RAG evaluation.
  • Evaluates: RAG
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 121
• winobias-variation1
  • This dataset contains gender-bias based on the professions from the Labor Force Statistics (https://www.bls.gov/cps/cpsaat11.htm), which contain some gender-bias.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 396
• Winogrande
  • This dataset is used for commonsense reasoning, expert-crafted pronoun resolution problems designed to be unsolvable for statistical models.
  • Evaluates: LLM
  • Purposes: Q&A
  • Tests: 1
  • Test Cases: 41665