Metric: mcq_eval

This metric is designed to evaluate MCQ generations tasks.

If this is a text-based metric, make sure to wrap you input in double quotes. Alternatively you can use a JSON-formatted list as input.

Metric Card for MCQ_eval

Metric Description

MCQ_eval aims to evaluate LM performances in a MCQ context where it has to predict a single letter.

How to Use

This metric takes 2 mandatory arguments : generations (a list of string), golds (a list of string containing only one letter).

import evaluate
mcq_eval = evaluate.load("rfr2003/mcq_eval")
results = mcq_eval.compute(generations=["A", "B"], golds=["A", "D"])
print(results)
{'accuracy': 0.5}

This metric doesn't have any optional argument.

Output Values

This metric outputs a dictionary with the following values:

accuracy: The number of correct generations, which ranges from 0.0 to 1.0.

Values from Popular Papers

Examples

MCQ answers normally range from A to D but you can use any set of letters.

import evaluate
mcq_eval = evaluate.load("rfr2003/mcq_eval")
results = mcq_eval.compute(generations=["A", "B"], golds=["A", "D"])
print(results)
{'accuracy': 0.5}

import evaluate
mcq_eval = evaluate.load("rfr2003/mcq_eval")
results = mcq_eval.compute(generations=["J", "K"], golds=["U", "K"])
print(results)
{'accuracy': 0.5}