Overview

Recent advances in generative AI are opening up new possibilities for education, particularly when it comes to reducing the burden of time-consuming tasks. One area of growing interest is the assessment of student writing, which can be both labor-intensive and critical for understanding student learning. By helping teachers evaluate written responses more efficiently, AI-powered assessment tools could free up valuable time for individualized instruction, allowing educators to focus more attention on students who need it most and ultimately supporting better learning outcomes.

At the same time, assessing student writing remains a challenging task for AI. Student responses often express the same ideas in different ways, contain incomplete reasoning, or mix correct concepts with misconceptions. Accurately evaluating these responses requires models to reason about scientific concepts and relationships rather than simply matching keywords or phrases.

This benchmark evaluates how well different pre-trained models and modeling approaches can assess students' short-answer science responses. Given a question, a reference answer, and a students' response, models must determine whether the response is correct, partially correct, contradictory, irrelevant, or non-domain. By measuring performance on this task, the benchmark provides insight into current capabilities of AI systems for educational assessment.

How to use this benchmark

The benchmark leaderboard provides a snapshot of how well current AI models can evaluate student-written science responses. By comparing performance across a range of modeling approaches, it highlights both the progress and remaining challenges in automated assessment.

Educators, researchers, and developers can use the leaderboard to understand which models are most effective at reasoning about student explanations and distinguishing between different types of responses. Benchmark performance can serve as a useful reference point when deciding which pre-trained model to use:

As a baseline to build on in research to push performance forwards
In a real-world use cases to help inform human decision makers

This benchmark is not yet open for users to add new model submissions.

Benchmark task

This benchmark is based on the SemEval dataset, which contains short-answer science questions, reference answers, and graded student responses collected from an online tutoring system. Questions span a variety of science domains, including electricity, electronics, geology, chemistry, and experimental design.

The goal is to predict one of five assessment categories: correct, partially correct, contradictory, irrelevant, or non-domain.

An example question, reference answer, and graded student responses.

Success on this task requires both understanding the scientific concepts being discussed, and accurately interpreting a student response.

References

Any work produced from this data or benchmark should site the original works:

Dzikovska, M. O., Nielsen, R., Brew, C., Leacock, C., Giampiccolo, D., Bentivogli, L., Clark, P., Dagan, I., & Dang, H. T. (2013). SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013), held in conjunction with the Second Joint Conference on Lexical and Computational Semantics (SEM 2013). Association for Computational Linguistics. Atlanta, Georgia, USA, June 13–14. https://aclanthology.org/S13-2045/
Dzikovska, M., Nielsen, R., & Leacock, C. (2015). The joint student response analysis and recognizing textual entailment challenge: Making sense of student responses in educational applications. Language Resources and Evaluation. 50. 10.1007/s10579-015-9313-8. https://doi.org/10.1007/s10579-015-9313-8

SAGE: Science Answer Grading & Evaluation

Navigation

Overview

How to use this benchmark

Benchmark task

References