The SemEval dataset originated from the Student Response Analysis corpus (SRA corpus), which combines BEETLE and SciEntsBank subsets. The BEETLE corpus contains 56 short-answer questions on basic electricity and electronics concepts with approximately 3,000 student responses. The SciEntsBank corpus contains approximately 197 short-answer questions across 15 science domains with approximately 10,000 student answers. Questions represent explanation and definition questions typically seen in science assignments and tests.
BEETLE data was originally captured from students interacting with BEETLE II tutoring system. SciEntsBank data was created from student answers to assessment questions collected by Nielsen et al. (2008).
Opt-out was available
No
Data processing
Student responses were evaluated with 3-way and 5-way correctness schemes.
The 5-way scheme categorized answers into one of five categories:
Contradictory: answer directly contradicted the reference answer.
Irrelevant: answer discussed domain content without the necessary information.
Non_domain: answers without any domain content (e.g. I don't know).
The 3-way scheme only used correct, incorrect, or contradictory.
The BEETLE corpus was manually labeld by human-annotators. The SciEntsBank corpus were first automatically labeled using a set of question-specific heuristics then manually revised for correctness.
In preparing the SRA data for the SemEval competition, additional quality checks were performed. Questions relying on external material (e.g. charts and graphs) and questions with more than one possible answer were removed from the corpus.
The data were split into train and test sets. To test models' ability to generalize across scenarios, three types of test set were created:
The dataset has two subsets of questions: BEETLE and SciEntsBank. BEETLE contains questions on basic electricity and electronics concepts. For lower-level questions (grades 6 - 12), use the SciEntsBank subset.
Appropriate uses and limitations
Some questions in this corpus are above grade school level. It may be beneficial to subset the data to questions at lower levels. Models trained on college-level data may not translate to an elementary context.
SemEval-2013 Task 7: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge
Myroslava Dzikovska1
,
Rodney Nielsen2
,
Chris Brew3
,
Claudia Leacock4
,
Danilo Giampiccolo5
,
Luisa Bentivogli6
,
Peter Clark7
,
Ido Dagan8
,
Hoa Trang Dang9
1University of Edinburgh
2University of North Texas
3Nuance Communications
4CTB McGraw-Hill
5CELCT
6FBK
7Vulcan Inc.
8Bar-Ilan University
9NIST
Organization
Vulcan Inc., Bar-Ilan University, ACL-SIGLEX, University of Edinburgh, NIST, FBK, University of North Texas, Nuance Communications, CTB McGraw-Hill, CELCT
Authors
Chris Brew, Claudia Leacock, Danilo Giampiccolo, Peter Clark, Ido Dagan, Rodney Nielsen, Myroslava Dzikovska, Hoa Trang Dang, Luisa Bentivogli
University of Edinburgh
University of North Texas
Nuance Communications
CTB McGraw-Hill
CELCT
FBK
NIST