Model: Gemini 3.1 - Zero-shot
Abstract
Gemini zero-shot prompting
This solution that uses the gemini-3.1-pro-preview model. It is a zero-shot prompting approach.
Key prompt used:
Classify the accuracy of the following student answer based on the question.
Definitions:
- Correct: the answer is correct and fully answers the question.
- Partially_correct_incomplete: everything stated in the answer is correct and the answer responds to part of the question, but does not fully answer the question.
- Contradictory: the answer contradicts scientifically correct information.
- Irrelevant: the answer contains domain content but does not respond to any part of the question.
- Non_domain: if the student utterance does not include domain content, e.g., "I don't know", "what the book says", "you are stupid".
Decision rules:
- If the student answer is scientifically complete and correct, choose `correct`.
- If the answer has some relevant truth but is missing required information, choose `partially_correct_incomplete`.
- If the core claim conflicts with the scientifically correct information, choose `contradictory`.
- If the answer contains domain words but does not answer this question, choose `irrelevant`.
- If the answer is "I don't know", abusive, conversational, or otherwise lacks domain content, choose `non_domain`.
Here is one example of a complete, correct answer to the question: "{reference_answer_text}"
question: "{question_text}", student answer: "{student_answer_text}".
Scores
| Name | Weighted F1 | Classification rate |
|---|---|---|
|
Gemini 3.1 - Zero-shot
|
0.4619 | 0.4562 |
SAGE: Science Answer Grading & Evaluation: Rules and Terms of Data Use
DrivenData Benchmark Rules
GUIDELINES
These guidelines are intended to summarize a few main features of the benchmark rules. Full details are provided in the benchmark rules below.
One account per participant
You cannot sign up to DrivenData from multiple accounts and therefore you cannot submit from multiple accounts.
Submission limits
The rate of allowed submissions is restricted to a fixed value on a per-benchmark basis. Any attempt to circumvent this limit will result in submission removal and account ban.
External data with rights
External data is permitted in this Benchmark provided Participants have all rights, licenses and permissions to use the external data as contemplated in the Benchmark Rules, and that such data complies with any external data restrictions stated on the Benchmark Website. Any use of external data must be pursuant to and in accordance with a valid license that permits use of the external data in this Benchmark without the need for additional approvals from any third party.
Source code and models
Closed-source code and models are allowed. Participants are encouraged but not required to submit models that are able to be released under a permissive open-source license (e.g., CC0 or CC-BY).
Questions?
The benchmark rules are in place to promote fair benchmarking and useful solutions. If you are ever unsure whether your solution meets the benchmark rules, ask the organizers in the benchmark forum or send an email to info@drivendata.org.
FULL RULES
NO PURCHASE NECESSARY TO PARTICIPATE IN THIS BENCHMARK.
BY PARTICIPATING IN THIS BENCHMARK, PARTICIPANT AGREES TO THESE BENCHMARK RULES, WHICH ARE A CONTRACT, SO READ THEM CAREFULLY BEFORE PARTICIPATING.
Definitions
These are the complete, official rules (the "Benchmark Rules") for the benchmark (the "Benchmark") described on the website listed above or linking to these Benchmark Rules (the "Benchmark Website"). The Benchmark Rules incorporate by reference the contents of the Benchmark Website listed above.
By downloading a dataset linked from the Benchmark Website, submitting an entry to this Benchmark (a "Submission"), or joining a Team in this Benchmark, you are agreeing to be bound by these Benchmark Rules which constitute a binding agreement between you and DrivenData and, if applicable, any rules and restrictions that may be imposed by the third-party sponsor of this benchmark (the "Benchmark Sponsor").
- "Data" means the data or datasets linked from the Benchmark Website for the purpose of use by Participants in the Benchmark. For the avoidance of doubt, Data includes any prototype or executable code provided to Participants by DrivenData or via the Benchmark Website.
- "Submission" means any code, model artifacts, documentation, or other materials submitted to the Benchmark Website for evaluation.
Benchmark host and period
The Benchmark is hosted by DrivenData, LLC ("DrivenData"). The Benchmark will run according to the dates listed on the Benchmark Website (the "Benchmark Period").
Participation
The Benchmark is open to all participants, subject to applicable laws and any restrictions listed on the Benchmark Website. You are solely responsible for complying with your employer's policies regarding participation. Participants (including each member of a Team) must not be under any legal or contractual obligation that would prohibit their participation in this Benchmark as described in these Benchmark Rules.
If participating in this Benchmark would result in a violation by Participant of any law applicable to them or any agreement to which they are a party or employer or company policies and procedures, such Participant is ineligible.
If you are participating within the scope of your employment, as an employee, contractor, or agent of a legal entity, you warrant that such party has full knowledge of your actions and has consented thereto.
BY PARTICIPATING IN THE BENCHMARK YOU ACCEPT THE CONDITIONS STATED IN THESE BENCHMARK RULES INCLUDING THE RULES ON THE BENCHMARK WEBSITE AND WARRANT THAT YOU ARE ELIGIBLE TO PARTICIPATE IN THE BENCHMARK.
IF YOU DO NOT ACCEPT ALL OF THESE BENCHMARK RULES, THEN PLEASE DO NOT REGISTER FOR THE BENCHMARK.
WE RECOMMEND THAT YOU PRINT OUT A COPY OF THESE BENCHMARK RULES FOR YOUR FUTURE REFERENCE.
Registration and teams
To participate, register through the Benchmark Website within the Benchmark Period. You may participate individually or as part of a team. Each participant must have a single, unique DrivenData account and may not participate on more than one team.
Each registered individual or Team is referred to as a Participant.
You may only participate using a single, unique DrivenData account registered at http://www.drivendata.org.
Participating using more than one DrivenData account per individual is a breach of these Benchmark Rules and DrivenData reserves the right to remove any individual (or Team including an individual) who is found to breach these Benchmark Rules.
Multiple individuals may collaborate as a Team. You may not participate on more than one Team. Each Team member must be a single individual operating a separate DrivenData account.
You must register individually for the Benchmark before joining a Team. You must confirm your Team membership to make it official by responding to the Team notification message which will be sent to your Account.
Submissions
Each Submission must be uploaded to the Benchmark Website in the manner and format specified on the Benchmark Website. All Submissions must be received during the Benchmark Period.
Submissions must comply with all applicable laws and regulations and must not include content that infringes third-party intellectual property rights.
Submission criteria
User submissions must in their entirety comply with all applicable federal, state, local, and international laws and regulations.
Without limiting the foregoing, in order to be eligible, you warrant that your Submission for this Benchmark meets the following Submission Criteria, all as determined in the sole discretion of Benchmark Sponsor and/or DrivenData:
- Does not include or anticipate the inclusion of any content that is in violation of or infringes third party intellectual property rights including, but not limited to copyrights, including music copyrights, patents, trade secrets, trademarks, and rights of publicity or privacy.
- Is free and clear of all liens, claims, encumbrances or demands of any third parties.
- Does not include or anticipate the inclusion of any unsuitable or offensive content, including nudity, sexually explicit, disparaging, libelous or other inappropriate content.
- Is suitable for a general audience.
- Does not contain any claims that are not and cannot be substantiated or that would be false and/or misleading to a reasonable consumer.
Participants retain all intellectual property rights in their Submissions.
By submitting, you grant DrivenData a limited, non-exclusive license to run, evaluate, and display your Submission for the administration of the Benchmark and for public benchmarking results (e.g., leaderboard display).
Evaluation and results
The evaluation metric used for scoring and ranking Submissions will be displayed on the Benchmark Website.
Scores displayed on the public leaderboard while the benchmark is running may or may not be the same as the final scores on the private leaderboard, depending on how samples from the Data are used for evaluation.
Data use and code sharing
DATA
Participants must use the Data only as permitted by these Benchmark Rules and any associated data use rules specified on the Benchmark Website.
Participants are responsible for ensuring that any external data, code, or models they use comply with the applicable licenses and terms.
Unless otherwise permitted by the terms of the Benchmark Website, Participants must use the Data solely for the purpose and duration of the Benchmark.
Participants explicitly agree to make no attempt to utilize the Data to identify specific individuals.
Participants agree to notify DrivenData immediately upon learning of any possible unauthorized transmission or unauthorized access of the Data.
EXTERNAL DATA
Participants’ use of external data is subject to the External Data Rule specified in the Benchmark Guidelines and any rules on external data expressly stated on the Benchmark Website.
NO EXTERNAL DATA
Unless otherwise expressly stated on the Benchmark Website, Participants must not use data other than the Data to develop and test their models and Submissions.
OPEN EXTERNAL DATA
Participants may use external data other than the Data, including pre-trained models, provided that:
- they comply with any external data restrictions stated on the Benchmark Website
- the external data is freely and publicly available
- the data is under a permissive open license that does not prohibit commercial use
EXTERNAL DATA WITH RIGHTS
Participants may use external data other than the Data, including pre-trained models, provided that the Participant has all rights, licenses and permissions to use the external data as contemplated in these Benchmark Rules.
Participants agree to provide DrivenData with proof of such license upon request.
Failure to satisfy these requirements may result in removal from the Benchmark.
Additional restrictions
Unless otherwise expressly stated on the Benchmark Website:
- Participants may annotate provided training data as long as they are included with solutions to enable reproduction and do not overfit to the test set.
- Participants may not add any manual annotations to the provided test data.
- Participants may use test data for unsupervised learning.
- Participants may not access, reconstruct, or disclose the test data in any form.
Any attempt to access or disclose the test data will result in immediate submission removal and account ban.
Participant warranties and obligations
By registering, you agree that:
- your Account is complete, correct and accurate
- your registration may be rejected or terminated if your Account information is inaccurate
You are solely responsible for your Account.
Participation is subject to all applicable laws and regulations.
You are responsible for:
- checking applicable laws in your jurisdiction
- abiding by your employer's policies
- obtaining necessary equipment and internet access
By entering a Submission, you represent and warrant that your Submission:
- complies with all applicable laws
- is your own original work or properly credited
- does not violate intellectual property rights
- does not contain malicious code
- does not violate any law or regulation
A breach of any of these warranties will result in the corresponding Submission being invalid.
Cheating
Participating using more than one DrivenData account is deemed cheating and may result in:
- removal from the Benchmark
- bans or deactivation of accounts
DrivenData reserves the right to request information associated with suspected cheating investigations.
Failure to respond within ten (10) days is grounds for removal.
Delivery and receipt of entries
DrivenData is not responsible for:
- late or lost Submissions
- transmission failures
- data corruption
- unauthorized access
- email delivery failures
- damage to computer systems
Reservation of rights
DrivenData reserves the right to:
- remove any Submission or Participant violating the rules
- cancel, terminate, or modify the Benchmark
- remove Participants engaging in inappropriate or illegal behavior
General release
By participating in the Benchmark, you release and discharge any Benchmark Sponsor and DrivenData from liability related to the Benchmark.
You also agree to indemnify Benchmark Sponsor and DrivenData from losses arising from your participation or breach of these rules.
Limitations of liability
By participating, each Participant agrees that:
- disputes will be resolved individually
- claims are limited to actual out-of-pocket costs
- punitive or consequential damages are waived
The Released Parties are not liable for:
- lost or damaged entries
- technical malfunctions
- transmission errors
- disruptions beyond their control
- injuries or damages from participation
- printing or typographical errors
Disclaimer of warranties
The Benchmark Website and all related services are provided on an:
"AS IS" and "AS AVAILABLE" basis
without warranties of any kind.
The Released Parties disclaim all warranties including:
- merchantability
- non-infringement
- fitness for a particular purpose
Governing Law and Jurisdiction
DrivenData shall be the sole interpreter of these Benchmark Rules.
These Benchmark Rules are governed by the laws of the Commonwealth of Massachusetts.
Legal proceedings must be brought in:
- federal courts of the United States, or
- courts of the Commonwealth of Massachusetts
The invalidity of any provision does not affect the remainder of the Benchmark Rules.