Challenge

Trace the Ace

Predict the learning gains from a tutoring session measured by quiz performance in this tutoring outcomes prediction challenge

Tasks
Tutoring
Data types
Human transcript
Tabular
$50,000 in prizes
3 weeks left model submissions
6 weeks left solution write-up
729 joined
Image generated using Gemini

Publication bonus

Authored by the National Tutoring Observatory

Judges will invite top teams to participate in the publication bonus prize program. Invited teams will be eligible to receive a publication bonus prize by submitting a publishable-quality preprint submission. Up to 9 teams will receive a publication bonus prize of $2,000 and will be invited to publish in a journal special issue.

Background


As digital learning environments increasingly shift toward conversational interfaces—powered by both human and artificial intelligence—there is a growing need for robust, discourse-based models of student learning. In response, the National Tutoring Observatory and DrivenData are hosting a competition to develop models that predict proximal learning outcomes from human tutoring dialogue. Following the competition, we will invite selected participants to submit original research to a special issue. This issue seeks to move beyond 'black-box' predictive accuracy, prioritizing research that uncovers the underlying mechanisms and generalizable components that ensure model reliability across diverse educational contexts

Focus & Scope

We are seeking submissions that provide technical explanations of modeling approaches while offering insights into the mechanisms connecting educational discourse to proximal learning outcomes. Papers should provide a rigorous post-hoc analysis of models developed during the competition. Successful submissions must go beyond reporting performance metrics, ablation studies, and subgroup analyses. Instead, they should also explain what underlying features the model captures as a 'signal' and how the integration of contextual metadata clarifies which components of the model are truly generalizable versus those that are context-specific.

We are particularly interested in papers that address the following:

  • Mechanisms of Prediction: Provide a detailed explanation of why your model predicts student performance. What features or latent representations are driving the results? What sequences of tutor and learner behaviors indicate the student is learning?
  • Educational Implications: Discuss how the interpretability of your model can inform pedagogical interventions or policy decisions.
  • Metadata Integration: An analysis of how additional metadata (e.g., demographics, school-level variables, or temporal data) informs the model’s generalizability.
  • Generalizability Components: How does your model handle distribution shift? Use the provided metadata and cross-contextual data to identify which components of your model remain robust (or fail) when applied to different student populations, subjects, or instructional settings.
How to submit

Publication bonus prize submissions will be emailed directly to the National Tutoring Observatory. Detailed submission instructions will be provided to eligible participants.

Evaluation


Evaluation Criteria

Submissions will be judged based on the following criteria:

  • Scientific Value: Submissions should provide findings that advance the science of teaching and learning.

  • Technical Validation: Authors must demonstrate that their findings are grounded in the competition data while offering new insights not covered in the initial competition report.

  • Methodological Rigor: We encourage the use of interpretability frameworks (e.g., SHAP, LIME, or counterfactual explanations) and robust cross-validation techniques.

Peer Review Evaluation Process
  • All submissions will undergo a rigorous double-blind peer review where the identities of both the authors and the reviewers remain completely confidential.
  • Reviewers will include academic and industry experts in AI tutoring
  • You must remove all author names, institutional affiliations, acknowledgments, and funding sources from the main manuscript file.
  • Cite your own prior work in the third person (e.g., use "As shown by Smith (2025)..." instead of "As we showed previously...").

Up to 9 submissions that show promise for full publication will receive publication bonus prizes. These submissions will also be invited to submit to a journal special issue. Note that submissions will still undergo a standard peer-review process before being published. Receiving a publication bonus prize is not a guarantee of publication.

Submission Requirements


File Format and Requirements

The manuscript should be structured as a standard academic research paper.

  • Include an abstract of no more than 250 words.
  • The main text must fall between 7,000 and 10,000 words (excluding references, tables, and appendices).
  • All submissions will undergo double-blind peer review. Please ensure your manuscript is fully anonymized before submitting.
  • Submit your document as either a PDF or Microsoft Word file (.doc / .docx). We recommend using the ACM LaTeX or Word templates, or the ScienceDirect LaTeX template.
  • Please thoroughly spell-check and grammar-check your manuscript to ensure a polished final submission.
Use of AI in the manuscript writing process
  • Allowed Uses: You may use AI tools to improve language, readability, and organization, or to help synthesize complex literature and identify research gaps during the writing process.
  • Mandatory Disclosure: If you use AI for manuscript preparation (such as making substantive changes to sentence structure or organization), you must include an AI declaration statement upon submission. This statement must detail the name of the tool, its purpose, and the extent of human oversight.
  • Exemptions: Basic AI-assisted checks for grammar, spelling, and punctuation do not require a disclosure statement.
  • Human Responsibility: Authors are fully responsible and accountable for the accuracy, originality, and integrity of the entire manuscript. You must verify all AI-generated output to ensure there are no factual errors, biases, or incomplete data.