Challenge

Trace the Ace

Predict the learning gains from a tutoring session measured by quiz performance in this tutoring outcomes prediction challenge

Tasks
Tutoring
Data types
Human transcript
Tabular
$50,000 in prizes
4 weeks left model submissions
7 weeks left solution write-up
639 joined
Image generated using Gemini
Join competition

Navigation

Code submission format

This benchmark uses code execution to evaluate models. Rather than submitting your predicted labels, you'll package everything needed to perform inference and submit that for containerized execution. The runtime repository contains the complete specification for the runtime.

All submissions for inference must run on Python 3.12. No other languages or versions of Python are supported.

What to submit

Your final submission should be a zip archive named with the extension .zip (for example, submission.zip). The root level of the submission.zip file must contain a main.py which performs inference on the test data and writes the predictions to a file named submission.csv in the same directory as main.py.

Here's an example of the unpacked contents of a zipped submission:

submission root directory
├── assets/
│   └── ... (all assets needed for inference, e.g., model weights)
└── main.py         # Inference script

Be sure that when you unzip the submission, main.py exists in the folder where you unzip. This file must be present at the root level of the zip archive. There should be no folder that contains it.

During code execution, your submission will be unzipped in our cloud compute cluster. In that cluster, your inference code in main.py will be run on the test set features to generate test set predictions. While your submission is running, logging output will be limited to a max of 500 lines, with a max of 500 characters per line.

Available data

During inference, your script will have access to the following directory structure:

submission root directory
├── data/
│   ├── submission_format.csv
│   ├── test_features.csv
│   └── test_transcripts
│       ├── {session_id_1}.csv
│       ├── ...
│       └── {session_id_n}.csv
├── main.py
└── <additional assets included in the submission archive>

The structure of test_features.csv and test_transcripts/ exactly matches that of the training data features.

submission_format.csv demonstrates what response IDs and columns should be included in your predictions. Participants must submit predictions as probabilities between 0 and 1 representing the likelihood that the student answered the next question correctly. For example, the first few rows of your submission might be:

response_id,probability
aaafrcy,0.5
aabbmio,0.1
aabpxat,0.9

Final structure

At the end of inference, your submission root directory should look like this:

submission root directory
├── data/
│   ├── submission_format.csv
│   ├── test_features.csv
│   └── test_transcripts
│       ├── {session_id_1}.csv
│       ├── ...
│       └── {session_id_n}.csv
├── main.py
├── submission.csv   # NEW FILE GENERATED BY YOUR SUBMISSION
└── <additional assets included in the submission archive>

Testing your submission

The machines for executing your inference code are a shared resource across competitors, so please be conscientious in your use of them. Before you make a full submission, you should:

  1. Test locally
  2. Test in the limited smoke test environment
  3. Once your submission runs successfully both locally and in the smoke test environment, go to the "Code jobs" tab to make a full competition submission

Make sure to cancel any jobs that will run longer than the allowed time. The allowed time is 6 hours for a full submission, and 10 minutes for a smoke test submission. Smoke tests, cancelled jobs, and failed jobs won't count against your submission limit. You will be limited to three full submissions per week.

Testing your submission locally

In order to ensure that your submission runs inference successfully and to debug any issues, you should first and foremost test your submission locally using the just test-submission command in the runtime container repository. This is a great way to work out any bugs locally and ensure that your model will run quickly enough.

Smoke tests

For additional debugging, we provide a "smoke test" environment that replicates the test inference runtime but runs only on a small set of responses. In the smoke test environment, the test data structure is the same (test_features.csv, test_transcripts/, and submission_format.csv). But the available data represents a sample of 100 responses drawn from the training set, rather than the full test set data.

Submission requirements

Key requirements checklist

  • Submission includes main.py in the root directory of the zip. There can be extra files with more code that is called (see assets folder in the example).
  • Submission contains any model weights that need to be loaded. There will be no network access inside the runtime environment.
  • Submission does not print or log any information about the test dataset, including excerpts of transcripts, learning objective descriptions, and/or aggregations such as sums, means, or token counts. Doing so may be grounds for disqualification based on the requirement that test samples are processed independently.
  • The main.py script loads the data for inference from the data folder. This folder is read-only.
  • The main.py script writes submission.csv to the root directory when inference is finished. The format of this file must match the submission format exactly: each row corresponds to one response, with columns for response_id and probability.

Runtime details

Your code is executed within a container that is defined in our runtime repository. The limits are as follows:

  • Your submission must be written in Python and run Python 3.12 using the packages defined in the runtime repository.
  • The submission must complete execution in 6 hours or less. We expect most submissions will complete much more quickly and computation time per participant will be monitored to prevent abuse. If you find yourself requiring more time than this limit allows, open a GitHub issue in the repository to let us know. Smoke test submissions must complete in 10 minutes or less.
  • The container will not have network access. All necessary files (code and model assets) must be included in your submission.
  • Your submission zip archive should not exceed 60GB, including all code and model assets. Larger submissions are likely to fail. This size limit may be increased in the future. Any updates will be reflected here.
  • The container runtime has access to a single GPU. All of your code should run within the GPU environments in the container, even if actual computation happens on the CPU.
  • The container has access to 24 vCPUs powered by an AMD EPYC™ 7V13 chip and 220GB RAM.
  • The container has 1 NVIDIA A100 GPU with 80GB VRAM.
  • The container execution will not have root access to the filesystem.

Requesting package installations

Since the Docker container will not have network access, all packages must be pre-installed. The runtime is built using uv, PyTorch, and vLLM backed by CUDA 12.9. We are happy to add packages as long as they do not conflict and can build successfully. Packages must be available through the supported build process for Python 3.12. To request an additional package be added to the Docker image, follow the instructions in the runtime repository.

Happy building! Once again, if you have any questions or issues you can always head on over to the user forum!