Challenge

Where the Wild Features Are

Predict kindergarteners' end-of-year reading scores from literacy screening audio, and find which speech features carry signal beyond correctness scores

Tasks
Reading assessment
Data types
Audio
Tabular
$50,000 in prizes
8 weeks left model submissions
53 joined
Adapted from a photo by Guy Basabose on Unsplash

Problem description

Your task is to predict each student's end-of-year reading score from audio recordings, screening scores (item scores and task ability estimates), and metadata from their beginning-of-year literacy screening tasks, and to identify which acoustic features carry signal. Each sample is one kindergarten student.

This competition has two phases. In Phase 1, you will submit predictions for the test set, scored on a leaderboard. In Phase 2, the top 15 teams on the private leaderboard will submit a write-up explaining what their models learned from the audio. Prizes are decided by the write-ups, so keep the research question in mind while you model:

  • Which features of children's speech predict reading outcomes beyond the screening scores, and how much do they add?
  • Do these features come from the child's speech, and not from other sounds in the clips, such as adult speech, background noise, or room acoustics?
  • Do your findings hold across students, schools, and the two outcome assessments?

See the write-up requirements for details. Notably, write-ups are scored on model interpretability, so plan to show which features drive your predictions.

Not sure where to start? Read over the basic steps in the "How to compete" section of the home page.

The features in this dataset


The data come from the Reach Every Reader literacy screener. See the about page for background on the screening tasks, the outcome assessments, and how the audio was anonymized.

The dataset has 24,518 audio clips from 902 students. Students are split between the training and test sets, and all of a student's clips are in the same set.

Training features are provided in three components (the test set is described below):

.
├── train_audio
│   ├── 0008441d5a6b.flac
│   ├── ...
│   └── ffff3c7388fd.flac
├── train_clip_metadata.csv
└── train_student_metadata.csv

The audio is packaged as tar archives on the data download page: train_audio.tar for the training set and test_audio.tar for the test set. Extracting them gives the train_audio/ and test_audio/ folders. train_audio.tar is about 1.2 GB and test_audio.tar is about 810 MB.

Audio files

train_audio/ and test_audio/ contain one denoised and anonymized .flac file per clip, in 16-bit mono. FLAC is lossless, and common audio libraries such as soundfile, librosa, and torchaudio read it directly. All clips are provided at 16 kHz, resampled from the 24 kHz output of anonymization. The original_sample_rate column in the student metadata gives the sample rate of each student's raw recordings. Each clip is one student's response to one screener item. The file name is an opaque clip ID. Clips range from under 1 second to about 60 seconds.

The audio reflects real screening conditions. Clips may include background noise, adult speech, or no speech at all, and some clips may not contain the child's answer. Denoising and anonymization can also distort the audio, sometimes enough that the child's answer is hard to recognize.

Clip metadata

train_clip_metadata.csv has one row per clip, with the following columns:

  • filename (str) - audio file name in train_audio/
  • subject_id (str) - unique student identifier, which links to the student metadata and labels
  • task (str) - the screening task: blending, deletion, or sentence_repetition
  • expected_text (str) - the correct response: the target word for blending and deletion, or the full sentence for sentence_repetition
  • score_status (str) - scored, or the reason the clip has no score: no_score_value, no_score_row, proctor_ended_task, or no_option_selected. Training data only.
  • item_score (float) - 1 if the administrator scored the response correct, 0 if incorrect, missing if unscored
  • duration_sec (float) - clip length in seconds

A small number of clips have no item score, for example because the administrator ended the session. They are kept to reflect real-world screening. In the training data, score_status gives the reason; in the test data, these clips have a missing item_score.

Student metadata

train_student_metadata.csv has one row per student, with the following columns:

  • subject_id (str) - unique student identifier
  • instrument (str) - the end-of-year assessment used for the outcome score: ktea3 or wj4
  • original_sample_rate (int) - sample rate of the student's raw recordings in Hz: 11025 or 16000. All clips are provided at 16 kHz.
  • blending_theta (float) - task-level ability estimate for blending from item response theory (IRT), missing if the task was not given or no estimate is available
  • deletion_theta (float) - task-level IRT ability estimate for deletion, missing if the task was not given or no estimate is available
  • sentence_repetition_theta (float) - task-level IRT ability estimate for sentence_repetition, missing if the task was not given or no estimate is available
  • school_id (int) - anonymized school identifier, from 1 to 17. Training data only.

Not every student completed all three tasks. Thetas from the two data collection efforts are on a common scale, and are rounded to 2 decimal places.

School IDs are not provided for the test set, so your model cannot use school as an input at prediction time. Teams invited to Phase 2 will receive additional data, including test school IDs, to analyze school effects in their write-ups. See the write-up requirements for details.

Feature data example


For example, a single row in `train_clip_metadata.csv` has these values:

filename0009bcd3f93a.flac
subject_id26b900a80bbb
taskdeletion
expected_textSand
score_statusscored
item_score0.0
duration_sec2.70
And the corresponding student has these values in `train_student_metadata.csv`:
subject_id26b900a80bbb
instrumentktea3
original_sample_rate16000
blending_theta-1.63
deletion_theta-1.45
sentence_repetition_theta-1.03
school_id10

Labels


The label is each student's end-of-year standard score on a letter-word reading assessment, either KTEA-3 Letter & Word Recognition or WJ IV Letter-Word Identification. Scores from the two assessments are pooled into one target. The instrument column in the student metadata tells you which assessment each student took.

train_labels.csv has one row per student, with the following columns:

  • subject_id (str) - unique student identifier
  • outcome_score (int) - end-of-year standard score

For example, the first few rows of train_labels.csv are:

subject_id,outcome_score
002067c5badc,104
01a87157fa7d,88
02c6ac193043,89
...

Test set


Test features are provided in the same format as the training features, as test_audio/, test_clip_metadata.csv, and test_student_metadata.csv, with two differences:

  • test_clip_metadata.csv does not include score_status.
  • test_student_metadata.csv does not include school_id.

Test labels are not provided. You will submit predictions for every student in test_student_metadata.csv.

Teams invited to Phase 2 will receive the test labels and other additional data for their write-ups. See Additional data on the write-up page.

Use of test data

Your predictions must be generated entirely by code, with no manual input. Hand-labeling, manual transcription, or listening to test clips to inform predictions is not allowed. Test data may not be used to train models, including through pseudo-labeling. See the competition rules for details.

External data and models

External data and pretrained models, including pretrained speech models such as wav2vec2, Whisper, and the Pasketti children's speech models, are allowed if they are freely and publicly available to all participants under a permissive open license that does not prohibit commercial use, such as CC0 or CC-BY.

You may not upload competition data to any third-party services or APIs that retain the data. For example, you cannot submit data to ChatGPT, Claude, or Gemini. You can use external models by loading open-source model weights into an environment that you can wipe the data from afterwards, such as your local machine or a cloud compute environment. See the competition rules for details.

Performance metric


Performance is evaluated using the coefficient of determination, also known as R². R² measures the share of variance in the true outcome scores that your predictions explain. A higher value is better. The maximum is 1, and a model that always predicts the mean of the true test scores gets 0.

$$R^2 = 1 - \frac{\sum_{i=1}^{N} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{N} (y_i - \bar{y})^2}$$

Where:

  • |$N$| is the number of students
  • |$y_i$| is the true end-of-year score for student |$i$|
  • |$\hat{y}_i$| is the predicted score for student |$i$|
  • |$\bar{y}$| is the mean of the true scores

While Phase 1 is open, the public leaderboard shows each submission's score on a subset of the test set. When Phase 1 closes, the private leaderboard is revealed with final scores. Rankings on the two leaderboards may differ.

The top 15 teams on the private leaderboard at the close of Phase 1 will be invited to Phase 2. Among invited teams, final rankings are decided by the write-ups, not by leaderboard rank.

Submission format


Submit one predicted end-of-year score per test student, as a CSV file with the same columns as the training labels:

  • subject_id (str) - unique student identifier, matching the test student metadata
  • outcome_score (float) - predicted end-of-year standard score

Predictions must be numeric, and can be whole numbers or decimals. Missing values are not allowed. Your submission must include exactly one row for every student in submission_format.csv, in the same order. You can make up to 3 submissions per day.

For example, if you predicted that every test student had a score of 100, your `.csv` file would look like:
subject_id,outcome_score
0034c8197ebb,100.0
00d50a63bfa3,100.0
0201e89889e2,100.0
...
fb956ef61011,100.0
fc77cda73377,100.0
ff47b17d3828,100.0

Good luck!


Good luck and enjoy this problem! If you have any questions you can always visit the competition forum!