DrivenData Benchmark Rules

GUIDELINES

These guidelines are intended to summarize a few main features of the benchmark rules. Full details are provided in the benchmark rules below.

One account per participant

You cannot sign up to DrivenData from multiple accounts and therefore you cannot submit from multiple accounts.

Submission limits

The rate of allowed submissions is restricted to a fixed value on a per-benchmark basis. Any attempt to circumvent this limit will result in submission removal and account ban.

External data with rights

External data is permitted in this Benchmark provided Participants have all rights, licenses and permissions to use the external data as contemplated in the Benchmark Rules, and that such data complies with any external data restrictions stated on the Benchmark Website. Any use of external data must be pursuant to and in accordance with a valid license that permits use of the external data in this Benchmark without the need for additional approvals from any third party.

Source code and models

Closed-source code and models are allowed. Participants are encouraged but not required to submit models that are able to be released under a permissive open-source license (e.g., CC0 or CC-BY).

Questions?

The benchmark rules are in place to promote fair benchmarking and useful solutions. If you are ever unsure whether your solution meets the benchmark rules, ask the organizers in the benchmark forum or send an email to info@drivendata.org.

FULL RULES

NO PURCHASE NECESSARY TO PARTICIPATE IN THIS BENCHMARK.

BY PARTICIPATING IN THIS BENCHMARK, PARTICIPANT AGREES TO THESE BENCHMARK RULES, WHICH ARE A CONTRACT, SO READ THEM CAREFULLY BEFORE PARTICIPATING.

Definitions

These are the complete, official rules (the "Benchmark Rules") for the benchmark (the "Benchmark") described on the website listed above or linking to these Benchmark Rules (the "Benchmark Website"). The Benchmark Rules incorporate by reference the contents of the Benchmark Website listed above.

By downloading a dataset linked from the Benchmark Website, submitting an entry to this Benchmark (a "Submission"), or joining a Team in this Benchmark, you are agreeing to be bound by these Benchmark Rules which constitute a binding agreement between you and DrivenData and, if applicable, any rules and restrictions that may be imposed by the third-party sponsor of this benchmark (the "Benchmark Sponsor").

• "Data" means the data or datasets linked from the Benchmark Website for the purpose of use by Participants in the Benchmark. For the avoidance of doubt, Data includes any prototype or executable code provided to Participants by DrivenData or via the Benchmark Website.
• "Submission" means any code, model artifacts, documentation, or other materials submitted to the Benchmark Website for evaluation.

Benchmark host and period

The Benchmark is hosted by DrivenData, LLC ("DrivenData"). The Benchmark will run according to the dates listed on the Benchmark Website (the "Benchmark Period").

Participation

The Benchmark is open to all participants, subject to applicable laws and any restrictions listed on the Benchmark Website. You are solely responsible for complying with your employer's policies regarding participation. Participants (including each member of a Team) must not be under any legal or contractual obligation that would prohibit their participation in this Benchmark as described in these Benchmark Rules.

If participating in this Benchmark would result in a violation by Participant of any law applicable to them or any agreement to which they are a party or employer or company policies and procedures, such Participant is ineligible.

If you are participating within the scope of your employment, as an employee, contractor, or agent of a legal entity, you warrant that such party has full knowledge of your actions and has consented thereto.

BY PARTICIPATING IN THE BENCHMARK YOU ACCEPT THE CONDITIONS STATED IN THESE BENCHMARK RULES INCLUDING THE RULES ON THE BENCHMARK WEBSITE AND WARRANT THAT YOU ARE ELIGIBLE TO PARTICIPATE IN THE BENCHMARK.

IF YOU DO NOT ACCEPT ALL OF THESE BENCHMARK RULES, THEN PLEASE DO NOT REGISTER FOR THE BENCHMARK.

WE RECOMMEND THAT YOU PRINT OUT A COPY OF THESE BENCHMARK RULES FOR YOUR FUTURE REFERENCE.

Registration and teams

To participate, register through the Benchmark Website within the Benchmark Period. You may participate individually or as part of a team. Each participant must have a single, unique DrivenData account and may not participate on more than one team.

Each registered individual or Team is referred to as a Participant.

You may only participate using a single, unique DrivenData account registered at http://www.drivendata.org.

Participating using more than one DrivenData account per individual is a breach of these Benchmark Rules and DrivenData reserves the right to remove any individual (or Team including an individual) who is found to breach these Benchmark Rules.

Multiple individuals may collaborate as a Team. You may not participate on more than one Team. Each Team member must be a single individual operating a separate DrivenData account.

You must register individually for the Benchmark before joining a Team. You must confirm your Team membership to make it official by responding to the Team notification message which will be sent to your Account.

Submissions

Each Submission must be uploaded to the Benchmark Website in the manner and format specified on the Benchmark Website. All Submissions must be received during the Benchmark Period.

Submissions must comply with all applicable laws and regulations and must not include content that infringes third-party intellectual property rights.

Submission criteria

User submissions must in their entirety comply with all applicable federal, state, local, and international laws and regulations.

Without limiting the foregoing, in order to be eligible, you warrant that your Submission for this Benchmark meets the following Submission Criteria, all as determined in the sole discretion of Benchmark Sponsor and/or DrivenData:

• Does not include or anticipate the inclusion of any content that is in violation of or infringes third party intellectual property rights including, but not limited to copyrights, including music copyrights, patents, trade secrets, trademarks, and rights of publicity or privacy.
• Is free and clear of all liens, claims, encumbrances or demands of any third parties.
• Does not include or anticipate the inclusion of any unsuitable or offensive content, including nudity, sexually explicit, disparaging, libelous or other inappropriate content.
• Is suitable for a general audience.
• Does not contain any claims that are not and cannot be substantiated or that would be false and/or misleading to a reasonable consumer.

Participants retain all intellectual property rights in their Submissions.

By submitting, you grant DrivenData a limited, non-exclusive license to run, evaluate, and display your Submission for the administration of the Benchmark and for public benchmarking results (e.g., leaderboard display).

Evaluation and results

The evaluation metric used for scoring and ranking Submissions will be displayed on the Benchmark Website.

Scores displayed on the public leaderboard while the benchmark is running may or may not be the same as the final scores on the private leaderboard, depending on how samples from the Data are used for evaluation.

Data use and code sharing

DATA

Participants must use the Data only as permitted by these Benchmark Rules and any associated data use rules specified on the Benchmark Website.

Participants are responsible for ensuring that any external data, code, or models they use comply with the applicable licenses and terms.

Unless otherwise permitted by the terms of the Benchmark Website, Participants must use the Data solely for the purpose and duration of the Benchmark.

Participants explicitly agree to make no attempt to utilize the Data to identify specific individuals.

Participants agree to notify DrivenData immediately upon learning of any possible unauthorized transmission or unauthorized access of the Data.

EXTERNAL DATA

Participants’ use of external data is subject to the External Data Rule specified in the Benchmark Guidelines and any rules on external data expressly stated on the Benchmark Website.

NO EXTERNAL DATA

Unless otherwise expressly stated on the Benchmark Website, Participants must not use data other than the Data to develop and test their models and Submissions.

OPEN EXTERNAL DATA

Participants may use external data other than the Data, including pre-trained models, provided that:

• they comply with any external data restrictions stated on the Benchmark Website
• the external data is freely and publicly available
• the data is under a permissive open license that does not prohibit commercial use

EXTERNAL DATA WITH RIGHTS

Participants may use external data other than the Data, including pre-trained models, provided that the Participant has all rights, licenses and permissions to use the external data as contemplated in these Benchmark Rules.

Participants agree to provide DrivenData with proof of such license upon request.

Failure to satisfy these requirements may result in removal from the Benchmark.

Additional restrictions

Unless otherwise expressly stated on the Benchmark Website:

• Participants may annotate provided training data as long as they are included with solutions to enable reproduction and do not overfit to the test set.
• Participants may not add any manual annotations to the provided test data.
• Participants may use test data for unsupervised learning.
• Participants may not access, reconstruct, or disclose the test data in any form.

Any attempt to access or disclose the test data will result in immediate submission removal and account ban.

Participant warranties and obligations

By registering, you agree that:

• your Account is complete, correct and accurate
• your registration may be rejected or terminated if your Account information is inaccurate

You are solely responsible for your Account.

Participation is subject to all applicable laws and regulations.

You are responsible for:

• checking applicable laws in your jurisdiction
• abiding by your employer's policies
• obtaining necessary equipment and internet access

By entering a Submission, you represent and warrant that your Submission:

• complies with all applicable laws
• is your own original work or properly credited
• does not violate intellectual property rights
• does not contain malicious code
• does not violate any law or regulation

A breach of any of these warranties will result in the corresponding Submission being invalid.

Cheating

Participating using more than one DrivenData account is deemed cheating and may result in:

• removal from the Benchmark
• bans or deactivation of accounts

DrivenData reserves the right to request information associated with suspected cheating investigations.

Failure to respond within ten (10) days is grounds for removal.

Delivery and receipt of entries

DrivenData is not responsible for:

• late or lost Submissions
• transmission failures
• data corruption
• unauthorized access
• email delivery failures
• damage to computer systems

Reservation of rights

DrivenData reserves the right to:

• remove any Submission or Participant violating the rules
• cancel, terminate, or modify the Benchmark
• remove Participants engaging in inappropriate or illegal behavior

General release

By participating in the Benchmark, you release and discharge any Benchmark Sponsor and DrivenData from liability related to the Benchmark.

You also agree to indemnify Benchmark Sponsor and DrivenData from losses arising from your participation or breach of these rules.

Limitations of liability

By participating, each Participant agrees that:

1. disputes will be resolved individually
2. claims are limited to actual out-of-pocket costs
3. punitive or consequential damages are waived

The Released Parties are not liable for:

• lost or damaged entries
• technical malfunctions
• transmission errors
• disruptions beyond their control
• injuries or damages from participation
• printing or typographical errors

Disclaimer of warranties

The Benchmark Website and all related services are provided on an:

"AS IS" and "AS AVAILABLE" basis

without warranties of any kind.

The Released Parties disclaim all warranties including:

• merchantability
• non-infringement
• fitness for a particular purpose

Governing Law and Jurisdiction

DrivenData shall be the sole interpreter of these Benchmark Rules.

These Benchmark Rules are governed by the laws of the Commonwealth of Massachusetts.

Legal proceedings must be brought in:

• federal courts of the United States, or
• courts of the Commonwealth of Massachusetts

The invalidity of any provision does not affect the remainder of the Benchmark Rules.