Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions MLPerf_Results_Messaging_Guidelines.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,22 @@ SmartAI Corp announced an unverified score of 0.3 on the MLPerf® Training Image
[1] MLPerf® v0.5 Training ResNet-v1.5. Result not verified by MLCommons Association. Unverified results have not been through an MLPerf review and may use measurement methodologies and/or workload implementations that are inconsistent with the MLPerf specification for verified results. The MLPerf name and logo are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.
____

== Performance optimizations must not be used to disparage or diminish* valid MLPerf results

All valid MLPerf submissions satisfy the same accuracy or quality requirements for that benchmark, regardless of the performance optimizations used. Submitters disclose details of their optimizations, including but not limited to lowest numerical precision (e.g., FP8, FP4), parallelism strategies, communication overlaps, or compiler techniques, to increase transparency and provide more insights. Whether or not such disclosure is made, the following rules apply:

- Submitters may not imply or state that a valid result is inferior, does not sufficiently converge, or is otherwise less valid due to the performance optimizations used.

pass:[*]disparage or diminish means to negatively position another submitter's results

**Non-Compliant Example**:

____
LesserSmartAI Corp will not compare against BF16 results submitted by SmartAI Corp because BF16 recipes converge poorly.
____

This violates these Guidelines because it implies that valid MLPerf results achieved using BF16 precision are less legitimate than other higher results although both meet the same required accuracy target and are equally valid MLPerf submissions.

== When comparing MLPerf Results, use official MLPerf power metrics

System power measured using the MLPerf Power methodology is the only MLCommons officially-sanctioned power metric to be used for the purposes of portraying MLPerf results and/or making comparisons. When stating or comparing MLPerf power metrics results:
Expand Down
Loading