MLCommons continues to revise and expand its AI benchmarks, now offering a set specifically designed for microcontrollers and ultra-low-power accelerators. These simple models consume less than 512KB, typically fitting in on-chip memory. The small set of initial submissions, however, highlighted a limitation of the TinyML movement: most microcontroller CPUs aren’t particularly efficient for AI.

The initial version of the MLPerf Tiny benchmark, designated version 0.5, comprises four tests, all representative of the basic tasks that low-cost chips often handle. One detects keywords, which smart speakers, smartphones, and other voice-activated devices often use for wake words such as “Alexa” and “Hey Siri.” This DS-CNN model requires only 53KB. A second performs the visual equivalent, detecting whether a person is visible in an image; it’s helpful for doorbell cameras and occupancy detection. The 325KB MobileNet model operates on low-resolution 96x96-pixel images.

For image classification, the benchmark includes a simplified ResNet model that has only 96KB of parameters. It uses the CIFAR-10 database to place each tiny 32x32 image into one of 10 categories. The final test detects anomalies in an audio stream. It emulates monitoring of industrial equipment and employs a 270KB FC-AutoEncoder model. All models are pretrained using TensorFlow Lite and quantized to 8-bit integer (INT8) weights.