» Current | 2022 | 2021 | 2020

Linley Newsletter

Cerebras Performs Extreme Training

September 21, 2021

Author: Markus Levy

Cerebras hopes to meet the training demands of huge natural-language neural networks with its new MemoryX and SwarmX systems, which boast the capacity to store up to 120 trillion weights along with the ability to distribute the workload to as many as 192 CS-2 accelerators. Although the second-generation wafer-scale (WSE2) processor in the CS-2 can store up to 10 billion FP32 parameters in its massive on-chip SRAM—enough to train most neural networks—leading-edge NLP networks are far larger.

The company’s solution is to disaggregate the compute and storage functions. Instead of holding weights in the WSE2’s on-chip SRAM, MemoryX stores them in a DRAM/flash-memory system that can hold trillions of weights. The storage system also includes processors that execute the calculations for weight iteration and optimization, enabling those operations to run locally instead of spending more cycles and power transferring them back to the WSE2’s compute units.

The WSE2’s only peripheral links are high-speed serial interfaces, which enable data to stream directly into the Swarm mesh network that connects the chip’s compute cores. The new SwarmX boxes virtually extend that network, so MemoryX can broadcast weights in parallel to multiple CS-2s. In the backpropagation direction, SwarmX collects and reduces the gradients produced by the CS-2s, transmitting the results to MemoryX to initiate the next iteration.

Cerebras withheld the MemoryX and SwarmX performance and power specifications, but it says for extremely large neural networks, the disaggregated system enables training throughput to scale linearly with the number of CS-2s. Although the company also withheld pricing, the new MemoryX and SwarmX systems are available for shipping to customers.

Subscribers can view the full article in the Microprocessor Report.

Subscribe to the Microprocessor Report and always get the full story!

Free Newsletter

Linley Newsletter
Analysis of new developments in microprocessors and other semiconductor products


Linley Spring Processor Conference 2022
Registration now open: April 20-21, 2022
Hyatt Regency Hotel, Santa Clara, CA
Linley Fall Processor Conference 2021
Held October 20-21, 2021
Proceedings available
Linley Spring Processor Conference 2021
April 19 - 23, 2021
Proceedings Available
More Events »