The Azure Maia 100 and Cobalt 100 chips are the first two custom silicon chips designed by Microsoft for its cloud infrastructure.
The rumors are true: Microsoft has built its own custom AI chip that can be used to train large language models and potentially avoid a costly reliance on Nvidia. Microsoft has also built its own Arm-based CPU for cloud workloads. Both custom silicon chips are designed to power its Azure data centers and ready the company and its enterprise customers for a future full of AI.
Microsoft’s Azure Maia AI chip and Arm-powered Azure Cobalt CPU are arriving in 2024, on the back of a surge in demand this year for Nvidia’s H100 GPUs that are widely used to train and operate generative image tools and large language models. There’s such high demand for these GPUs that some have even fetched more than $40,000 on eBay.
“Microsoft actually has a long history in silicon development,” explains Rani Borkar, head of Azure hardware systems and infrastructure at Microsoft, in an interview with The Verge. Microsoft collaborated on silicon for the Xbox more than 20 years ago and has even co-engineered chips for its Surface devices. “These efforts are built on that experience,” says Borkar. “In 2017, we began architecting the cloud hardware stack and we began on that journey putting us on track to build our new custom chips.”
The new Azure Maia AI chip and Azure Cobalt CPU are both built in-house at Microsoft, combined with a deep overhaul of its entire cloud server stack to optimize performance, power, and cost. “We are rethinking the cloud infrastructure for the era of AI, and literally optimizing every layer of that infrastructure,” says Borkar.
The Azure Cobalt CPU, named after the blue pigment, is a 128-core chip that’s built on an Arm Neoverse CSS design and customized for Microsoft. It’s designed to power general cloud services on Azure. “We’ve put a lot of thought into not just getting it to be highly performant, but also making sure we’re mindful of power management,” explains Borkar. “We made some very intentional design choices, including the ability to control performance and power consumption per core and on every single virtual machine.”
Microsoft is currently testing its Cobalt CPU on workloads like Microsoft Teams and SQL server, with plans to make virtual machines available to customers next year for a variety of workloads. While Borkar wouldn’t be drawn into direct comparisons with Amazon’s Graviton 3 servers that are available on AWS, there should be some noticeable performance gains over the Arm-based servers Microsoft is currently using for Azure. “Our initial testing shows that our performance is up to 40 percent better than what’s currently in our data centers that use commercial Arm servers,” says Borkar. Microsoft isn’t sharing full system specifications or benchmarks yet.