Microsoft Introduces 128-Core Arm CPU for Cloud and Custom AI Accelerator

[AleksandarK]

During its Ignite conference, Microsoft introduced a duo of custom-designed silicon made to accelerate AI and excel in cloud workloads. First of the two is Microsoft's Azure Cobalt 100 CPU, a 128-core design that features a 64-bit Armv9 instruction set, implemented in a cloud-native design that is set to become a part of Microsoft's offerings. While there aren't many details regarding the configuration, the company claims that the performance target is up to 40% when compared to the current generation of Arm servers running on Azure cloud. The SoC has used Arm's Neoverse CSS platform customized for Microsoft, with presumably Arm Neoverse N2 cores.

The next and hottest topic in the server space is AI acceleration, which is needed for running today's large language models. Microsoft hosts OpenAI's ChatGPT, Microsoft's Copilot, and many other AI services. To help make them run as fast as possible, Microsoft's project Athena now has the name of Maia 100 AI accelerator, which is manufactured on TSMC's 5 nm process. It features 105 billion transistors and supports various MX data formats, even those smaller than 8-bit bit, for maximum performance. Currently tested on GPT 3.5 Turbo, we have yet to see performance figures and comparisons with competing hardware from NVIDIA, like H100/H200 and AMD, with MI300X. The Maia 100 has an aggregate bandwidth of 4.8 Terabits per accelerator, which uses a custom Ethernet-based networking protocol for scaling. These chips are expected to appear in Microsoft data centers early next year, and we hope to get some performance numbers soon.



View at TechPowerUp Main Site | Source

 

[AsRock]

I hope it works better thas the WinChip, and as seen as AI is all bs in the first place.

 

[dragontamer5788]

ARM Neoverse is very intriguing. Now that ARM cores are a truly commodity design, all the big players can custom make a chip (albeit at very high costs, maybe $X00 Million or so). But for companies like Microsoft who own cloud-datacenters with maybe a million servers, it could very well be worth the costs.

 

[Denver]

ARM Neoverse is very intriguing. Now that ARM cores are a truly commodity design, all the big players can custom make a chip (albeit at very high costs, maybe $X00 Million or so). But for companies like Microsoft who own cloud-datacenters with maybe a million servers, it could very well be worth the costs.

It's a huge cost that only these multi-billion dollar companies can risk, the problem is that they are still inferior in many aspects to AMD's x86 designs. Who knows if this is paying off in the end.

 

[bonehead123]

And just think, after they fully implement these cpu's into their operations, they will make Windows a cloud-only product, and start charging a proverbial arm & a leg (& probably a kidney too) for it just to recoup their development costs.....

Yea, you go, Microsloth

 

[dragontamer5788]

It's a huge cost that only these multi-billion dollar companies can risk, the problem is that they are still inferior in many aspects to AMD's x86 designs. Who knows if this is paying off in the end.

Its a huge cost that used to be even huger.

Before ARM Neoverse, if you wanted to custom create your own high performance cores, you'd need like $10 Billion and like 10,000 employees designing just the core, creating the compiler / OS / etc. etc.. Dropping the costs below the $1 Billion mark opened up the chips business to many more companies (ie: from like 5 companies to maybe 20 companies). Its still requires behemoth-sized Microsoft-like companies to design a high performance core (Amazon, Microsoft, etc. etc.), but its way easier today than ever before.

Total costs for R&D of this Cobalt is probably just $500 Million, maybe far less than that. And Microsoft is absolutely the kind of company where I can imagine ~1-million computers being deployed to Azure alone. All Microsoft needs to do is save $500 per computer/server and this entire venture is worthwhile. Just napkin math. I don't know how much Microsoft really spent on R&D or their servers. But you can see how quickly this works out for them.

Rerun the math for a decade or two ago with a $15 Billion R&D project, and something like that would never pay for itself.

 

[Nanochip]

The arm invasion is coming. The question is will x86 survive as the dominant pc/server platform, or will arm achieve dominance and strong arm x86?

 

[Daven]

The arm invasion is coming. The question is will x86 survive as the dominant pc/server platform, or will arm achieve dominance and strong arm x86?

Why can’t both be true? A heterogeneous, diverse hardware ecosystem is the best possible outcome. The only one who stands to lose from such an arrangement is Intel. The Intel business model requires market dominance in order to keep so many fabs running 24/7/365 while maintaining the 60%+ gross margins its board demands. The only way that will happen is for Intel to turn to a 100% fab for hire business model and stop making its own chips.

Intel fabs are needed to make all those different architectures at the volume growth required. No serious players will ever fab using Intel due to competitive secrecy.

 

[Easy Rhino]

It probably spies on you and forces ad pop-ups to access RAM

 

[Beermotor]

Why can’t both be true? A heterogeneous, diverse hardware ecosystem is the best possible outcome. The only one who stands to lose from such an arrangement is Intel. The Intel business model requires market dominance in order to keep so many fabs running 24/7/365 while maintaining the 60%+ gross margins its board demands. The only way that will happen is for Intel to turn to a 100% fab for hire business model and stop making its own chips.

Intel fabs are needed to make all those different architectures at the volume growth required. No serious players will ever fab using Intel due to competitive secrecy.

Yep 100%. ARM vs x64 isn't a zero-sum game and many of us can remember that a lot of the progress in the 90s was because there was solid competition in the CPU space from RISC vendors.

I'm pretty sure this isn't necessarily as much about ARM being superior as it simply is Microsoft wanting other options in the event that neither Intel or AMD can overcome the current (no pun intended) x64 power and thermal challenges. It's really expensive to power and cool a datacenter full of 400 watt Xeons, so if you can only get 80% of the performance for 50% of the power it's a net win.

 

[Nanochip]

Why can’t both be true? A heterogeneous, diverse hardware ecosystem is the best possible outcome. The only one who stands to lose from such an arrangement is Intel. The Intel business model requires market dominance in order to keep so many fabs running 24/7/365 while maintaining the 60%+ gross margins its board demands. The only way that will happen is for Intel to turn to a 100% fab for hire business model and stop making its own chips.

Intel fabs are needed to make all those different architectures at the volume growth required. No serious players will ever fab using Intel due to competitive secrecy.

Because it costs billions of dollars to fab a cpu generation on a given fabrication node. And as the nodes shrink, the wafer costs increase. So companies will necessarily have to be choosy as to which technology they deploy and why.

the question I have is: what will consumers choose and why? There is one future where both x86 and arm coexist. But there’s also other potential futures where one dominates due to perhaps superior power efficiency and the other languishes. Of course, one major thing that the incumbent x86 has going for it is software compatibility…

Ever since Apple divorced Intel, more and more companies are considering arm. Microsoft. Amazon. Qualcomm Oryon is coming, and many of the major laptop vendors have announced a partnership...dell hp ASUS etc. it could become the case where there are multiple arm vendors that enter, further eroding intel’s sales.

I think Intel may be in trouble. It has to get power consumption in check quickly. Because if nvidia enters the arena, with a strong cpu + GPU offering, I could see many gamers going for the nvidia solution. Right now nvidia is focused on ai and is dominating there. It’s also dominating in GPU sales as well. Intel may just end up being relegated to not much more than the TSMC of the USA. Microsoft had a major AI keynote yesterday and mentioned nvidia and AMD but barely spoke of Intel…