OpenAI Could Make Custom Chips to Power Next-Generation AI Models

[AleksandarK]

OpenAI, the company behind ChatGPT and the GPT-4 large language model, is reportedly exploring the possibility of creating custom silicon to power its next-generation AI models. According to Reuters, Insider sources have even alluded to the firm evaluating potential acquisitions of chip design firms. While a final decision is yet to be cemented, conversations from as early as last year highlighted OpenAI's struggle with the growing scarcity and escalating costs of AI chips, with NVIDIA being its primary supplier. The CEO of OpenAI, Sam Altman, has been rather vocal about the shortage of GPUs, a sector predominantly monopolized by NVIDIA, which holds control over an astounding 80% of the global market for AI-optimized chips.

Back in 2020, OpenAI banked on a colossal supercomputer crafted by Microsoft, a significant investor in OpenAI, which harnesses the power of 10,000 NVIDIA GPUs. This setup is instrumental in driving the operations of ChatGPT, which, as per Bernstein's analyst Stacy Rasgon, comes with its own hefty price tag. Each interaction with ChatGPT is estimated to cost around 4 cents. Drawing a comparative scale with Google search, if ChatGPT queries ever burgeoned to a mere tenth of Google's search volume, the initial GPU investment would skyrocket to an overwhelming $48.1 billion, with a recurring annual expenditure of approximately $16 billion for sustained operations. For an invitation to comment, OpenAI declined to provide any statements. The potential entry into the world of custom silicon signals a strategic move towards greater self-reliance and cost optimization so further development of AI can be sustained.



View at TechPowerUp Main Site | Source

 

[Space Lynx]

AI hasn't done shit for me yet, so all this can fuck off imo

 

[thesmokingman]

Lmao, such a big market is suddenly shrinking fast with the biggest consumers of compute pivoting to make their own AI chips because they are getting raped by the slightly decent gpu supplier whom currently has the market all to themselves but not for long.

 

[SJZL 2.0]

OpenAI has relied on Volta GPUs to power their models. But when you consider the usability of Volta GPUs in gaming & workstation applications, its no wonder that OpenAI want to create processors entirely for AI models and have no usability elsewhere.

 

[thesmokingman]

OpenAI has relied on Volta GPUs to power their models. But when you consider the usability of Volta GPUs in gaming & workstation applications, its no wonder that OpenAI want to create processors entirely for AI models and have no usability elsewhere.

They're on the cluster MSFT built with 10K H100s now. They want to inhouse compute because Nvidia one can't supply the demand and two they are raking it in at over 70% margin on the accelerators. Some are guessing its over 80% now.

Microsoft and Nvidia team up to build new Azure-hosted AI supercomputer | TechCrunch

Microsoft and Nvidia have announced a multi-year partnership to build a new AI supercomputer in the Azure cloud.

techcrunch.com

 

[AusWolf]

So it's not just gamers that complain about (Nvidia) GPU prices. This explains a lot.

 

[fancucker]

Nvidia needs to capitalize on its lead and develop purely AI focused silicon

 

[cvaldes]

So it's not just gamers that complain about (Nvidia) GPU prices. This explains a lot.

Lots of people complain about the price of ASICs, not just AI chips. And not just now, basically since the first silicon rolled off a manufacturing line.

In fact, people will complain about pretty much everything, whether it's a diamond ring, roll of toilet paper, eggs, gasoline (petrol), vintage French champagne, sack of fertilizer, 80" OLED television, whatever.

We live in a free market economy. Nvidia (and other chipmakers) can charge what the market will bear. In this sense, Nvidia is no different than that bee farmer charging (insert price) for a jar of organic honey.

Hell, a lot of people who saw the first Model A roll out of Henry Ford's factory said, "Too expensive, I'll stick with my horse and wagon."

 

[Space Lynx]

We live in a free market economy.

you sweet little thing, no we don't. I have a hundred examples as to why, but I will let you live in bliss, for ignorance is bliss, I assure you.

 

[cvaldes]

you sweet little thing, no we don't.

Well, it's closer to a free market economy than some sort of socialized, government-mandated market. What we have today isn't pure but nothing ever is.

Even if the government dictates the price of something, there will be a grey or black market that is closer to a free market system. People will find a way to make a buck whether it's spices, liquor, pharmaceuticals, cigarettes, ASICs, whatever.

Ask any economist whether or not the Western world is basically a free market economy. The short answer will always be a "Yes, but..." reply. There are more things in the "Yes" portion than what follows after "but..."

Anyhow, there's a finite number of silicon wafers. Chip manufacturers will allocate more wafers to high margin products because they can. In the context of this particular thread, this is definitely a free market. Nvidia could quit the machine learning market and just make really cheap GPUs for gaming. But they don't.

And as an indirect shareholder of all of the major chip designers (AMD, Intel, Nvidia, Apple, Qualcomm, etc.) I am glad that they behave like it is a free market economy. My retirement plan depends on them doing so.

And if you are an American with a retirement plan, undoubtedly you will benefit from this as well at some point.

 

[Space Lynx]

Well, it's closer to a free market economy than some sort of socialized, government-mandated market. What we have today isn't pure but nothing ever is.

I would argue just the opposite, Finland is more socialized than any other nation state, and its free markets are lovely. Been there several times, seen it first hand.

 

[thesmokingman]

Lmao, we don't need a treatise on free markets.

 

[Deleted member 185088]

Makes sense to create custom silicon, more efficiency and stop relaying on 3rd party.

 

[cvaldes]

Lmao, we don't need a treatise on free markets.

No, we do not. However the original post points out that there are possible cost efficiencies in rolling your own AI silicon rather than buying at the market.

The US government certainly isn't telling Nvidia how much to price their AI accelerators.

Right now it's basically this:

AI customer: "How much is this?"
Nvidia: "How much you got?"
AI customer: "I don't like your attitude."
Nvidia: "We have competitors. Ask them about pricing and availability."
AI customer: "But your stuff is better. And you have a better dev platform. And our staff is already familiar with your products."
Nvidia: "Don't worry, we'll sell what we have to someone."

It's important to note that Apple makes their own machine learning silicon (the Neural Engine in their marketing speak), but for their own use. They have already done what the original post basically describes since about 2017. And they probably started working on in earnest back in 2010.

There are no Nvidia Tensor cores in my iPhone, iPad, and Mac. Apple is not affected by the market's demand for Nvidia's AI products.

Silicon Valley was built by people who left larger companies to start their own thing, hoping to do better than the status quo. There's nothing new about OpenAI's thoughts. Google, Meta, Amazon, they all have teams working on custom silicon designs.

 

[Mpt]

I'm going to quote Will Smith from I robot.
"Robots building robots, now that's just stupid"

 

[Tropick]

AI hasn't done shit for me yet, so all this can fuck off imo

Biggest impact AI has had on me so far is the latest W11 AI update breaking my goddamn Adreanlin drivers so I can definitely relate to this sentiment.

Can't wait for OpenAI's custom chips to allow advertisers to stuff my face with food ads at an unprecedented 2 nanoseconds after I realize I'm hungry. Verily I say unto you we have entered the next golden age of technology

 

[evernessince]

Well, it's closer to a free market economy than some sort of socialized, government-mandated market. What we have today isn't pure but nothing ever is.

In the US at least the economy is definitely heavily socialized. The difference is it's socialism for the rich. Nvidia and Intel and going to be taking in CHIPS act money. They also received tax breaks for retaining employees during the pandemic, received free money in the form of PPP loans, and additional tax benefits just from the pandemic alone. 76% of pandemic relief money during the pandemic went to businesses.

America has bailed out both the big car manufacturer's and big banks. America is a mixed market but really it's the poster child for corporate socialism. The government is always there for big business, meanwhile everyone else has to foot the bill. Companies that constantly exploit the "free market" to jack up prices precisely when people are struggling the most, meanwhile stealing tax dollars from those people's pockets while lowering their own tax bill to as little as possible.

Even if the government dictates the price of something, there will be a grey or black market that is closer to a free market system. People will find a way to make a buck whether it's spices, liquor, pharmaceuticals, cigarettes, ASICs, whatever.

This is not always true, there has to be demand for said product outside conditions set by the government. For example, if the government sets prices on non-controlled drugs it's unlikely that a black market will form given that it's the governments controls that are making those more affordable. There are also instances where the risk factor or additional cost overhead of black market goods doesn't make sense. The profit has to be there in order for people to take the risk is selling something illegally.

Ask any economist whether or not the Western world is basically a free market economy. The short answer will always be a "Yes, but..." reply. There are more things in the "Yes" portion than what follows after "but..."

The short answer is nearly every 1st world country is a mixed market. There's no real world example of a free market, doesn't exist because it would quickly kill itself. A free market must operate independent of any government regulation but companies inherently rely on the government to protect it's IP and trademarks in order to operate and they rely on the government to ensure reasonable market conditions. In addition a free market requires that none of the participants wield influence over the market itself, which is almost never the case. A Free Market is not something that could exist in real life because it's an idea that ignores reality.

Anyhow, there's a finite number of silicon wafers. Chip manufacturers will allocate more wafers to high margin products because they can. In the context of this particular thread, this is definitely a free market. Nvidia could quit the machine learning market and just make really cheap GPUs for gaming. But they don't.

Inherently not a free market due to government regulations, tax breaks, public money, protection of IP, ect. In addition, certain chip vendors wield significant influence over the market (Nvidia, Intel) and have used that influence in the past to leverage their own market positions while hindering competition. CUDA in particular but any of Nvidia's proprietary technologies inhibit new competitors from entering the GPU market and competing on free market values. On top of that Nvidia and AMD hold a ton of patents that cripple even the possiblity of designing a GPU without getting sued out of existence. Ditto goes for Intel and AMD's x86 CPU patents.

Out of all the markets you could have chosen to call a free market, you probably picked the least free. From memory, CPUs, to GPUs these related markets have all had frequent issues with anti-competitive practices, entrenchment, and pricing issues.

 

[HisDivineOrder]

From the moment Nvidia saw dollar signs with AI, I thought they'd price themselves right out of this market and force companies to start up competitors to drive them out, which is exactly how all this ends. It isn't even the first time this has happened to Nvidia. And this ends with Nvidia and AMD back to making discrete GPU's. They should just probably stick to doing it in the first place because they're risking someone like Intel sneaking in the night and stealing all their golden geese while they're off daydreaming. Intel's already closing in. If AMD had any sense, they would have done it years ago, but Lisa's got Zen on the brain and now she's fallen prey to the same trap as Jen with AI.

But like always they'll find these get-rich-quick schemes evaporate some year and imperil their stock's consistency and I sincerely hope that they turn around and find gamers have left them both behind this time, too, because a viable alternative finally arrived.

 

[thesmokingman]

I'm going to quote Will Smith from I robot.
"Robots building robots, now that's just stupid"

That's ironic lol. Bots building bots is exactly Tesla's business model.

That said CNBS did a piece on all the AI compute consumers starting their own AI chip projects. MSFT and OpenAI are jined at the hip so they're in it together. Amazon and Google are doing it as well. AMD is releasing their accelerators in Q4. And of course Tesla who saw this coming years ago wasn't mentioned by CNBS (no surprise) and are already ramping their AI chip Dojo up. Nvidia best enjoy their captive market while it can.

 

[eidairaman1]

Biggest impact AI has had on me so far is the latest W11 AI update breaking my goddamn Adreanlin drivers so I can definitely relate to this sentiment.

Can't wait for OpenAI's custom chips to allow advertisers to stuff my face with food ads at an unprecedented 2 nanoseconds after I realize I'm hungry. Verily I say unto you we have entered the next golden age of technology

Disable windows update completely and vette your updates, you can use powershell to install only needed updates

 

[kondamin]

4cents per query wtf.
that is unbelievably expensive.

i can understand they want their very own asic to run the mature model for the plebs that does things a bit more efficiently.
i doubt an O365 account would cover the cost of its doing windows search.

 

[AusWolf]

Hell, a lot of people who saw the first Model A roll out of Henry Ford's factory said, "Too expensive, I'll stick with my horse and wagon."

Well, charging a lot for something new and experimental is one thing, but charging more and more for something that people are used to getting for pennies is another.

Other than that, I agree. People like to complain. But sometimes, there is a valid reason behind the complaints.

 

[thesmokingman]

4cents per query wtf.
that is unbelievably expensive.

i can understand they want their very own asic to run the mature model for the plebs that does things a bit more efficiently.
i doubt an O365 account would cover the cost of its doing windows search.

You've clearly missed the point that these H100s cost $40K a piece and neural nets don't start working well till you get the gpu count over the thousands. Then there's the upkeep price which is insane. Any idea the cost to power and maintain a 10K gpu cluster buddy?

They want their own AI chip because they SEE that Elon's Dojo is providing the same performance as a 10K gpu cluster at 1/6th the cost and 1/4 the footprint.

People must think AI is free or something. There are huge costs for this level of compute and that's not even mentioning that the whole industry is compute constrained.

 

[Assimilator]

Every time I see these "other companies are just going to invert their own cheap AI chips and then NVIDIA will die" posts I LOL at the inability to understand basic facts. Guess what, designing your own silicon is difficult and guess what, those companies are going to be headbutting the exact same wall of lack of foundry capacity that NVIDIA is.

What is actually happening is NVIDIA is telling these companies "we can't get capacity but if you're willing to buy some of Apple's allocation at TSMC for <stupid price>, we'll happily make those GPUs for you" and these companies are going "HOLY SHIT THAT'S EXPENSIVE" and NVIDIA is going "yeah, now you see our problem". Then these companies shit out press releases subtly threatening NVIDIA, who's sitting back going "why are we getting blamed when Apple is the one taking all the capacity"?

I'm more and more surprised every day that NVIDIA is still making consumer GPUs. Switch all your production to server GPUs, let the consumer market sit it out a few generations, it's not like we won't be waiting a couple of generations down the line. Especially given how AMD is barely competitive and Intel isn't in any way shape or form.

They want their own AI chip because they SEE that Elon's Dojo is providing the same performance as a 10K gpu cluster at 1/6th the cost and 1/4 the footprint.

You mean the Dojo that also uses 10,000 NVIDIA GPUs?

 

[thesmokingman]

You mean the Dojo that also uses 10,000 NVIDIA GPUs?

lol, no that's not Dojo.

“We’ve been scaling our neural network training compute dramatically over the last few years,” said Milan Kovac, Tesla’s director of autopilot engineering. “Today, we’re barely shy of ten thousand GPUs. … But that’s not enough.”


Enter Dojo, the design for which was revealed during Tesla’s AI Day event, along with the design for its constituent D1 chip.

Enter Dojo: Tesla Reveals Design for Modular Supercomputer & D1 Chip

Two months ago, Tesla revealed a massive GPU cluster that it said was “roughly the number five supercomputer in the world,” and which was just a precursor to Tesla’s real supercomputing moonshot: the long-rumored, little-detailed Dojo system.

www.hpcwire.com

Hot Chips 34 – Tesla’s Dojo Microarchitecture

To say Tesla is merely interested in machine learning is an understatement.

chipsandcheese.com