Monday, March 25th 2024
Tiny Corp. Prepping Separate AMD & NVIDIA GPU-based AI Compute Systems
George Hotz and his startup operation (Tiny Corporation) appeared ready to completely abandon AMD Radeon GPUs last week, after experiencing a period of firmware-related headaches. The original plan involved the development of a pre-orderable $15,000 TinyBox AI compute cluster that housed six XFX Speedster MERC310 RX 7900 XTX graphics cards, but software/driver issues prompted experimentation via alternative hardware routes. A lot of media coverage has focused on the unusual adoption of consumer-grade GPUs—Tiny Corp.'s struggles with RDNA 3 (rather than CDNA 3) were maneuvered further into public view, after top AMD brass pitched in.
The startup's social media feed is very transparent about showcasing everyday tasks, problem-solving and important decision-making. Several Acer Predator BiFrost Arc A770 OC cards were purchased and promptly integrated into a colorfully-lit TinyBox prototype, but Hotz & Co. swiftly moved onto Team Green pastures. Tiny Corp. has begrudgingly adopted NVIDIA GeForce RTX 4090 GPUs. Earlier today, it was announced that work on the AMD-based system has resumed—although customers were forewarned about anticipated teething problems. The surprising message arrived in the early hours: "a hard to find 'umr' repo has turned around the feasibility of the AMD TinyBox. It will be a journey, but it gives us an ability to debug. We're going to sell both, red for $15,000 and green for $25,000. When you realize your pre-order you'll choose your color. Website has been updated. If you like to tinker and feel pain, buy red. The driver still crashes the GPU and hangs sometimes, but we can work together to improve it."
Tiny Corp. recommends that potential clients spend a little extra on a stable NVIDIA Ada Lovelace-based system, but work will continue on ironing out nitty-gritty details with AMD engineers—today's tweet outlined a short roadmap: "going to start documenting the Radeon RX 7900 XTX GPU, and we're going right to the kernel in tinygrad with a KFD backend. Also, expect an announcement from AMD, it's not everything we asked for, but it's a start. If you want 'it just works' buy green. You pay the tax, but it's rock solid. Not too much more to say about it. Compare to more expensive 6x GeForce RTX 4090 boxes elsewhere. Hopefully we get both colors of TinyBox on MLPerf in June." It is not clear, at the time of writing, whether the Intel-based TinyBox will be prepped for pre-orders.
Sources:
tinygrad Tweet #1, Tom's Hardware, Wccftech, tinygrad Tweet #2
The startup's social media feed is very transparent about showcasing everyday tasks, problem-solving and important decision-making. Several Acer Predator BiFrost Arc A770 OC cards were purchased and promptly integrated into a colorfully-lit TinyBox prototype, but Hotz & Co. swiftly moved onto Team Green pastures. Tiny Corp. has begrudgingly adopted NVIDIA GeForce RTX 4090 GPUs. Earlier today, it was announced that work on the AMD-based system has resumed—although customers were forewarned about anticipated teething problems. The surprising message arrived in the early hours: "a hard to find 'umr' repo has turned around the feasibility of the AMD TinyBox. It will be a journey, but it gives us an ability to debug. We're going to sell both, red for $15,000 and green for $25,000. When you realize your pre-order you'll choose your color. Website has been updated. If you like to tinker and feel pain, buy red. The driver still crashes the GPU and hangs sometimes, but we can work together to improve it."
17 Comments on Tiny Corp. Prepping Separate AMD & NVIDIA GPU-based AI Compute Systems
I'm sure Jensen looked over at his senior team, chuckled and said "I told you!"
:):p:D
Systems like the TinyBox aren't really anything special hardware-wise, since most likely all their parts can be bought online. The interesting part is selling it as a working appliance, and providing support for the clients.
Issues were mostly coming up beyond 4 cards. Why not sell a dual-system chassis with 2x4 cards?
From what I can tell, bigger is better for many of these nascent AI models. It's not just the number of AI processing cores, it also extends to VRAM, memory bandwidth, etc.
A distributed workload amongst several systems is likely less efficient than one larger system in many cases. There are probably some significant overhead demands from distributing AI computing. I thought I heard Nvidia (probably Jensen) say that in many AI systems, the AI cores only spend 60% of their time doing compute, the rest of it is talking to other cores. They are trying to develop new interconnects and infrastructural improvements to reduce that so AI cores do more what they're supposed to be doing (compute). These types of challenges aren't captured in synthetic benchmarks.
From a vendor standpoint, Tiny might be thinking that supporting ten customers with half-sized systems will cost them more than supporting five customers with full-sized systems. Customer support costs do not scale linearly.
planning to buy a new vehicle instead
Also, $25k is still probably considerably more affordable than the 'big boy' solutions on the market.
Example: When I was researching *where* my MI25s came from, the first servers w/ 4+ of them were $50K+.
(Kinda made me grin ear-to-ear knowing I paid ~$50 ea for the cards, that came out of 1000x more expensive systems :laugh:)
All the other companies are raising money from investors. But that's not "making money", that's just them calling investors and investors throwing money at them without understanding what the money is even doing.
Effectively scamming investors has been a time-proven money maker, sadly...
Nvidia is making money from AI. So are AMD and Intel. It's just that Nvidia is making far more money than the other two at this point in time, taking a larger percentage of the pie. But Nvidia is far from the only one making money from people jumping onto the AI bandwagon. There are other AI players. For sure, AMD is accepting purchase orders from customers who don't want to get in the back of Nvidia's line and wait 12-18 months for Team Green's latest and greatest AI accelerator.
There's also a synergistic effect the semiconductor boom from the AI boom. Semiconductor manufacturing companies like Applied Materials, ASML, Lam Research, KLA, etc. are all benefiting indirectly. Same with memory manufacturers like Micron, SK Hynix, etc. If you crack open any graphics card or AI accelerator, there are hundreds of ICs but only one or two GPUs. All those other components are being made by someone. All those companies benefit whether it's Texas Instruments or some no-name company making filters, capacitors, whatever.
For sure TSMC isn't a charity just pumping out GPUs for giggles. And an AI accelerator card is just a fancy paperweight unless you stick it in a rack, in a server room, with miles of cabling, multiple HVAC units on the roof, et cetera ad nauseam. Even the power company benefits from the time you press the power button.
As a GPU designer, Nvidia already has an established customer base of AI customers, some of whom are probably thinking about their third and fourth generation purchases. And what is very frequently ignored here by many TPU commenters is Nvidia's formidable developer ecosystem, something that is not measured in a synthetic AI benchmark run.
As for this Tiny company, it's too early to tell whether it's just smoke and mirrors or something substantial.
I know a lot of people in these discussion groups never acknowledge history. Nvidia was once a struggling startup. Same as AMD. Same as Intel. Same as Fairchild Semiconductor. Same as their customers: Amazon, Alphabet/Google, Meta/Facebook, Tesla, FedEx, Walmart, Ford Motors, etc.
Sure there are some people who just want to bilk their investors. They typically don't stay in business very long. They also end up not having many friends and don't get invited for repeat performances. There are certainly people who make poor wagers and end up creating things that aren't interesting to the marketplace. Ask Microsoft about their mobile phone lineup. Or Alphabet about their Google Stadia cloud gaming service.
Whether you do or do not have a market capitalization of a trillion dollars, you can still make good and bad decisions. It's up to Tiny to increase shareholder value even if when they still a private company.
And it's not just AI. If you buy a Ford Mustang with leather seats, it's not just Ford Motor that benefits. Some cattle rancher who raises steer for hide got paid too.
Remember that Nvidia was once a startup. In fact, this company's business plan is right in Nvidia's domain, selling AI solutions to corporations. I would be very, very surprised if Tiny Corporation didn't have former Nvidia engineers on its payroll maybe even ones that Jensen knows by name.
More than that CEO George Hotz previously ran an AI-powered autonomous driving car company which likely would have used Nvidia technology. On top of that this guy has a pretty bombastic personality. He's not very subtle about anything he does, not in the past, not now, and probably not in the future. This guy likes to be the controversial headline.
And besides, Silicon Valley isn't that big, people generally know about stuff, even if it's not the specifics. This is hard to understand if you don't live near Silicon Valley. It's simply not that big of a place. And people hop around all the time. Word travels fast.
And there are plenty of people here in Silicon Valley like the energy of a startup. You cleverly ignored the point that Nvidia itself was a startup. The same sort of people that started up Nvidia would leave a big Fortune 100 firm to go work for some scrappy little startup like Tiny Corporation.
Jensen likely knows about this guy. Like I said, Silicon Valley is SMALL. There's a good chance Jensen will not say anything even if he is aware of this guy. It's not like Nvidia needs any more purchase orders for 4090s. They would rather use the wafers to make AI accelerator chips that they can charge 5x to 10x more for.
But I strongly disagree with the assumption that Jensen doesn't know of this guy. Holz's loud posturing has been covered multiple times by multiple media organizations that specialize in tech news. It's not like Holz is complaining to People magazine or TMZ.
And at some point (which has probably already happened), someone with REAL money (like a venture capitalist) is going to bring it up.
VC: "Hey Jensen, there's this dude George Holz who thinks he can get rich quick stuffing gaming cards into an AI dev box and selling them for $25K. Should I give him any money?"
Jensen: "That guy?" (laughs) "You're buying the next round."