• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

Two-ExaFLOP El Capitan Supercomputer Starts Installation Process with AMD Instinct MI300A

AleksandarK

News Editor
Staff member
Joined
Aug 19, 2017
Messages
2,669 (0.99/day)
When Lawrence Livermore National Laboratory (LLNL) announced the creation of a two-ExaFLOP supercomputer named El Capitan, we heard that AMD would power it with its Instinct MI300 accelerator. Today, LNLL published a Tweet that states, "We've begun receiving & installing components for El Capitan, @NNSANews' first #exascale #supercomputer. While we're still a ways from deploying it for national security purposes in 2024, it's exciting to see years of work becoming reality." As published images show, HPE racks filled with AMD Instinct MI300 are showing up now at LNLL's facility, and the supercomputer is expected to go operational in 2024. This could mean that November 2023 TOP500 list update wouldn't feature El Capitan, as system enablement would be very hard to achieve in four months until then.

The El Capitan supercomputer is expected to run on AMD Instinct MI300A accelerator, which features 24 Zen4 cores, CDNA3 architecture, and 128 GB of HBM3 memory. All paired together in a four-accelerator configuration goes inside each node from HPE, also getting water cooling treatment. While we don't have many further details on the memory and storage of El Capitan, we know that the system will exceed two ExFLOPS at peak and will consume close to 40 MW of power.



View at TechPowerUp Main Site | Source
 
Joined
Sep 6, 2013
Messages
3,427 (0.83/day)
Location
Athens, Greece
System Name 3 desktop systems: Gaming / Internet / HTPC
Processor Ryzen 5 7600 / Ryzen 5 4600G / Ryzen 5 5500
Motherboard X670E Gaming Plus WiFi / MSI X470 Gaming Plus Max (1) / MSI X470 Gaming Plus Max (2)
Cooling Aigo ICE 400SE / Segotep T4 / Νoctua U12S
Memory Kingston FURY Beast 32GB DDR5 6000 / 16GB JUHOR / 32GB G.Skill RIPJAWS 3600 + Aegis 3200
Video Card(s) ASRock RX 6600 / Vega 7 integrated / Radeon RX 580
Storage NVMes, ONLY NVMes / NVMes, SATA Storage / NVMe, SATA, external storage
Display(s) Philips 43PUS8857/12 UHD TV (120Hz, HDR, FreeSync Premium) / 19'' HP monitor + BlitzWolf BW-V5
Case Sharkoon Rebel 12 / CoolerMaster Elite 361 / Xigmatek Midguard
Audio Device(s) onboard
Power Supply Chieftec 850W / Silver Power 400W / Sharkoon 650W
Mouse CoolerMaster Devastator III Plus / CoolerMaster Devastator / Logitech
Keyboard CoolerMaster Devastator III Plus / CoolerMaster Devastator / Logitech
Software Windows 10 / Windows 10&Windows 11 / Windows 10
And I guess they might replace some 300As with 300Xs if the GPU compute is more important based on how much AI has skyrocketed lately.
 
Joined
Dec 12, 2016
Messages
1,960 (0.67/day)
And I guess they might replace some 300As with 300Xs if the GPU compute is more important based on how much AI has skyrocketed lately.
I wonder how that would work since the 300Xs have no CPUs in them. Would the motherboard need to be replaced with one that also has Epyc sockets?
 
Joined
Oct 27, 2009
Messages
1,195 (0.22/day)
Location
Republic of Texas
System Name [H]arbringer
Processor 4x 61XX ES @3.5Ghz (48cores)
Motherboard SM GL
Cooling 3x xspc rx360, rx240, 4x DT G34 snipers, D5 pump.
Memory 16x gskill DDR3 1600 cas6 2gb
Video Card(s) blah bigadv folder no gfx needed
Storage 32GB Sammy SSD
Display(s) headless
Case Xigmatek Elysium (whats left of it)
Audio Device(s) yawn
Power Supply Antec 1200w HCP
Software Ubuntu 10.10
Benchmark Scores http://valid.canardpc.com/show_oc.php?id=1780855 http://www.hwbot.org/submission/2158678 http://ww
WoW! This employees are in very dirty clothes and shoes?
What? How bad are your...
I didn't recognise formula on the image. My science is very bad. Eyes too.
Oh, yeah that makes more sense.
Clean clothes, worn knees from... gasp, kneeling to get to the bottom servers.
What an odd take...

This is is aiming for the same >2 exaflops Aurora is aiming at but at 40MW instead of 70MW.
Curious to see how far off both systems will be, the slingshot networking doesn't seem to scale as well as expected (frontier hit a bit lower than expected), but its also ground breaking and factors of scale not previously encountered are sure to be popping up.

They won't and can't just pop in mi300x as per reasons stated, these are purpose built for El Capitan and the cpu "on die" is supposed to help with scaling. the 128gb vs 192gb doesn't matter when you scale to this node count... keeping scaling as linear as possible does.

The mi250x is showing 70-80% a100 performance in ai, and absolutely obliterates it in FP64/ traditional HPC work, the claimed 8x ai improvements the mi300a is bringing should make it very competitive against the H100.
AMD's datacenter show was clearly too technical for investors to grasp, the 55B parameter model on 1 gpu was absolutely insane.
 
Last edited:
Joined
Jan 3, 2021
Messages
3,635 (2.50/day)
Location
Slovenia
Processor i5-6600K
Motherboard Asus Z170A
Cooling some cheap Cooler Master Hyper 103 or similar
Memory 16GB DDR4-2400
Video Card(s) IGP
Storage Samsung 850 EVO 250GB
Display(s) 2x Oldell 24" 1920x1200
Case Bitfenix Nova white windowless non-mesh
Audio Device(s) E-mu 1212m PCI
Power Supply Seasonic G-360
Mouse Logitech Marble trackball, never had a mouse
Keyboard Key Tronic KT2000, no Win key because 1994
Software Oldwin
In a liquid-cooled installation like this, does air cooling (by means of fans and fins) exist at all? Probably everything has to be cooled by liquid, including the chips in network switches, the SSDs in storage nodes and the power supplies.
 
Joined
Dec 12, 2016
Messages
1,960 (0.67/day)
As much as Nvidia is dominating, you don’t hear too much about exascale deployments of Nvidia accelerators. The all Intel Aurora and the all AMD frontier and El Capitan are the only ones so far.
 
Joined
Apr 7, 2023
Messages
60 (0.09/day)
AMD Instinct MI300A, which has 24 Zen4 cores, CDNA3 architecture and 128 GB of HBM3 memory. The captain will continue to command this sector with this gigantic change.
We all know that this Instinct MI300A is superior to Nvidia.
We will have to see what functions the great CAPTAIN will do
 
Joined
Jan 3, 2021
Messages
3,635 (2.50/day)
Location
Slovenia
Processor i5-6600K
Motherboard Asus Z170A
Cooling some cheap Cooler Master Hyper 103 or similar
Memory 16GB DDR4-2400
Video Card(s) IGP
Storage Samsung 850 EVO 250GB
Display(s) 2x Oldell 24" 1920x1200
Case Bitfenix Nova white windowless non-mesh
Audio Device(s) E-mu 1212m PCI
Power Supply Seasonic G-360
Mouse Logitech Marble trackball, never had a mouse
Keyboard Key Tronic KT2000, no Win key because 1994
Software Oldwin
The El Capitan supercomputer is expected to run on AMD Instinct MI300A accelerator
The illustration actually shows an MI300X, the one with four GPU chiplets and no CPU part. The MI300A is this:
1688640175135.png

What's also interesting is that the frame looks like a ... socket! Strange but apparently AMD is planning to also release socketed variants of the chip, or else they wouldn't have made this illustration.
 
Joined
Dec 12, 2016
Messages
1,960 (0.67/day)
The illustration actually shows an MI300X, the one with four GPU chiplets and no CPU part. The MI300A is this:
View attachment 303724
What's also interesting is that the frame looks like a ... socket! Strange but apparently AMD is planning to also release socketed variants of the chip, or else they wouldn't have made this illustration.
I could be completely wrong but I believe the metal parts are part of the top stiffner of an OCP Accelerator Module (OAM).


See page 10.

Edit: Nevermind, it looks more like an SP6 socket.
 
Last edited:
Joined
Apr 6, 2020
Messages
70 (0.04/day)
System Name Carnival of Glass
Processor Intel i9 14900K (previously 12900K/9900K, 8086K/Xeon X5670)
Motherboard ASRock Z790 PG SONIC (Gigabyte Z690 Aorus Master, Gigabyte Z370 Aorus Gaming 7/390 Des/X58A-UD7)
Cooling Corsair Hydro open loop, 480mm XR7, 360mm XR5!
Memory 32GB Corsair Dominator 6000MT DDR5 @6466 CL36-38-38-72-114-2
Video Card(s) Zotac RTX 3090 w/Corsair XG7 block (previously 1080Ti/970) +200 core +800 RAM +shunt mod
Storage 1x 2TB Samsung Evo 980 boot, 2TB Sabrent RQ, 2x2TB Crucial MX, 2x4TB WD SN850X, 16TB NAS!
Display(s) Acer Nitro 27" 4K, Koorui 27" 2K144Hz Acer 24" 1080p LED, 65" and 55" 4K TVs
Case Corsair 7000X (previously Corsair X570 Crystal SE)
Audio Device(s) Onboard + EVGA Nu Audio Pro 7.1, Yamaha Y2K AV Amp, Rotel RX-970B + 4x Kef Coda IIIs :D
Power Supply Corsair HX1500i Modular PSU
Mouse Corsair Darkstar/M65/Logitech G502 Lightspeed (previously G600 MMO)
Keyboard Corsair K70 Pro Optomech Axon, Logitech G910 Orion Spectrum (previously G19)
VR HMD HTC Vive Cosmos x2
Software Windows 11 x64 Enterprise (legal!)
Benchmark Scores https://www.3dmark.com/spy/18709841 https://valid.x86.fr/s9zmw1 https://valid.x86.fr/t0vrwy
Its AMD, couldn't care less. Hopefully they don't go nuclear and melt their own copper interconnects, silicon and crack the die! ;)
 
Joined
Jun 2, 2017
Messages
9,411 (3.40/day)
System Name Best AMD Computer
Processor AMD 7900X3D
Motherboard Asus X670E E Strix
Cooling In Win SR36
Memory GSKILL DDR5 32GB 5200 30
Video Card(s) Sapphire Pulse 7900XT (Watercooled)
Storage Corsair MP 700, Seagate 530 2Tb, Adata SX8200 2TBx2, Kingston 2 TBx2, Micron 8 TB, WD AN 1500
Display(s) GIGABYTE FV43U
Case Corsair 7000D Airflow
Audio Device(s) Corsair Void Pro, Logitch Z523 5.1
Power Supply Deepcool 1000M
Mouse Logitech g7 gaming mouse
Keyboard Logitech G510
Software Windows 11 Pro 64 Steam. GOG, Uplay, Origin
Benchmark Scores Firestrike: 46183 Time Spy: 25121
Its AMD, couldn't care less. Hopefully they don't go nuclear and melt their own copper interconnects, silicon and crack the die! ;)
Here comes Hot Wheels vs Matchbox! What an obtuse statement.
 

AleksandarK

News Editor
Staff member
Joined
Aug 19, 2017
Messages
2,669 (0.99/day)
Joined
Jan 3, 2021
Messages
3,635 (2.50/day)
Location
Slovenia
Processor i5-6600K
Motherboard Asus Z170A
Cooling some cheap Cooler Master Hyper 103 or similar
Memory 16GB DDR4-2400
Video Card(s) IGP
Storage Samsung 850 EVO 250GB
Display(s) 2x Oldell 24" 1920x1200
Case Bitfenix Nova white windowless non-mesh
Audio Device(s) E-mu 1212m PCI
Power Supply Seasonic G-360
Mouse Logitech Marble trackball, never had a mouse
Keyboard Key Tronic KT2000, no Win key because 1994
Software Oldwin
Joined
Jun 2, 2017
Messages
9,411 (3.40/day)
System Name Best AMD Computer
Processor AMD 7900X3D
Motherboard Asus X670E E Strix
Cooling In Win SR36
Memory GSKILL DDR5 32GB 5200 30
Video Card(s) Sapphire Pulse 7900XT (Watercooled)
Storage Corsair MP 700, Seagate 530 2Tb, Adata SX8200 2TBx2, Kingston 2 TBx2, Micron 8 TB, WD AN 1500
Display(s) GIGABYTE FV43U
Case Corsair 7000D Airflow
Audio Device(s) Corsair Void Pro, Logitch Z523 5.1
Power Supply Deepcool 1000M
Mouse Logitech g7 gaming mouse
Keyboard Logitech G510
Software Windows 11 Pro 64 Steam. GOG, Uplay, Origin
Benchmark Scores Firestrike: 46183 Time Spy: 25121
What are national security workloads that need a super computer?
There are so many uses. As an example to discover threats to fuel delivery systems.
 
Joined
Aug 2, 2012
Messages
2,026 (0.45/day)
Location
Netherlands
System Name TheDeeGee's PC
Processor Intel Core i7-11700
Motherboard ASRock Z590 Steel Legend
Cooling Noctua NH-D15S
Memory Crucial Ballistix 3200/C16 32GB
Video Card(s) Nvidia RTX 4070 Ti 12GB
Storage Crucial P5 Plus 2TB / Crucial P3 Plus 2TB / Crucial P3 Plus 4TB
Display(s) EIZO CX240
Case Lian-Li O11 Dynamic Evo XL / Noctua NF-A12x25 fans
Audio Device(s) Creative Sound Blaster ZXR / AKG K601 Headphones
Power Supply Seasonic PRIME Fanless TX-700
Mouse Logitech G500S
Keyboard Keychron Q6
Software Windows 10 Pro 64-Bit
Benchmark Scores None, as long as my games runs smooth.

AleksandarK

News Editor
Staff member
Joined
Aug 19, 2017
Messages
2,669 (0.99/day)
Joined
Dec 10, 2022
Messages
486 (0.65/day)
System Name The Phantom in the Black Tower
Processor AMD Ryzen 7 5800X3D
Motherboard ASRock X570 Pro4 AM4
Cooling AMD Wraith Prism, 5 x Cooler Master Sickleflow 120mm
Memory 64GB Team Vulcan DDR4-3600 CL18 (4×16GB)
Video Card(s) ASRock Radeon RX 7900 XTX Phantom Gaming OC 24GB
Storage WDS500G3X0E (OS), WDS100T2B0C, TM8FP6002T0C101 (x2) and ~40TB of total HDD space
Display(s) Haier 55E5500U 55" 2160p60Hz
Case Ultra U12-40670 Super Tower
Audio Device(s) Logitech Z200
Power Supply EVGA 1000 G2 Supernova 1kW 80+Gold-Certified
Mouse Logitech MK320
Keyboard Logitech MK320
VR HMD None
Software Windows 10 Professional
Benchmark Scores Fire Strike Ultra: 19484 Time Spy Extreme: 11006 Port Royal: 16545 SuperPosition 4K Optimised: 23439
Well, that's going to be quite the beast. I only hope that they're using a clean energy source for it like hydro or nuclear.
 
Joined
Aug 25, 2021
Messages
1,183 (0.97/day)
What are national security workloads that need a super computer?
Wikipedia: "Its principal responsibility is ensuring the safety, security and reliability of the nation's nuclear weapons through the application of advanced science, engineering, and technology. The laboratory also applies its special expertise and multidisciplinary capabilities towards preventing the proliferation and use of weapons of mass destruction, bolstering homeland security, and solving other nationally important problems, including energy and environmental needs, scientific research and outreach, and economic competitiveness. "

So, lots of simulations for nuclear and other weapons, their impact and development, but also environmental disasters, etc. Such simulations and calculations need a lot of horse power, both CPU and GPU... MI300 is a perfect tool for this job.

And I guess they might replace some 300As with 300Xs if the GPU compute is more important based on how much AI has skyrocketed lately.
That depends. We do not know exactly the structure of the system. It might be APUs only, as they do not do LLMs but complex simulations with hundreds of variables, so they need both CPU and GPU power.

In a liquid-cooled installation like this, does air cooling (by means of fans and fins) exist at all? Probably everything has to be cooled by liquid, including the chips in network switches, the SSDs in storage nodes and the power supplies.
No air-cooling. Too loud, too dusty.

As much as Nvidia is dominating, you don’t hear too much about exascale deployments of Nvidia accelerators. The all Intel Aurora and the all AMD frontier and El Capitan are the only ones so far.
Nvidia has a few too.
 
Joined
Apr 24, 2020
Messages
2,738 (1.60/day)
What are national security workloads that need a super computer?

Designing a COVID19 vaccine in one week.

Weather modeling.

Nuclear research (shhhhhhhh, that one's on the hush-hush except everyone knows that Department of Energy is the USA's nuke experts. And given that a lot of these supercomputers are top-secret, we can only assume what's going on...)

Like, what do a bunch of nuclear scientists want with a top-secret supercomputer that they aren't allowed to tell us the details of? Hmmm, I wonder.... fortunately, these strategic supercomputers have plenty of downtime from their main mission so that the rest of the scientific community can run on them on their spare cycles. I've heard of obscure mathematical theories being tested on these supercomputers, Ph.D thesis being written on data discovered in these, etc. etc. So its still to the benefit of the general USA's scientific community (at least when its not doing whatever nuclear research is going on...)
 
Joined
Aug 25, 2021
Messages
1,183 (0.97/day)
Its AMD, couldn't care less. Hopefully they don't go nuclear and melt their own copper interconnects, silicon and crack the die! ;)
Humour is always welcomed, stupid comments are spam.

Well, that's going to be quite the beast. I only hope that they're using a clean energy source for it like hydro or nuclear.
Nuclear is clean as soon as you are not one of countries that needs to store nuclear waste for centuries...
 
Joined
Oct 27, 2009
Messages
1,195 (0.22/day)
Location
Republic of Texas
System Name [H]arbringer
Processor 4x 61XX ES @3.5Ghz (48cores)
Motherboard SM GL
Cooling 3x xspc rx360, rx240, 4x DT G34 snipers, D5 pump.
Memory 16x gskill DDR3 1600 cas6 2gb
Video Card(s) blah bigadv folder no gfx needed
Storage 32GB Sammy SSD
Display(s) headless
Case Xigmatek Elysium (whats left of it)
Audio Device(s) yawn
Power Supply Antec 1200w HCP
Software Ubuntu 10.10
Benchmark Scores http://valid.canardpc.com/show_oc.php?id=1780855 http://www.hwbot.org/submission/2158678 http://ww
Nvidia has a few too.

Nvidia likes comparing apples to oranges, DLSS3 with native.
In the same manner it is impossible for 256 or 1024 Grace superchips to be anywhere near an Exaflop. as they are 67Tflops a pop FP32 which is how supercomputers are measured.
They could at most hit... 67 Petaflops with that announced and undeployed Euro cluster.

If we apply Nvidia metrics to AMD's MI300A El Capitan it should measure >64 exaflops but Nvidia isn't listing their metric, if that is fp16, bfloat16, fp8 or int8 or int4 even since they say exascale rather than exaflop...

My numbers are based on AMD's mi300a 228cu vs mi250 220cu scaling fp32 performance and 8x ai improvement claim, but like Nvidia's claim, we don't know what precision that is in.
 
Joined
Mar 6, 2017
Messages
3,358 (1.18/day)
Location
North East Ohio, USA
System Name My Ryzen 7 7700X Super Computer
Processor AMD Ryzen 7 7700X
Motherboard Gigabyte B650 Aorus Elite AX
Cooling DeepCool AK620 with Arctic Silver 5
Memory 2x16GB G.Skill Trident Z5 NEO DDR5 EXPO (CL30)
Video Card(s) XFX AMD Radeon RX 7900 GRE
Storage Samsung 980 EVO 1 TB NVMe SSD (System Drive), Samsung 970 EVO 500 GB NVMe SSD (Game Drive)
Display(s) Acer Nitro XV272U (DisplayPort) and Acer Nitro XV270U (DisplayPort)
Case Lian Li LANCOOL II MESH C
Audio Device(s) On-Board Sound / Sony WH-XB910N Bluetooth Headphones
Power Supply MSI A850GF
Mouse Logitech M705
Keyboard Steelseries
Software Windows 11 Pro 64-bit
Benchmark Scores https://valid.x86.fr/liwjs3
Nuclear is clean as soon as you are not one of countries that needs to store nuclear waste for centuries...
There's ways of reusing the spent fuel but nobody wants to actually do it.
 
Top