• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

NASA Scales SGI Pleiades InfiniBand Cluster to 25,000 Intel Xeon Processor Cores

btarunr

Editor & Senior Moderator
Staff member
Joined
Oct 9, 2007
Messages
47,294 (7.52/day)
Location
Hyderabad, India
System Name RBMK-1000
Processor AMD Ryzen 7 5700G
Motherboard ASUS ROG Strix B450-E Gaming
Cooling DeepCool Gammax L240 V2
Memory 2x 8GB G.Skill Sniper X
Video Card(s) Palit GeForce RTX 2080 SUPER GameRock
Storage Western Digital Black NVMe 512GB
Display(s) BenQ 1440p 60 Hz 27-inch
Case Corsair Carbide 100R
Audio Device(s) ASUS SupremeFX S1220A
Power Supply Cooler Master MWE Gold 650W
Mouse ASUS ROG Strix Impact
Keyboard Gamdias Hermes E2
Software Windows 11 Pro
SGI, the trusted leader in technical computing, announced today that with over 60 miles of InfiniBand cabling in place at the NASA Advanced Supercomputing (NAS) Division at NASA Ames Research Center at Moffett Field, Calif., a scientist was able to utilize 25,000 SGI ICE Intel Xeon processor cores on Pleiades to run a space weather simulation.

One particular area of study is magnetic reconnection, a physical process in highly conducting plasmas such as those that occur in the Earth's magnetosphere, in which the magnetic topology is rearranged and magnetic energy converted to kinetic or thermal energy. This field of research is critical, as these disturbances can disable wide scale power grids, affect satellite transmissions and disrupt airline communications.



"We study the efficiency of reconnection under different conditions in the magnetosphere," said Dr. Homa Karimabadi, space physics group leader at the University of California, San Diego. "This includes such things as the amount of mixing of the plasma from the solar wind and from the magnetosphere. The team then uses the local-scale details to improve models of magnetic reconnection in 'global' simulations encompassing a region starting at about three earth radii and extending to 30 to 200 earth radii, with the Earth's radius being about 6400 kilometers."

As detailed in the article, "Cracking the Mysteries of Space Weather," by Jarrett Cohen of NASA, Earth is mostly protected from solar flares, coronal mass ejections, and other space weather events by the magnetosphere, a magnetic field cocoon that surrounds it. But sometimes Earth's magnetosphere 'cracks' and lets space weather inside, where it can cause damage. Getting space weather details right means capturing everything from the 1.28 million kilometer-sized magnetosphere down to subatomic-scale electrons. Doing that in one simulation would require supercomputers more than 1,000 times faster than those available today, so the research team breaks the problem into two parts. They start with 'local' simulations that include full electron physics of regions in the magnetosphere where reconnection is known to occur, followed by 'global' simulations.

Accessing up to 25,000 processor cores on Pleiades, Dr. Karimabadi said that his group can run 'kinetic' simulations that treat each electron with its full properties and understand how electrons allow reconnection to occur. In the local simulations, electrons are treated as individual particles. In the global simulations, electrons are treated as fluids and ions (electrically charged atoms) as particles. With Pleiades, simulations can run for five days straight, enabling many parameter studies. Among recent findings is that magnetic reconnection by itself is quite turbulent, producing vortices in the plasma that create many interacting flux ropes-twisted bundles of magnetic field. As observed by spacecraft, flux ropes can extend several times the radius of Earth.

Space weather is just one application running in the petascale range today that will benefit from the move to Exascale computing. SGI is committed to bringing Exascale solutions to the marketplace in an open computing paradigm in which InfiniBand will have an integral role by providing the key interconnect elements. Having demonstrated that InfiniBand can enable petaflop-sized systems at NASA, SGI will also be partnering with Total to bring InfiniBand into the multi-petaflop commercial space. Coupled with software tools such as SGI Management Center and SGI Performance Suite, as well as big data InfiniteStorage solutions, SGI is positioned to offer an optimal user experience for multi-petaflop deployments moving to the exascale range.

"The growing complexity of problems that Pleiades is expected to solve is why we remain committed to developing and providing exascale solutions," said Praveen K. Mandal, senior vice president of engineering at SGI. "NASA has been a long-time partner of SGI in pushing the boundaries of technology development and scientific discovery. For five generations, SGI ICE has tightly integrated InfiniBand technology to achieve top performance and scaling capabilities in a completely open networking architecture, and at the same time, with five different networking topologies, ICE is flexible enough to fit all the workloads that benefit science."

Maximizing productivity in today's HPC cluster platforms requires using enhanced data messaging techniques. On Pleiades, every single node has a "direct" InfiniBand connection to the rest of the network. That is why Pleiades has the largest InfiniBand network of any HPC system in the current Top 500 list. By providing low-latency, high-bandwidth, and a large message rate, high efficiency interconnect solutions are used as high-speed interconnects for large-scale simulations such as these conducted by NASA, and are replacing proprietary or low-performance solutions.

"The scalable architectural design of the SGI ICE platform provides an excellent vehicle for showcasing the performance, scalability and efficiency of Mellanox InfiniBand server and storage interconnect solutions," said David Barzilai, vice president of marketing at Mellanox Technologies. "Utilizing InfiniBand's ability to efficiently scale to 10,000 and more server nodes, Pleiades is able to run space weather simulations of greater complexity faster and with more precision."

The Pleiades supercomputer is currently ranked the #7 most powerful HPC system in the world based upon the Top 500 list published in November 2011. More information about Pleiades is available here.

View at TechPowerUp Main Site
 

Completely Bonkers

New Member
Joined
Feb 6, 2007
Messages
2,576 (0.39/day)
Processor Mysterious Engineering Prototype
Motherboard Intel 865
Cooling Custom block made in workshop
Memory Corsair XMS 2GB
Video Card(s) FireGL X3-256
Display(s) 1600x1200 SyncMaster x 2 = 3200x1200
Software Windows 2003
How much did that optical interference pattern cost to design and build!



Quite a system!
 
Last edited:
Joined
Oct 30, 2008
Messages
1,768 (0.30/day)
System Name Lailalo
Processor Ryzen 9 5900X Boosts to 4.95Ghz
Motherboard Asus TUF Gaming X570-Plus (WIFI
Cooling Noctua
Memory 32GB DDR4 3200 Corsair Vengeance
Video Card(s) XFX 7900XT 20GB
Storage Samsung 970 Pro Plus 1TB, Crucial 1TB MX500 SSD, Segate 3TB
Display(s) LG Ultrawide 29in @ 2560x1080
Case Coolermaster Storm Sniper
Power Supply XPG 1000W
Mouse G602
Keyboard G510s
Software Windows 10 Pro / Windows 10 Home
Bet that could play Crysis ;D
 
Joined
Sep 8, 2007
Messages
304 (0.05/day)
Location
Israel
System Name ¯\_(ツ)_/¯
Processor AMD Ryzen 7 7800X3D
Motherboard MSI MEG X670E Ace
Cooling EK Nucleus CR360
Memory 64GB DDR5 Corsair Dominator Titanium 6000Mhz CL30 1.4v
Video Card(s) Asus RTX 4090 ROG Strix OC
Storage Samsung 990 Pro 2TB \ Kingston KC3000 2TB
Display(s) LG GQ950-B 4K 144hz 32"
Case Lian Li Lancool 216
Audio Device(s) Focusrite Scarlett 2i4 Gen2 \ SS Arctis Nova Pro \ DT 770 Pro
Power Supply MSI A1000G 1000W ATX 3.0
Mouse HyperX Pulsefire Haste 2
Keyboard NuPhy Filed75
Software Windows 11 Pro
Joined
Nov 17, 2011
Messages
298 (0.06/day)
System Name Game Raver
Processor Core i7-770K
Motherboard ASRock Z170A-X1
Cooling Coolermaster AIO
Memory 16gb
Video Card(s) GTX 1060
Storage 1TB Crucial SSD. 4TB Seagate HDD
Display(s) AOC 1080p 144Hz
Case NZXT Phantom 410 /w/ Silverstone FN121P-BL fans
Audio Device(s) ASUS Essence STX
Power Supply Corsair AX760
I can imagine their use now that NASA doesn't run space programs anymore.

I'll give you a hint:
 

FordGT90Concept

"I go fast!1!11!1!"
Joined
Oct 13, 2008
Messages
26,259 (4.44/day)
Location
IA, USA
System Name BY-2021
Processor AMD Ryzen 7 5800X (65w eco profile)
Motherboard MSI B550 Gaming Plus
Cooling Scythe Mugen (rev 5)
Memory 2 x Kingston HyperX DDR4-3200 32 GiB
Video Card(s) AMD Radeon RX 7900 XT
Storage Samsung 980 Pro, Seagate Exos X20 TB 7200 RPM
Display(s) Nixeus NX-EDG274K (3840x2160@144 DP) + Samsung SyncMaster 906BW (1440x900@60 HDMI-DVI)
Case Coolermaster HAF 932 w/ USB 3.0 5.25" bay + USB 3.2 (A+C) 3.5" bay
Audio Device(s) Realtek ALC1150, Micca OriGen+
Power Supply Enermax Platimax 850w
Mouse Nixeus REVEL-X
Keyboard Tesoro Excalibur
Software Windows 10 Home 64-bit
Benchmark Scores Faster than the tortoise; slower than the hare.
As a taxpayer, I demand they let me use that computer for important scientific research: the Ultimate Answer to the Ultimate Question of Life, The Universe, and Everything! I wonder if it will get the same answer as Deep Thought. :D
 
Last edited:
Joined
Nov 17, 2011
Messages
298 (0.06/day)
System Name Game Raver
Processor Core i7-770K
Motherboard ASRock Z170A-X1
Cooling Coolermaster AIO
Memory 16gb
Video Card(s) GTX 1060
Storage 1TB Crucial SSD. 4TB Seagate HDD
Display(s) AOC 1080p 144Hz
Case NZXT Phantom 410 /w/ Silverstone FN121P-BL fans
Audio Device(s) ASUS Essence STX
Power Supply Corsair AX760
As a taxpayer, I demand they let me use that computer for important scientific research: the Ultimate Answer to the Ultimate Question of Life, The Universe, and Everything! I wonder if it will get the same answer as Deep Thought. :D

You heard Deep Thought, it was 42!
 
Top