Saturday, December 15th 2018

AMD 7nm EPYC "Rome" CPUs in Upcoming Finnish Supercomputer, 200,000 Cores Total

During the next year and a half, the Finnish IT Center for Science (CSC) will be purchasing a new supercomputer in two phases. The first phase consists of Atos' air-cooled BullSequana X400 cluster which makes use of Intel's Cascade Lake Xeon processors along with Mellanox HDR InfiniBand for a theoretical performance of 2 petaflops. Meanwhile, system memory per node will range from 96 GB up to 1.5 TB with the entire system receiving a 4.9 PB Lustre parallel file system as well from DDN. Furthermore, a separate partition of phase one will be used for AI research and will feature 320 NVIDIA V100 NVLinked GPUs configured in 4-GPU nodes. It is expected that peak performance will reach 2.5 petaflops. Phase one will be brought online at some point in the summer of 2019.

Where things get interesting is in phase two, which is set for completion during the spring of 2020. Atos' will be building CSC a liquid-cooled HDR-connected BullSequana XH2000 supercomputer that will be configured with 200,000 AMD EPYC "Rome" CPU cores which for the mathematicians out there works out to 3,125 64 core AMD EPYC processors. Of course, all that x86 muscle will require a great deal of system memory, as such, each node will be equipped with 256 GB for good measure. Storage will consist of an 8 PB Lustre parallel file system that is to be provided by DDN. Overall phase two will increase computing capacity by 6.4 petaflops (peak). With deals like this already being signed it would appear AMD's next-generation EPYC processors are shaping up nicely considering Intel had this market cornered for nearly a decade.
When both phases are complete, the entire system will be capable of 11 petaflops of theoretical performance which is an increase of over five times what currently Finnish scientists had available. The system will be used by numerous agencies and universities in multiple studies such as astrophysics, drug development, nanoscience, and AI research. All that said, performance like this doesn't come cheap either with Finland investing €37 million ($41.8 million) in their endeavor to upgrade and update their high-performance computing infrastructure.
Source: HPCwire
Add your own comment

47 Comments on AMD 7nm EPYC "Rome" CPUs in Upcoming Finnish Supercomputer, 200,000 Cores Total

#26
R-T-B
Vayra86Both, but it really is essential these machines run within a certain power envelope or the cost of ownership will skyrocket. There is a balance to be struck, if you project a supercomputer to run X years you might as well just add more racks instead of dealing with all the disadvantages of high clocks/perf per 1U. Extra surface area is extremely cheap anyway considering where these centers are built.

Another aspect is probably also the fact that bottlenecking occurs, storage and RAM are much more important in this space. No point oversaturating anything. And on top of all that, there are limitations to what can be fitted under the IHS, before you straight burn a hole in your server + yield issues. High clocking many cores are progressively harder to make, its the whole reason Epyc and TR are so amazing, they cut that yield risk / the number of dies.
Makes sense. I'm a desktop guy, so I'll fully admit I've never built or budgeted a super computer... :laugh:
Posted on Reply
#27
m4dn355
lynx29well i think they are dumb for not waiting and going all in with AMD 7nm... why wasiting money on power hungry cascade lake is beyond me.
Maybe they want to internally test price/perf/power equation of intel vs AMD vs nVidia
Posted on Reply
#28
renz496
HTCThey most surely are not.

These supercomputers tend to cost as much in power bills over a few years as the cost of building them in the 1st place: it's why server chips have much lower clocks VS regular desktops because it helps tremendously with the power bills.
Yes power consumption and efficiency is one important factor when building a supercomputer but that is not the reason why the CPU or GPU in such system have much lower clock than regular consumer part. They have much lower clock to ensure stability on 24/7 full load operation. Our PC at home did not under go such stress the whole time. That's why to ensure stability in overclocking we often run stuff like Prime95 for a few hours or even the entire day.
lynx29well i think they are dumb for not waiting and going all in with AMD 7nm... why wasiting money on power hungry cascade lake is beyond me.
Professional client are not like regular consumer that can change their hardware provider anytime they like. Before they go all in with AMD they still need to evaluate AMD based system in small scale first. Just look at Hawaii S9150. When it comes to raw power and effiiency it simply kick nvidia GK210 ass. Nvidia need two GK210 just so they can beat a single Hawaii in DP. But a few years later how many supercomputer in the top ten even employ S9150?
Posted on Reply
#29
HTC
renz496Yes power consumption and efficiency is one important factor when building a supercomputer but that is not the reason why the CPU or GPU in such system have much lower clock than regular consumer part. They have much lower clock to ensure stability on 24/7 full load operation. Our PC at home did not under go such stress the whole time. That's why to ensure stability in overclocking we often run stuff like Prime95 for a few hours or even the entire day.
I should have said "it's one of the reasons" instead of "it's why".

Question for you: GPUs in supercomputers also run @ lower speeds than regular consumer parts? I know about it in CPUs but not GPUs.
Posted on Reply
#30
renz496
Yes
HTCI should have said "it's one of the reasons" instead of "it's why".

Question for you: GPUs in supercomputers also run @ lower speeds than regular consumer parts? I know about it in CPUs but not GPUs.
Yes. Not just supercomputer any professional grade hardware like quadro have much lower clock than geforce.
Posted on Reply
#31
Space Lynx
Astronaut
renz496Yes power consumption and efficiency is one important factor when building a supercomputer but that is not the reason why the CPU or GPU in such system have much lower clock than regular consumer part. They have much lower clock to ensure stability on 24/7 full load operation. Our PC at home did not under go such stress the whole time. That's why to ensure stability in overclocking we often run stuff like Prime95 for a few hours or even the entire day.


Professional client are not like regular consumer that can change their hardware provider anytime they like. Before they go all in with AMD they still need to evaluate AMD based system in small scale first. Just look at Hawaii S9150. When it comes to raw power and effiiency it simply kick nvidia GK210 ass. Nvidia need two GK210 just so they can beat a single Hawaii in DP. But a few years later how many supercomputer in the top ten even employ S9150?
Finland has 5 million people. This two phase plan is all in for that country... your point is moot.
Posted on Reply
#32
renz496
So? It doesn't mean they just can blindly commit themselves to certain hardware for their need without proper evaluation.
Posted on Reply
#33
Space Lynx
Astronaut
renz496So? It doesn't mean they just can blindly commit themselves to certain hardware for their need without proper evaluation.
I mean the software they will be using they should know already if it runs better on AMD or Intel... they are not the only ones building a "supercomputer". Surely, they could have called around and been like hey does this modeling work well for you on EPYC or XEON better? ..... ugh nevermind, I don't feel like typing anymore
Posted on Reply
#34
RichF
renz496Before they go all in with AMD they still need to evaluate AMD based system in small scale first. Just look at Hawaii S9150. When it comes to raw power and effiiency it simply kick nvidia GK210 ass. Nvidia need two GK210 just so they can beat a single Hawaii in DP. But a few years later how many supercomputer in the top ten even employ S9150?
I was surprised to see multiple Piledriver systems in lists of top supercomputers in recent times. They must have gotten quite a good bulk price from AMD. Also, if the work is GPU-centric, the CPU is less important.
Posted on Reply
#35
Vya Domus
lynx29well i think they are dumb for not waiting and going all in with AMD 7nm... why wasiting money on power hungry cascade lake is beyond me.
Time, these things are conceived at least a year prior to their deployment. That being said back then they had no idea about 7nm Epyc, this was likely a last minute thing.
Posted on Reply
#36
Vayra86
R-T-BMakes sense. I'm a desktop guy, so I'll fully admit I've never built or budgeted a super computer... :laugh:
Most of what I know comes from reading articles of people who visited these sites :) Not building these either :D
Posted on Reply
#37
HTC
renz496Professional client are not like regular consumer that can change their hardware provider anytime they like. Before they go all in with AMD they still need to evaluate AMD based system in small scale first. Just look at Hawaii S9150. When it comes to raw power and effiiency it simply kick nvidia GK210 ass. Nvidia need two GK210 just so they can beat a single Hawaii in DP. But a few years later how many supercomputer in the top ten even employ S9150?
I'd argue there's another reason: certain loads favor a manufacturer over the other and, by using both, they have "the best of both worlds".

OTOH, i wouldn't be surprised if this stems from a previous contract with Intel forcing them to have @ least a certain amount of processors every generation. No: i don't have a source for this but, given Intel's track record in "the shady deal department", i wouldn't be that farfetched if current Finnish contract (from previous supercomputers) forced them to go this route.
Posted on Reply
#38
stimpy88
trog100excuse my ignorance but what the f-ck does one do with such a machine.. assuming its all being put together for some practical purpose..

trog
To play a nice game of Thermonuclear War.
Posted on Reply
#39
DeathtoGnomes
stimpy88To play a nice game of Thermonuclear War.
The only winning move is not to play.
Posted on Reply
#40
lemonadesoda
Darmok N JaladI think performance per watt is a significant consideration, as well as cooling the entire configuration. Just think of the difference that even 5W more TDP per socket would add up to in a 3125 socket system. That’s 15000+ more watts of cooling required.
Finland is so cold, they only need to open the windows.
Posted on Reply
#41
m4dn355
lemonadesodaFinland is so cold, they only need to open the windows.
Impossible since they are using Linux.
Posted on Reply
#43
FordGT90Concept
"I go fast!1!11!1!"
Peak capacity is always given in binary32 (IEEE 754) performance. Yes, their motivation for going big on CPUs and small on GPUs may be because they're intending to do more logic or binary64 computations on it rather than brute force SIMD binary32 operations which GPUs can do with relatively few transistors. Then again, the OP sounds like the machine is a cobble job--jack of all trades and master of none. LLNL has a variety of machines suitable for all kinds of tasks. This might be the only one CSC has which means it has to be flexible.
Posted on Reply
#44
John Naylor
How long before we'll be seeing posts claiming not enough cores to run today's AAA games. ?
Posted on Reply
#45
hat
Enthusiast
John NaylorHow long before we'll be seeing posts claiming not enough cores to run today's AAA games. ?
Well, we already had the now over 10 year old Crysis joke...
Posted on Reply
#47
Tartaros
hatWell, we already had the now over 10 year old Crysis joke...
I really advocate for minecraft loading time jokes. Is the second most ludicrous shit I ever seen on gaming, first being you can recreate the events of the movie Your Name on Fallout 76 (less groping your own breasts if you change from man to woman. That would be sick).
Posted on Reply
Add your own comment
Jan 5th, 2025 11:17 EST change timezone

New Forum Posts

Popular Reviews

Controversial News Posts