UK Prepares $1.6 Billion for the Most Powerful Weather Forecasting Supercomputer

[notb]

All the computing power cant help if the weather algorithm is bad

Theoretically it's not that hard to write them, so - I'd imagine - there aren't many bad models in use by large (both national and private) entities.
It's PDEs after all. I mean: there are many lines of code, but the idea is quite simple.

Some fairly decent models / libraries are free and open-source. COAMPS is very popular:

COAMPS

Met Office uses a model called Unified Model. AFAIK it's not open-source, but it's fairly popular as well (I worked with it at university in Poland).

Synoptic and climate stations

The map shows the current network of automatic (synoptic) and manual (climate) stations covering the UK.

www.metoffice.gov.uk

As in many other problems in physics: it's actually the input data that makes this difficult - not the actual model complexity.
1) you're mostly using input from the surface, but you're actually modeling in 3D

2) the number of stations is limited. In case of UK and Met: just ~270 stations report in real time or hourly

Synoptic and climate stations

The map shows the current network of automatic (synoptic) and manual (climate) stations covering the UK.

www.metoffice.gov.uk

3) no matter how small your country is, you have to model the whole atmosphere anyway
This means you're forced to use 3rd party data for the rest of the planet. Of course national institutes cooperate (and some data is just free), but that means you have very little impact on data quality and data point density.

i really can't imagine why this super computer is needed for forecasting; the satellites give us already the movement of air masses and the path also; sometimes is impossible to forecast in long term as a sudden change of air movement change all and no matter how powerful computer you use is not-predictable

I'm not sure where you're going with this. Numerical weather forecasting has been around for decades. It works.

We know equations that have proven to be very accurate. We come up with something better from time to time. It's not magic.

As for satellite imaging: you can't even learn the current conditions based on that, which implies that you can't make any kind of forecast.
Sure, they can be used an additional source, but there's very little data that can be extracted.

 

[phill]

It's a cluster. You need 1000 cores, you buy 1000 cores.
AMD CPUs offer better value at the moment, i.e. you need less sockets to deliver the required cluster performance.

And of course there's a good chance this system can benefit from AVX-512. It really is a big selling point for Intel.

First of all: speeds you've mentioned are for home use, not for datacenters...
Second: this supercomputer will likely be kept offline, i.e. no direct Internet access (it isn't needed).

If they need that many cores, they will have less physical kit with AMD due to the core count, so I'd hope that would help them out in that way to be honest. But either way, I'd love to find out the specs of the hardware they are looking at.. Ignoring the same old question about Crysis, but seeing how it's put together would be amazing
I thought that AMD had that instruction set in the newer models of CPUs now? I thought that was the case anyways but I think there might be other versions of that set I guess?

Uh, the cost of internet that fast in the UK is negligible. 1Gb/s internet is typically £65/month for consumers, our business fibre (Daul 4Gb/s symmetric with failover to a different trunk) is £2700/quarter or £900/month. There is healthy fibre competition in Exeter where the Met office is based.

Assume that the power and cooling costs of a supercomputer cluster are 10X higher than the internet connectivity, and you'll be in the right ballpark at least.

You're very lucky if you can get 1Gb for £65, I'm paying about half that for 30Mb!! The line we have at work is 1Gb up/down, I think that was something like £27k for so many years I think (dual lines different exchanges etc etc..

 

[Basard]

Budget build... so definitely EPYC....

10 Xeons = 15 EPYC and 100+ more cores = budget build.

I agree, Epyc would be the right choice.... but government loves to waste taxpayer dollars, so probably Xeons.

 

[ARF]

Will it be less amazing with Intel inside?
What's suddenly wrong with Xeons?

They are corrupt and less fair, and among other things - several times more expensive, slower and more power hungry.

It needs to be EPYC + AMD GPUs.

Definitely going to cost more than 1.6 billion with the way how the UK approach things, also they can just continue reporting its going to rain everyday in the UK which is what we are known for having, don't need super computer for that.

But its purpose will be climate simulation, not weather forecast.

 

[Cheeseball]

I thought that AMD had that instruction set in the newer models of CPUs now? I thought that was the case anyways but I think there might be other versions of that set I guess?

Zen 2 currently has 256-bit-wide registers (single-cycle!!!) for SIMD instructions, so if they had implemented AVX-512 it would not turn out well for them as it would push computational load dramatically just like how AVX/AVX2 does for current Intel CPUs.

The alternative to this approach for ML (which is one few use cases for AVX-512) is pairing the server CPUs with GPGPUs, preferably one with dedicated tensor cores (mostly NVIDIA at the moment) for higher efficiency.

 

[notb]

If they need that many cores, they will have less physical kit with AMD due to the core count, so I'd hope that would help them out in that way to be honest.

That's only when you assume they'll go for the highest core count models, which may not be the case.
Their current cluster uses E5-2695 v4, which is 18C. That generation offered SKUs with higher core count and higher frequencies (also both).

HPC systems are carefully selected and fine-tuned.
You can't assume everyone gets the 64-core EPYC right now so there's no reason to go with Intel.

Check the latest Top500 (54th). Out of 147 systems started in 2019, 99 don't use any accelerator - i.e. they're focused on CPU performance.
95 of them use Intel CPUs (from few generations).
Histogram of "Cores per Socket":
10: 1
12: 6
14: 6
16: 15
18: 5
20: 53
24: 5
28: 2
48: 2

Also, people somehow focused on AMD vs Intel, wheareas they may also go for something GPU-based or even ARM. Both approaches are being tested in weather modeling with good results.

 

[phill]

Zen 2 currently has 256-bit-wide registers (single-cycle!!!) for SIMD instructions, so if they had implemented AVX-512 it would not turn out well for them as it would push computational load dramatically just like how AVX/AVX2 does for current Intel CPUs.

The alternative to this approach for ML (which is one few use cases for AVX-512) is pairing the server CPUs with GPGPUs, preferably one with dedicated tensor cores (mostly NVIDIA at the moment) for higher efficiency.

Thanks for clearing that up

That's only when you assume they'll go for the highest core count models, which may not be the case.
Their current cluster uses E5-2695 v4, which is 18C. That generation offered SKUs with higher core count and higher frequencies (also both).

HPC systems are carefully selected and fine-tuned.
You can't assume everyone gets the 64-core EPYC right now so there's no reason to go with Intel.

Check the latest Top500 (54th). Out of 147 systems started in 2019, 99 don't use any accelerator - i.e. they're focused on CPU performance.
95 of them use Intel CPUs (from few generations).
Histogram of "Cores per Socket":
10: 1
12: 6
14: 6
16: 15
18: 5
20: 53
24: 5
28: 2
48: 2

Also, people somehow focused on AMD vs Intel, wheareas they may also go for something GPU-based or even ARM. Both approaches are being tested in weather modeling with good results.

For things in general going at any price point, you do get more for your money with AMD over Intel at the moment but again comes back to the if there's a better tool for the job, then they can use it. I just mentioned I hope they go with AMD More demand for AMD is going to be a good thing for AMD

 

[HammerOn1024]

For weather forecasting?! In the imortal words of Monty Python "Pull the other one!"

 

[tomo82]

I think some of the reasoning behind this whether justified or not is a wish to get faster more detailed and localised predictions, for example, in the last 2 weeks or so we have had 2 huge storms coming in on the Jetstream, one a hurricane with months worth of rain in a day or two, huge parts of the country are currently flooded, most rivers throughout the country have burst their banks, many by up to 8ft of water, thousands homeless. Most of these places have flood defences, significant in some places but even they can't handle an extra 8 feet of water depth running in torrents, current forecasting says that areas of the country will get x amount of rain but we can't/don't break those areas down into small pieces and define within an area which towns/cities will get it the worst where some may not get it at all, not sure if this will work but it's in part the Government reacting to an outcry of home owners across the country asking what Government is going to do to make them safe as many of these homeowners cannot even get buildings or contents insurance because of where they live in "flood zones" and they cannot move because nobody wants to buy a house these days on a flood plain.

(I'm a brit) - TBH they could run it off a Pi1 - it just needs to display the word "SHIT". There - I saved you 1.2 billion quid. Maybe that will offset the cost of leaving the EU. Ha! who am I kidding.

 

[notb]

For things in general going at any price point, you do get more for your money with AMD over Intel at the moment

That's only if you look at CPU price floating in vacuum.

The fact that your needs can be covered with 50% less sockets (when going 64-core EPYC) means indisputable savings if the 64-core EPYC is a product for you.

Down the ladder it's not that obvious, because you end up with the same number of sockets (nodes, blades) either way, so possible savings are limited.
And it's really not obvious that a EPYC one is cheaper just because EPYC CPU is cheaper. Other aspects may mean that a company like Cray is able to offer Intel variant for less - even if the CPU alone is more expensive.

I just mentioned I hope they go with AMD More demand for AMD is going to be a good thing for AMD

You're a shareholder or what?

 

[ARF]

You're a shareholder or what?

It looks like the market reacts really positively on the news and the AMD stock is on the rise while Intel's stock is on the decline.

Advanced Micro Devices, Inc. (AMD) Stock Price, News, Quote & History - Yahoo Finance

Find the latest Advanced Micro Devices, Inc. (AMD) stock quote, history, news and other vital information to help you with your stock trading and investing.

finance.yahoo.com

 

[notb]

It looks like the market reacts really positively on the news and the AMD stock is on the rise while Intel's stock is on the decline.

At this point it's really hard to say why AMD is going up or down. Or why they're priced where they are.

It's certainly not based on this Met Office contract - even if we assume AMD is sure to get it.

 

[Chrispy_]

You're very lucky if you can get 1Gb for £65, I'm paying about half that for 30Mb!! The line we have at work is 1Gb up/down, I think that was something like £27k for so many years I think (dual lines different exchanges etc etc..

Well if you're stuck on Openreach's copper last mile then yes, you're completely stuffed
~30Mb/s is all you can hope for with fibre to the cabinet in the UK and BT/Openreach's decision to roll out FTTC as an upgrade to ADSL has probably set back the UK by a decade in the internet bandwidth race, IMO. I'm stuck on FTTC myself in london and at least 6 months away from a 1Gb/s HyperOptic FTTP installation.

We are, however, a minority. BT have installed FTTP to 4.1 million UK homes as of October 2019 and In addition to that, here are hybrid fibre/coax FTN networks. Take Virgin Media - who are already offering at least 108Mb/s to 75% of the UK, 53% of which is 360Mb/s capable network. Pricing is similar to Openreach FTTC, so although not quite FTTP it's good enough for most people at the moment. If you're not served by Virgin or Openreach's FTTP then you are probably one of the 15% who are either living out in the wilderness (and completely screwed until 5G can cover your internet bandwidth needs), or you're likely to be in the queue for an FTTP provider to cover your area.

As for business connections to the Met Office in Exeter, they are located are within half a kilometre of at least one Fibre ISP, and there are datacenters in the same radius for numerous bandwidth-hungry businesses including Amazon. I assume the Met office location was picked because it's such a well-connected, datacenter-heavy region, and the physical proximity will keep their internet costs down, not that it's remotely an issue given the running costs of the hardware itself!

In case you're wondering how I know this, my former employer (15 years ago) ran a datacenter in that area (Sowton Industrial Estate) and I spent years driving past all this every day.

 

[phill]

That's only if you look at CPU price floating in vacuum.

The fact that your needs can be covered with 50% less sockets (when going 64-core EPYC) means indisputable savings if the 64-core EPYC is a product for you.

Down the ladder it's not that obvious, because you end up with the same number of sockets (nodes, blades) either way, so possible savings are limited.
And it's really not obvious that a EPYC one is cheaper just because EPYC CPU is cheaper. Other aspects may mean that a company like Cray is able to offer Intel variant for less - even if the CPU alone is more expensive.

You're a shareholder or what?

I wish I was considering where they were 3 years ago or so... They have done amazing things and wonders with their hardware
I'm interested in finding out how they'd even put together a system like that, I just hope that they are going to use AMD hardware that's all It'll be a lovely push forward for them if they do I think

Well if you're stuck on Openreach's copper last mile then yes, you're completely stuffed
~30Mb/s is all you can hope for with fibre to the cabinet in the UK and BT/Openreach's decision to roll out FTTC as an upgrade to ADSL has probably set back the UK by a decade in the internet bandwidth race, IMO. I'm stuck on FTTC myself in london and at least 6 months away from a 1Gb/s HyperOptic FTTP installation.

We are, however, a minority. BT have installed FTTP to 4.1 million UK homes as of October 2019 and In addition to that, here are hybrid fibre/coax FTN networks. Take Virgin Media - who are already offering at least 108Mb/s to 75% of the UK, 53% of which is 360Mb/s capable network. Pricing is similar to Openreach FTTC, so although not quite FTTP it's good enough for most people at the moment. If you're not served by Virgin or Openreach's FTTP then you are probably one of the 15% who are either living out in the wilderness (and completely screwed until 5G can cover your internet bandwidth needs), or you're likely to be in the queue for an FTTP provider to cover your area.

As for business connections to the Met Office in Exeter, they are located are within half a kilometre of at least one Fibre ISP, and there are datacenters in the same radius for numerous bandwidth-hungry businesses including Amazon. I assume the Met office location was picked because it's such a well-connected, datacenter-heavy region, and the physical proximity will keep their internet costs down, not that it's remotely an issue given the running costs of the hardware itself!

In case you're wondering how I know this, my former employer (15 years ago) ran a datacenter in that area (Sowton Industrial Estate) and I spent years driving past all this every day.

It's even worse for me, aluminum cabling from the DSlam to my home.. I'm at about 700m I believe, I used to be getting upwards of 44Mb, now lucky if I can get 30Mb.. My sister who's up the road manages 65Mb and she and her husband are not even close to using it anywhere near to what I try and do with mine here If I don't laugh I'll be crying!!

Exeter isn't too far from me, but I'm not so fond about living in a busy city... But still, I digress

If only we had some more information about the build of the system etc.... I'd love to see what even goes in a system like this, let alone how it all works together Ah, being a hardware w^&%$ has it's fun spots I guess

 

[DeathtoGnomes]

I agree, Epyc would be the right choice.... but government loves to waste taxpayer dollars, so probably Xeons.

when you put it like that....I cannot disagree.

 

[Prima.Vera]

Everybody is talking about hardware, but nobody talks about the software.
I would like some good quality info like this:
- O.S. of the machine;
- The Software which is going to use;
- Who is going to program it, and how many parallel tasks can it run ?

 

[notb]

Everybody is talking about hardware, but nobody talks about the software.
I would like some good quality info like this:
- O.S. of the machine;
- The Software which is going to use;
- Who is going to program it, and how many parallel tasks can it run ?

It has been mentioned multiple times?

Met Office is running an atmospheric model (currently one called Unified Model) and delivering weather forecasts. That's the primary purpose of their current infrastructure (Cray XC40) which this is going to replace.

OS - whatever the supplier provides for the cluster (Linux-based).
It will be used by Met staff, obviously...
I don't understand the "how many parallel tasks" part. 64-bit Linux is limited to 4194304 (2^22).

 

[TheUn4seen]

i really can't imagine why this super computer is needed for forecasting; the satellites give us already the movement of air masses and the path also; sometimes is impossible to forecast in long term as a sudden change of air movement change all and no matter how powerful computer you use is not-predictable ; i'm sure the new one will forecast the same as the old one and both will be incorrect at long term forecast as you can't foresee how nature behave...

Satellite data is the input (or one of many inputs to be precise), which only gives current conditions. And weather forecasting has gotten fairly good over the years, but it's ridiculously demanding, hence the need for dedicated supercomputers. This is a problem which can be solved with throwing more computational power at it.
You can definitely forsee how nature behaves, it's called science, just as as physics can predict how galaxies behave, meteorology can predict how weather behaves. We don't live in fairy tale land, we can actually use our brains and tools to solve even the most complex problems. It's just another tool.

But, it's UK, so it's never going to happen.

 

[Imsochobo]

I can't believe you don't know why forecasting weather makes sense... Maybe that's not what you meant.

As for forecast quality:
Local forecasts (i.e. what you check to know if it's going to rain tomorrow) are very precise for 2-3 days ahead. This is not a computational limit - weather is just too random.
Large area forecasts (i.e. what will happen in your province/country as a whole - large pressure zones, hurricanes etc) can be modeled even few weeks ahead.

There is nothing better. It's a cluster. You make it as big as you need.
What you actually mean is: cheaper. Based on EPYC it may be cheaper.

What you forget is that - even if EPYC's offer better value, they also offer less supply.
So not every supercomputer can be built with AMD interiors. This is one of the fundamental issues behind current Intel pricing (in all segments). 7nm is still relatively limited.

no, AMD can offer chips, intel cannot.
if we want to buy 200 epyc cpu's we can get them in a week.
If we order 200 intel chips we wait 3 months minimum