Intel Statement on 13th and 14th Gen Core Instability: Faulty Microcode Causes Excessive Voltages, Fix Out Soon

[phanbuey]

No. Production just means that the environments are used by end users. TYPICALLY it has to be very stable and so forth, but it is not a requirement at all for some environment to be 'production'. Production workloads then are any workloads that are run in production environments.

Deployment environment - Wikipedia

en.wikipedia.org

Marketing jargon is its own thing, and has nothing to do with the topic. Saying something is "production class" is also just marketing.


Yes, smaller, but failure rate has likely been something around 1% for consumer stuff and 0.1% for xeon stuff - pretty small for both. For some production use it makes total sense to use non xeon parts, as they are faster and cheaper. Now suddenly the failure rates are 50% and the message from some users here has been like "They should have know", which is total bullcrap.

They're not 'users' though -- they're my colleagues - I work with people that do this, and I do this. They absolutely know what they're doing. And when they put consumer 'overclocking' SKUs, whether intel or AMD, CPUs into their server farm and servers start crashing and they get all outraged and start tweeting about their farm, I have 0 sympathy.

100% sympathy to the end-users and the SIs that Intel did screw over. My point was that "SERVER" crash rates in a conversation about desktop overclocking cpus is less relevant since other sub-optimal decisions had to be made for that to happen.

 

[trparky]

I know that this question is going to go off on a bit of a tangent but... Is anyone going to really trust the next generation of Intel chips to have no faults? Has Intel's trust been tarnished?

 

[Dredi]

They're not 'users' though -- they're my colleagues - I work with people that do this, and I do this. They absolutely know what they're doing. And when they put consumer 'overclocking' SKUs, whether intel or AMD, CPUs into their server farm and servers start crashing and they get all outraged and start tweeting about their farm, I have 0 sympathy.

100% sympathy to the end-users and the SIs that Intel did screw over. My point was that "SERVER" crash rates in a conversation about desktop overclocking cpus is less relevant since other sub-optimal decisions had to be made for that to happen.

Users in this example case would be the players playing the games on the servers that the game hosting entity manages. For that use, again, the consumer parts have been vastly superior to the xeon parts. Those are still headless servers and they are running production workloads. And those workloads do not need to be 100% stable, but 50% stable as with the 14th gen parts is VASTLY different to the one people have been used to in the last X years.

Your typical production system is probably not something that matches said use. Based on your description of colleagues using the systems, you are likely describing some internal environment, and not a production one.

Not every counter strike server needs to be hosted on a triplicated, geographically redundant xeon system with ecc memory. They are still servers and running production workloads. 50% failure rate. And then you comment ”they should have known”.. like how??

 

[MikeSnow]

They're not 'users' though -- they're my colleagues - I work with people that do this, and I do this. They absolutely know what they're doing. And when they put consumer 'overclocking' SKUs, whether intel or AMD, CPUs into their server farm and servers start crashing and they get all outraged and start tweeting about their farm, I have 0 sympathy.

100% sympathy to the end-users and the SIs that Intel did screw over. My point was that "SERVER" crash rates in a conversation about desktop overclocking cpus is less relevant since other sub-optimal decisions had to be made for that to happen.

It has nothing to do with sympathy, or what the owners of those servers should have or should not have been using. Servers have been brought into these discussions mainly to prove that not only regular consumers are affected. Until recently, Intel suggested it's either the motherboard manufacturers pushing the CPUs too far, or the consumers themselves. The discussion about servers showed that there was more than consumers or MB manufacturers doing "the wrong thing", because for server/workstation class motherboards it is less likely that non-recommended defaults or overclocks have been used. And that proved to be true, the CPU itself was the main cause of these issues.

 

[64K]

I know that this question is going to go off on a bit of a tangent but... Is anyone going to really trust the next generation of Intel chips to have no faults? Has Intel's trust been tarnished?

My following opinion has nothing to do with what I think Intel probably deserves. Just an observation of what people say on the internet and what they do being entirely different things sometimes and about how entrenched the Intel brand is with the vast majority of consumers.

With tech enthusiasts like on here maybe a few will not buy Intel for awhile. With the vast majority of consumers I don't think this is enough to switch. Probably what would have a greater effect on Intel going forward is an expensive class action suit. Shareholders don't like messy things like that.

 

[phanbuey]

Users in this example case would be the players playing the games on the servers that the game hosting entity manages. For that use, again, the consumer parts have been vastly superior to the xeon parts. Those are still headless servers and they are running production workloads. And those workloads do not need to be 100% stable, but 50% stable as with the 14th gen parts is VASTLY different to the one people have been used to in the last X years.

Your typical production system is probably not something that matches said use. Based on your description of colleagues using the systems, you are likely describing some internal environment, and not a production one.

Not every counter strike server needs to be hosted on a triplicated, geographically redundant xeon system with ecc memory. They are still servers and running production workloads. 50% failure rate. And then you comment ”they should have known”.. like how??

Because they are datacenter engineers and they know that a "production" counterstrike server that is making money <> your counterstrike server that you're running for funsies in your living room. And if you use ROG boards and K sku processors, yes your tic rate will be higher and your performance/$ higher, but your uptime will suffer. That is exactly why people that make money with their servers tend to use xeons. There are game servers using xeons and none of those people are crying on twitter rn.

That's the tradeoff -- they made that tradeoff purposely, and now that it's biting them and they're crying. This is like using walgreens power strips instead of PDUs and then crying about their uptime or when there's a defect in them. Yes 99% of the time it's fine, and yes if they're failing in people's homes its really bad... but you put one in your rack when everyone warns against that - so no it's not your fault that it melted while you were using it 'in spec' but the decision to use it there, if you require any kind of uptime, is extremely questionable.

It has nothing to do with sympathy, or what the owners of those servers should have or should not have been using. Servers have been brought into these discussions mainly to prove that not only regular consumers are affected. Until recently, Intel suggested it's either the motherboard manufacturers pushing the CPUs too far, or the consumers themselves. The discussion about servers showed that there was more than consumers or MB manufacturers doing "the wrong thing", because for server/workstation class motherboards it is less likely that non-recommended defaults or overclocks have been used. And that proved to be true, the CPU itself was the main cause of these issues.

But these are not "Server" cpus... so yes your point stands the CPU is 100% at fault, and the desktops crashing is bad, and the laptop users is even worse (imo since it makes the reason of "voltages too high" make no sense). But the all of the sudden datacenter people coming out and being like 'AND OUR SERVERS ARE ALSO CRASHING' -wait what... why are you putting K sku desktop chips in your server. And the response of "but not everyone needs a xeon/epyc" is not valid -- this is exactly why you need a xeon.

If you show up with at a construction site with your Home Depot Ryobi Drill that you bought for $70 and it burns out after 8 hours of usage EVERYONE there will make fun of you. They'll most likely do it as soon as you take it out of the bag. Use the right tool for the job.

 

[MikeSnow]

But these are not "Server" cpus... so yes the desktops crashing is bad, and the laptop users is even worse (imo since it makes the reason of "voltages too high" make no sense). But the all of the sudden datacenter people coming out and being like 'AND OUR SERVERS ARE ALSO CRASHING' -wait what... why are you putting K sku desktop chips in your server. And the response of "but not everyone needs a xeon/epyc" is not valid -- this is exactly why you need a xeon.

If you show up with at aconstruction site with your Home Depot Ryobi Drill that you bought for $70 and it burns out after 8 hours of usage EVERYONE there will make fun of you. They'll most likely do it as soon as you take it out of the bag. Use the right tool for the job.

Nobody said those people using them on servers expected no crashes on them, or Xeon level stability. Normal consumer level stability and failure rates would have been enough for those people, otherwise they wouldn't have used them. The problem is that these CPUs are not even consumer level, as they expected. In my opinion these CPUs are "do not use for any purpose unless you want to risk wasting time or money" level. At least until the new microcode is released.

 

[R0H1T]

But "6Ghz" has never been normal. Why would you go with something absolutely brand new & having basically zero reliability or history to lean on

The more you think this through the more you'll realize the guys doing this are a lot more at fault that you'd admit to!

 

[phanbuey]

Nobody said those people using them on servers expected no crashes on them, or Xeon level stability. Normal consumer level stability and failure rates would have been enough for those people, otherwise they wouldn't have used them. The problem is that these CPUs are not even consumer level, as they expected. In my opinion these CPUs are "do not use for any purpose unless you want to risk wasting time or money" level. At least until the new microcode is released.

Absolutely -- you're right they are consumer level and they failed on that level beyond any excuse.

They used them because they have a downtime tolerance and they save a ton of money and get substantially more performance/$. It's generally bad practice though, and many consider it in the "f@#% around and find out" territory. And oftentimes they find out-- not always through catastrophic degredation, but erratta, memory issues, bugs etc.

This one guy tweeted "Production systems need to be stable" and in the same tweet thought that most of the failures were on ROG Strix Boards.

That's like saying - "our commercial vehicles need to be stable" and then being like "most of our failures have been on the pink Toys for Tots Jeeps."

 

[agent_x007]

Production workloads, by definition, require stability and performance, but first and foremost stability. You can build a mac mini farm, or a rasberry pi farm, or a farm of blades running game servers run a specific workload and call it 'Production', but if your processor comes with marketing materials with the words 'Exxxtreme' or 'Overclocking' or 'Gamers' it is not a production class system.

That is a VERY dangerous statement, because you are telling non-pro consumers to expect a BSOD or a "locked" system once for a while, because it's "consumer grade" and not "pro" grade, if they did something "non-consumer" with it.

"Server workload", "consumer workload", is just that : A "workload".
If CPU can't handle a workload it supports under normal operating conditions, and using recommended/stock settings - it's simply defective.
Defective CPUs should be covered under warranty (unless it expired).

From past perspective, there are multiple users of old hardware (10 years+), that simply put a private Minecraft server on consumer grade hardware, and have no issues (only CPU side). Expecting something else from modern stuff is a downgrade to quality of CPUs that must NOT happen.
I DO NOT want a CPU that will die just after warranty, because it was designed to fail that way or was "not rated for something, so it died faster".
Same goes for "you run bad program on it, so it failed faster" <= this is NOT how things worked so far.

You can argue that manufacturers job is to keep in mind all workloads a client can run on it, and make sure CPUs they make can take them in minimum warranty period time frame. However, giving something extra (I own a working Pentium MMX CPU), is how Intel got to where it is today.
Giving up life expectancy of CPUs past warranty period for few more % of performance earlier in their life is insanely easy way to grind company market share into the ground in long term (20 years+). I just assume competition simply doesn't do it and who will buy CPU that is guaranteed to die right after warranty period ends - "pro-sumer" or "casual" doesn't mean anything here - when alternative will just work fine for just few more years.

 

[phanbuey]

That is a VERY dangerous statement, because you are telling non-pro consumers to expect a BSOD or a "locked" system once for a while, because it's "consumer grade" and not "pro" grade, if they did something "non-consumer" with it.

that's not a dangerous statement - that's exactly the defense that will be used in court...

"We lost money by putting these cpus in our production servers"
Lawyer: "And are these SKUs desinged for 24/7 operation and uptime in production servers?"
"No"
...

 

[agent_x007]

@phanbuey OK...
Q : Does Intel provide number for MTBF on CPUs ?

 

[Dredi]

but your uptime will suffer. That is exactly why people that make money with their servers tend to use xeons.

Yep. They used to suffer like 1%, while xeons were at 0.1%, a calculated trade-off. Now they suffer like 50% and ”they should have known”. Like how?

no intel lawyer is going to state that the K sku’s are designed to only last a few months of actual use.

 

[thesmokingman]

Lmao, from the other thread to this one, now the cope is blaming the chip users ie. devs using these chips in game servers. Lmao, the cope is at the max now.

 

[Dredi]

This one guy tweeted "Production systems need to be stable" and in the same tweet thought that most of the failures were on ROG Strix Boards.

That's like saying - "our commercial vehicles need to be stable" and then being like "most of our failures have been on the pink Toys for Tots Jeeps."

Quite many production vehicles are renault clios, toyota yaris’s, etc. Pizza delivery. Security crews. Etc.

Toyota hilux is not necessary for such tasks.

Lmao, from the other thread to this one, now the cope is blaming the chip users ie. devs using these chips in game servers. Lmao, the cope is at the max now.

”YoU aRe UsInG iT WrOnG”.

LMAO

 

[phanbuey]

Yep. They used to suffer like 1%, while xeons were at 0.1%, a calculated trade-off. Now they suffer like 50% and ”they should have known”. Like how?

no intel lawyer is going to state that the K sku’s are designed to only last a few months of actual use.

That's not what they're going to argue. They're going to argue that they're not liable for the damages because that product is not designed for that use - that their liability will stop at replacing the CPU.

"YoU aRe UsInG [the] WrOnG [chip/platform for your servers]”.

Fixed.

 

[Dredi]

That's not what they're going to argue. They're going to argue that they're not liable for the damages because that product is not designed for that use - that their liability will stop at replacing the CPU.

They are not even replacing the cpu’s …

that is the issue for some of the more audible affected users. The chips have been purchased in ’trays’, with limited warranties, and intel is being an ass and claiming that the failures are not defects covered by said limited warranties.

 

[phanbuey]

They are not even replacing the cpu’s …

They're 1000% getting sued though, so... they will.

 

[Dredi]

They're 1000% getting sued though, so... they will.

Hope so. One game hosting service has about 1000 RMA’s denied.

 

[Makaveli]

I'm not going to punish a company based on someone else's experience. Intel cpus have worked flawlessly for me so there is that, I'll keep buying until they don't.

This was probably the worse thing you could have posted in this thread.

So besides all the evidence lets just ignore it and keep our head in the sand?

All crediblity out the window!

 

[SRB151]

I can also offer a worse example: Sony.
I bought their Linkbuds S wireless earbuds in 2022 only for these to develop a battery discharge issue a week after my two year warranty period ended (both go from 100% to empty withing 15-30 minutes instead of usual ~8 hours).
Reading reviews and comments online there are many people facing the same issue with both the Linkbuds S and WF-1000 XM4 models produced and bought in 2022.

Yet Sony has not even acknowledged the issue nor provided any replacements for customers because in their eyes the warranty period for both products (for 2022 buyers at least) has ended and thus they feel they dont have to do anything.

A product failing a week after warranty ended feels like planned obsolescence...

If it was the same issue as my WF series buds, Sony did something similar to the auto manufactures, they would replace the internal batteries at no charge if you sent them in for repair without advertising/publishing it. Interestingly enough, kind of on topic here, one of the app updates sent out by Sony was the fix to keep the buds from wiping out the battery.

As for Intel, quiet options for handling this are no longer an option for them. Also, I read somewhere that as much as 1/3 of the 13th and 14th gen cpus are of the 13900 and 14900 models. Not sure if this number was for overall sales or just the boxed processors for end users. On top of that, one of the game developers were running about a 50% failure rate using W series server boards (read, no overclocking at all). I would assume with the latest bios patches as well.
What I don't get is why it has taken this long to "identify" the problem? Could Intel just be stalling for time to get past the Ryzen 9000 launch? Stall until Arrow Lake changes the narrative?

The other thing that has me wondering is that for Arrow Lake S, the latest news is that Intel is trying to push up the launch date by trying to get the QS samples out early and shorten the validation period. Sounds like they may not have learned any lessons from this experience.

P.S. For the record, the X370 bios issue was exactly about the size of the rom chips. When I upgraded my friends board, Gigabyte had two bios choices, early CPU's or later ones, If you updated for the newer chips, you could NOT load the older bios, and could no longer run 1st (and 2nd?) gen CPU's

Not a Defect: Intel Blames 13th, 14th Gen CPU Crashes on Software Bug (msn.com)

The part of a CPU with the microcode is called FIRMWARE, not software. It's not like you can just load it at will from anywhere, it's part of the CPU and (as it should be) not easily accessed or reflashed.

 

[efikkan]

"Server workload", "consumer workload", is just that : A "workload".
If CPU can't handle a workload it supports under normal operating conditions, and using recommended/stock settings - it's simply defective.
Defective CPUs should be covered under warranty (unless it expired).

From past perspective, there are multiple users of old hardware (10 years+), that simply put a private Minecraft server on consumer grade hardware, and have no issues (only CPU side).

The workload absolutely matters, and I think everyone knows that. Using consumer grade hardware for server use isn't going to cause legal problems for Intel.
Server grade hardware is certified for high load 24/7 for x years. Running a private Minecraft server isn't exactly something that causes sustained peak load all the time (but hosting a lot of them for a company may be a lot more). Running a machine 24/7 without a significant load isn't much of a problem, I've been doing that for most of my machines for over a decade, and like many others in the industry I too have a home "server" for files/media/git/building/etc., but it's nothing with sustained high load, if so I would have to choose hardware accordingly.

 

[MikeSnow]

The workload absolutely matters, and I think everyone knows that. Using consumer grade hardware for server use isn't going to cause legal problems for Intel.
Server grade hardware is certified for high load 24/7 for x years. Running a private Minecraft server isn't exactly something that causes sustained peak load all the time (but hosting a lot of them for a company may be a lot more). Running a machine 24/7 without a significant load isn't much of a problem, I've been doing that for most of my machines for over a decade, and like many others in the industry I too have a home "server" for files/media/git/building/etc., but it's nothing with sustained high load, if so I would have to choose hardware accordingly.

Ironically, in this particular case high load (multithreaded, all cores) could make these CPUs last longer, because they would reach the power limit and run the cores at lower frequencies than the maximum turbo boost, meaning also lower voltages. The worst would be loading just a core or two for long periods. And even keeping it mostly idle could result in periodic frequency and voltage spikes and some amount of degradation.

When I had issues with my 13900KF, which is now on its way to Intel, I could run prime95 on all the cores without any issues for hours. But even something as trivial opening a new browser tab, with the computer idle, could result in a crashed tab due to access violations.

If I had a computer with a 13th gen or 14th gen CPU that is not currently experiencing stability issues I would refrain from using it until the microcode update is released. If I really had to use it, I would probably limit the maximum multiplier to x50 or less.

 

[Tek-Check]

What is going to happen now that the genie is out of the bottle?
- will Intel publish serial numbers of CPUs affected by overvoltage and/or oxidation so that owners could identify them and file RMA?
- will second hand market go completely bonkers, as no buyer will know for sure whether they buy an affected CPU?
- how many online gaming companies will switch to AMD systems?
- will confidence in Intel brand and reliability suffer?
- so many questions...

 

[MikeSnow]

Intel seems to be brewing something regarding CPU warranties:

Processor warranty validation will be down for planned maintenance on the Online Service Center during July 24th, 2024, 8:00 PM (PST) until July 24th, 2024, 09:30 PM (PST).