Bug in AMD EPYC "Rome" Processors Puts Them to Sleep After 34 Months of Uptime

Easo · Jun 5, 2023

zmeul said:
live patching in GNU LInux exists, the servers don't need to be rebooted

No - because you still need to update firmware for hardware, chiefly BIOS. You ARE updating them, yes...?

OneMoar · Jun 5, 2023

no fixed planned?
its like amd doesn't want to sell datacenter chips
plenty of servers don't get rebooted for a year

Easo said:
No - because you still need to update firmware for hardware, chiefly BIOS. You ARE updating them, yes...?

brauh no
ipmi and idrac exist

zlobby · Jun 5, 2023

Yikes!

chrcoluk · Jun 5, 2023

TheoneandonlyMrK said:
While I agree , if you're not performing some sort of upgrade, maintenance or check, in three years of a server part in a normal IT department I would be surprised, filters need cleaning and three years is quite long to use the same server hardware ( considers own IT DPT), maybe though.

It's a surprising oversight for a server part.

@JAKra get a job a Cyberdyne, these firm's need this kind of thought

You be surprised, big projects I have worked on basically just add new hardware to the pool of servers, with the oldest stuff staying in place for long periods of time.

One year+ uptime not unusual either.

Jeager · Jun 5, 2023

JAKra said:
"It's Not a Bug, It's a Feature."
In case of AI uprising, all machines based on EPIC would automatically go to sleep after 34 months of uptime.
Humanity saved thanks to this "feature"!

Exactly + saving the planet from global warming, thanks you AMD !

Wirko · Jun 5, 2023

We will soon see, I mean in 7 months, if Zen 3 Epycs inherited the exact same bug.

Easo · Jun 5, 2023

OneMoar said:
brauh no
ipmi and idrac exist

Sure they do - and what is the message when you update BIOS in iLO? Reset power to the system. For HPE I can tell you that same happens with things like RAID controller, network card, FC card and disk firmwares (and all those are not iLO updatable).
You seem to be mistaking remote management chips with actual BIOS.

HD64G · Jun 5, 2023

One of the most harmless bugs ever. Servers don't get to sleep mode and no PC stays on for more than 1 year (or it gets neglected and gets damaged without proper maintenance).

Wirko · Jun 5, 2023

HD64G said:
One of the most harmless bugs ever. Servers don't get to sleep mode and no PC stays on for more than 1 year (or it gets neglected and gets damaged without proper maintenance).

On the other hand, an entire datacenter or a large part of it may go out in a timespan of a few minutes because all nodes were turned on at the same time. And it might be something mission-critical that never connects to the internet and does not need updates.

HD64G · Jun 5, 2023

Wirko said:
On the other hand, an entire datacenter or a large part of it may go out in a timespan of a few minutes because all nodes were turned on at the same time. And it might be something mission-critical that never connects to the internet and does not need updates.

I would like to learn from an official source if that bug happened to any of the installed servers. I guess not.

Dr. Dro · Jun 5, 2023

HD64G said:
I would like to learn from an official source if that bug happened to any of the installed servers. I guess not.

A late bug such as this was likely discovered and learned about in a production environment

HD64G · Jun 5, 2023

Dr. Dro said:
A late bug such as this was likely discovered and learned about in a production environment

Not going to argue on that but not sure about that. Are you?

Dr. Dro · Jun 5, 2023

HD64G said:
Not going to argue on that but not sure about that. Are you?

I can't prove it if that's what you're implying, but given its a bug that takes literally 3 years to manifest it's not so hard to believe that was the case. Or at a bare minimum, AMD must have one running 24/7 in a QA lab and ran into it

Scrizz · Jun 6, 2023

Dr. Dro said:
A late bug such as this was likely discovered and learned about in a production environment

yeah most likely a customer(enterprise) hit it and complained to AMD

AusWolf · Jun 6, 2023

Scrizz said:
yeah most likely a customer(enterprise) hit it and complained to AMD

I can imagine that.

"Hey, I haven't restarted our server for 3 years and now it's acting weird."
"Huh? You what now? 3 years?"
:roll:

Tsukiyomi91 · Jun 6, 2023

LMFAO.

zlobby · Jun 6, 2023

Dr. Dro said:
I can't prove it if that's what you're implying, but given its a bug that takes literally 3 years to manifest it's not so hard to believe that was the case. Or at a bare minimum, AMD must have one running 24/7 in a QA lab and ran into it

As a bare minimum, AMD (and everyone else for that matter) should at least evaluate what happens with counters and timers in border cases, and when.

Dr. Dro · Jun 6, 2023

zlobby said:
As a bare minimum, AMD (and everyone else for that matter) should at least evaluate what happens with counters and timers in border cases, and when.

Agreed. Edge cases indeed must be accounted for, and this one seems like it's a bug related to the CPU's low power mode functionality. Disabling CC6/arguing that many boards have CC6 off by default is one thing but, let's be honest, we live in the age where people make a big deal out of a few watts in their tree-hugging "save the planet" craze, and such functionality would likely be enabled by company directive if it's available, so it should, to the extent that it's offered, work.

Mussels · Jun 7, 2023

Uptime?
Naptime!

Definitely Errata, but also... highly unlikely to be a major issue for anyone.

claes · Jun 7, 2023

Not sure you mean “errata” here, but this editor is wonky as hell

Zareek · Jun 7, 2023

Running for a year without a reboot isn't abnormal for some servers. I should restate that. Running a year without rebooting isn't abnormal for some Linux servers. That being said, I have seen a server that ran for over 5 years without a reboot. It was not on purpose, just more like it got forgotten about and kept doing its thing without issues. I'm on the fence about this. Almost three years is a long time. Rome launched almost 4 years ago, I'm guessing that is why it isn't going to be fixed. I assume, since we are just hearing about it now, it wasn't in the first gen of EPYC. Is it still an issue? Is Milan or Genoa affected? Nice can of worms...

claes · Jun 7, 2023

Hell I once run a Mac mini server for five years without a reboot (only rebooted to update to APFS, probably unneeded). Have had a pi running for six now. Lucky not to have any power outages, only my nas and main are attached to UPSs. Idk why I am sharing this carry on

A Computer Guy · Jun 7, 2023

claes said:
Hell I once run a Mac mini server for five years without a reboot (only rebooted to update to APFS, probably unneeded). Have had a pi running for six now. Lucky not to have any power outages, only my nas and main are attached to UPSs. Idk why I am sharing this carry on

Don't forget to test your UPS batteries!

Wirko · Jun 7, 2023

Dr. Dro said:
Agreed. Edge cases indeed must be accounted for, and this one seems like it's a bug related to the CPU's low power mode functionality. Disabling CC6/arguing that many boards have CC6 off by default is one thing but, let's be honest, we live in the age where people make a big deal out of a few watts in their tree-hugging "save the planet" craze, and such functionality would likely be enabled by company directive if it's available, so it should, to the extent that it's offered, work.

Think about a large server installation or a HPC/supercomputing cluster. Each 2-processor node consumes a couple hundred watts on idle. It makes a lot of sense to put some nodes on sleep when the system is not fully loaded. I don't know if C6 (deep power down state) is used for that purpose or not but it seems to be the appropriate mechanism.

Aquinus · Jun 8, 2023

Fouquin said:
It's certainly impressive that this problem was discovered at all, as it suggests that more than a single system has been running for almost three years straight without a single restart.

Ehhhhh, when I first started my professional career I worked with a colo server with over 1000 days of uptime on it. It was a database server. That streak was broken when we switched from Rackspace to Google. Believe it or not, this is very normal for Linux, in particular with builds that can do live kernel patching while the system is hot.

System Name	Personal \\ Work - HP EliteBook 840 G6
Processor	7700X \\ i7-8565U
Motherboard	Asrock X670E PG Lightning
Cooling	Noctua DH-15
Memory	G.SKILL Trident Z5 RGB Black 32GB 6000MHz CL36 \\ 16GB DDR4-2400
Video Card(s)	ASUS RoG Strix 1070 Ti \\ Intel UHD Graphics 620
Storage	2x KC3000 2TB, Samsung 970 EVO 512GB \\ OEM 256GB NVMe SSD
Display(s)	BenQ XL2411Z \\ FullHD + 2x HP Z24i external screens via docking station
Case	Fractal Design Define Arc Midi R2 with window
Audio Device(s)	Realtek ALC1150 with Logitech Z533
Power Supply	Corsair AX860i
Mouse	Logitech G502
Keyboard	Corsair K55 RGB PRO
Software	Windows 11 \\ Windows 10

System Name	RPC MK2.5
Processor	Ryzen 5800x
Motherboard	Gigabyte Aorus Pro V2
Cooling	Thermalright Phantom Spirit SE
Memory	CL16 BL2K16G36C16U4RL 3600 1:1 micron e-die
Video Card(s)	GIGABYTE RTX 3070 Ti GAMING OC
Storage	Nextorage NE1N 2TB ADATA SX8200PRO NVME 512GB, Intel 545s 500GBSSD, ADATA SU800 SSD, 3TB Spinner
Display(s)	LG Ultra Gear 32 1440p 165hz Dell 1440p 75hz
Case	Phanteks P300 /w 300A front panel conversion
Audio Device(s)	onboard
Power Supply	SeaSonic Focus+ Platinum 750W
Mouse	Kone burst Pro
Keyboard	SteelSeries Apex 7
Software	Windows 11 +startisallback

System Name	Main PC
Processor	13700k
Motherboard	Asrock Z690 Steel Legend D4 - Bios 13.02
Cooling	Noctua NH-D15S
Memory	32 Gig 3200CL14
Video Card(s)	4080 RTX SUPER FE 16G
Storage	1TB 980 PRO, 2TB SN850X, 2TB DC P4600, 1TB 860 EVO, 2x 3TB WD Red, 2x 4TB WD Red
Display(s)	LG 27GL850
Case	Fractal Define R4
Audio Device(s)	Soundblaster AE-9
Power Supply	Antec HCG 750 Gold
Software	Windows 10 21H2 LTSC

Processor	5700X
Motherboard	ASRock Gaming X
Memory	2x8 3600 CL14
Video Card(s)	1080 SEAHAWK
Display(s)	ViewSonic XG2703-GS

Processor	i5-6600K
Motherboard	Asus Z170A
Cooling	some cheap Cooler Master Hyper 103 or similar
Memory	16GB DDR4-2400
Video Card(s)	IGP
Storage	Samsung 850 EVO 250GB
Display(s)	2x Oldell 24" 1920x1200
Case	Bitfenix Nova white windowless non-mesh
Audio Device(s)	E-mu 1212m PCI
Power Supply	Seasonic G-360
Mouse	Logitech Marble trackball, never had a mouse
Keyboard	Key Tronic KT2000, no Win key because 1994
Software	Oldwin

Bug in AMD EPYC "Rome" Processors Puts Them to Sleep After 34 Months of Uptime

Easo

OneMoar

There is Always Moar

zlobby

chrcoluk

Jeager

Wirko

Easo

HD64G

Wirko

HD64G

Dr. Dro

HD64G

Dr. Dro

Scrizz

AusWolf

Tsukiyomi91

zlobby

Dr. Dro

Mussels

Freshwater Moderator

claes

Zareek

claes

A Computer Guy

Wirko

Aquinus

Resident Wat-man

Processor	AMD Ryzen 5 5600@80W
Motherboard	MSI B550 Tomahawk
Cooling	ZALMAN CNPS9X OPTIMA
Memory	2*8GB PATRIOT PVS416G400C9K@3733MT_C16
Video Card(s)	Sapphire Radeon RX 6750 XT Pulse 12GB
Storage	Sandisk SSD 128GB, Kingston A2000 NVMe 1TB, Samsung F1 1TB, WD Black 10TB
Display(s)	AOC 27G2U/BK IPS 144Hz
Case	SHARKOON M25-W 7.1 BLACK
Audio Device(s)	Realtek 7.1 onboard
Power Supply	Seasonic Core GC 500W
Mouse	Sharkoon SHARK Force Black
Keyboard	Trust GXT280
Software	Win 7 Ultimate 64bit/Win 10 pro 64bit/Manjaro Linux

System Name	"Icy Resurrection"
Processor	13th Gen Intel Core i9-13900KS Special Edition
Motherboard	ASUS ROG MAXIMUS Z790 APEX ENCORE
Cooling	Noctua NH-D15S upgraded with 2x NF-F12 iPPC-3000 fans and Honeywell PTM7950 TIM
Memory	32 GB G.SKILL Trident Z5 RGB F5-6800J3445G16GX2-TZ5RK @ 7600 MT/s 36-44-44-52-96 1.4V
Video Card(s)	ASUS ROG Strix GeForce RTX™ 4080 16GB GDDR6X White OC Edition
Storage	500 GB WD Black SN750 SE NVMe SSD + 4 TB WD Red Plus WD40EFPX HDD
Display(s)	55-inch LG G3 OLED
Case	Pichau Mancer CV500 White Edition
Power Supply	EVGA 1300 G2 1.3kW 80+ Gold
Mouse	Microsoft Classic Intellimouse
Keyboard	Generic PS/2
Software	Windows 11 IoT Enterprise LTSC 24H2
Benchmark Scores	I pulled a Qiqi~

System Name	:)
Processor	Intel 13700k
Motherboard	Gigabyte z790 UD AC
Cooling	Noctua NH-D15
Memory	64GB GSKILL DDR5
Video Card(s)	Gigabyte RTX 4090 Gaming OC
Storage	960GB Optane 905P U.2 SSD + 4TB PCIe4 U.2 SSD
Display(s)	Alienware AW3423DW 175Hz QD-OLED + AOC Agon Pro AG276QZD2 240Hz QD-OLED
Case	Fractal Design Torrent
Audio Device(s)	MOTU M4 - JBL 305P MKII w/2x JL Audio 10 Sealed --- X-Fi Titanium HD - Presonus Eris E5 - JBL 4412
Power Supply	Silverstone 1000W
Mouse	Roccat Kain 122 AIMO
Keyboard	KBD67 Lite / Mammoth75
VR HMD	Reverb G2 V2
Software	Win 11 Pro

System Name	Nebulon B
Processor	AMD Ryzen 7 7800X3D
Motherboard	MSi PRO B650M-A WiFi
Cooling	be quiet! Dark Rock 4
Memory	2x 24 GB Corsair Vengeance DDR5-4800
Video Card(s)	AMD Radeon RX 6750 XT 12 GB
Storage	2 TB Corsair MP600 GS, 2 TB Corsair MP600 R2
Display(s)	Dell S3422DWG, 7" Waveshare touchscreen
Case	Kolink Citadel Mesh black
Audio Device(s)	Logitech Z333 2.1 speakers, AKG Y50 headphones
Power Supply	Seasonic Prime GX-750
Mouse	Logitech MX Master 2S
Keyboard	Logitech G413 SE
Software	Bazzite (Fedora Linux) KDE

System Name	Rainbow Puke Machine :D
Processor	Intel Core i5-11400 (MCE enabled, PL removed)
Motherboard	ASUS STRIX B560-G GAMING WIFI mATX
Cooling	Corsair H60i RGB PRO XT AIO + HD120 RGB (x3) + SP120 RGB PRO (x3) + Commander PRO
Memory	Corsair Vengeance RGB RT 2 x 8GB 3200MHz DDR4 C16
Video Card(s)	Zotac RTX2060 Twin Fan 6GB GDDR6 (Stock)
Storage	Corsair MP600 PRO 1TB M.2 PCIe Gen4 x4 SSD
Display(s)	LG 29WK600-W Ultrawide 1080p IPS Monitor (primary display)
Case	Corsair iCUE 220T RGB Airflow (White) w/Lighting Node CORE + Lighting Node PRO RGB LED Strips (x4).
Audio Device(s)	ASUS ROG Supreme FX S1220A w/ Savitech SV3H712 AMP + Sonic Studio 3 suite
Power Supply	Corsair RM750x 80 Plus Gold Fully Modular
Mouse	Corsair M65 RGB FPS Gaming (White)
Keyboard	Corsair K60 PRO RGB Mechanical w/ Cherry VIOLA Switches
Software	Windows 11 Professional x64 (Update 23H2)

System Name	Rainbow Sparkles (Power efficient, <350W gaming load)
Processor	Ryzen R7 5800x3D (Undervolted, 4.45GHz all core)
Motherboard	Asus x570-F (BIOS Modded)
Cooling	Alphacool Apex UV - Alphacool Eisblock XPX Aurora + EK Quantum ARGB 3090 w/ active backplate
Memory	2x32GB DDR4 3600 Corsair Vengeance RGB @3866 C18-22-22-22-42 TRFC704 (1.4V Hynix MJR - SoC 1.15V)
Video Card(s)	Galax RTX 3090 SG 24GB: Underclocked to 1700Mhz 0.750v (375W down to 250W))
Storage	2TB WD SN850 NVME + 1TB Sasmsung 970 Pro NVME + 1TB Intel 6000P NVME USB 3.2
Display(s)	Phillips 32 32M1N5800A (4k144), LG 32" (4K60) \| Gigabyte G32QC (2k165) \| Phillips 328m6fjrmb (2K144)
Case	Fractal Design R6
Audio Device(s)	Logitech G560 \| Corsair Void pro RGB \|Blue Yeti mic
Power Supply	Fractal Ion+ 2 860W (Platinum) (This thing is God-tier. Silent and TINY)
Mouse	Logitech G Pro wireless + Steelseries Prisma XL
Keyboard	Razer Huntsman TE ( Sexy white keycaps)
VR HMD	Oculus Rift S + Quest 2
Software	Windows 11 pro x64 (Yes, it's genuinely a good OS) OpenRGB - ditch the branded bloatware!
Benchmark Scores	Nyooom.

System Name	boomer--->zoomer not your typical millenial build
Processor	i5-760 @ 3.8ghz + turbo ~goes wayyyyyyyyy fast cuz turboooooz~
Motherboard	P55-GD80 ~best motherboard ever designed~
Cooling	NH-D15 ~double stack thot twerk all day~
Memory	16GB Crucial Ballistix LP ~memory gone AWOL~
Video Card(s)	MSI GTX 970 ~~GOLDEN EDITION~~ RAWRRRRRR
Storage	500GB Samsung 850 Evo (OS X, *nix), 128GB Samsung 840 Pro (W10 Pro), 1TB SpinPoint F3 ~best in class
Display(s)	ASUS VW246H ~best 24" you've seen FULL HD 1O80PP SLAPS~
Case	FT02-W ~the W stands for white but it's brushed aluminum except for the disgusting ODD bays; cries
Audio Device(s)	A LOT
Power Supply	850W EVGA SuperNova G2 ~hot fire like champagne~
Mouse	CM Spawn ~cmcz R c00l seth mcfarlane darawss~
Keyboard	CM QF Rapid - Browns ~fastrrr kees for fstr teens~
Software	integrated into the chassis
Benchmark Scores	9999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999

System Name	Gaming Rig
Processor	Ryzen 7 3800X
Motherboard	Gigabyte X570 Aurus Pro Wifi
Cooling	Noctua NH-D15 chromax.black
Memory	32GB(2x16GB) Patriot Viper DDR4-3200C16
Video Card(s)	EVGA RTX 3060 Ti
Storage	Samsung 970 EVO Plus 1TB (Boot/OS)\|Hynix Platinum P41 2TB (Games)
Display(s)	Gigabyte G27F
Case	Corsair Graphite 600T w/mesh side
Audio Device(s)	Logitech Z625 2.1 \| cheapo gaming headset when mic is needed
Power Supply	Corsair HX850i
Mouse	Redragon M808-KS Storm Pro (Great Value)
Keyboard	Redragon K512 Shiva replaced a Corsair K70 Lux - Blue on Black
VR HMD	Nope
Software	Windows 11 Pro x64
Benchmark Scores	Nope

System Name	Still not a thread ripper but pretty good.
Processor	Ryzen 9 7950x, Thermal Grizzly AM5 Offset Mounting Kit, Thermal Grizzly Extreme Paste
Motherboard	ASRock B650 LiveMixer (BIOS/UEFI version P3.08, AGESA 1.2.0.2)
Cooling	EK-Quantum Velocity, EK-Quantum Reflection PC-O11, D5 PWM, EK-CoolStream PE 360, XSPC TX360
Memory	Micron DDR5-5600 ECC Unbuffered Memory (2 sticks, 64GB, MTC20C2085S1EC56BD1) + JONSBO NF-1
Video Card(s)	XFX Radeon RX 5700 & EK-Quantum Vector Radeon RX 5700 +XT & Backplate
Storage	Samsung 4TB 980 PRO, 2 x Optane 905p 1.5TB (striped), AMD Radeon RAMDisk
Display(s)	2 x 4K LG 27UL600-W (and HUANUO Dual Monitor Mount)
Case	Lian Li PC-O11 Dynamic Black (original model)
Audio Device(s)	Corsair Commander Pro for Fans, RGB, & Temp Sensors (x4)
Power Supply	Corsair RM750x
Mouse	Logitech M575
Keyboard	Corsair Strafe RGB MK.2
Software	Windows 10 Professional (64bit)
Benchmark Scores	RIP Ryzen 9 5950x, ASRock X570 Taichi (v1.06), 128GB Micron DDR4-3200 ECC UDIMM (18ASF4G72AZ-3G2F1)

System Name	Apollo
Processor	Intel Core i9 9880H
Motherboard	Some proprietary Apple thing.
Memory	64GB DDR4-2667
Video Card(s)	AMD Radeon Pro 5600M, 8GB HBM2
Storage	1TB Apple NVMe, 4TB External
Display(s)	Laptop @ 3072x1920 + 2x LG 5k Ultrafine TB3 displays
Case	MacBook Pro (16", 2019)
Audio Device(s)	AirPods Pro, Sennheiser HD 380s w/ FIIO Alpen 2, or Logitech 2.1 Speakers
Power Supply	96w Power Adapter
Mouse	Logitech MX Master 3
Keyboard	Logitech G915, GL Clicky
Software	MacOS 12.1