Samsung 870 EVO - Beware, certain batches prone to failure!

[Ociya Syndor]

This reminds me of Maxtor Diamond Max 8 HDDs, (the slim models, the ones that look skinnier than the usual 3.5-inch sizing) which are failure prone. (The ones that are mainly model code 6Exxxxx)

While Diamond Max 9, IIRC, before Seagate bought Maxtor, were reliable. (ones with model code 6Yxxxxx were solid, OTOH)

yep! those & DeathStars

 

[chrcoluk]

@Daytrader
The whole idea that Samsung isn't aware of the failures that are happening across their entire 870 EVO series doesn't seem believable. There are plenty of reviews of 870 EVO's on newegg and amazon that mention these catastrophic failures.

It would be nice if Samsung would:
a. own up to the problems with their 870 EVO series
b. stop manufacturing said SSD's and/or recall the entire series

@chewie198
Those 4 TiB 870 EVO's aren't cheap either. What a clusterf*%k.

It needs a reviewer to draw attention to it, if they lose review samples for a couple of years so be it, this thread is on the TPU website, I wonder if someone with a defective SSD would accept sending their SSD if it meant getting media coverage, if I remember right, thats what triggered Samsung's acceptance of the 840 drives been broken after pcper covered it.

On a spindle once you get even 1 reallocated sector, you know its toast, research how they happen on spindles and can understand why, but not sure how these failures tend to work on SSDs in terms of the growth of defective sectors, but the only time I have seen the errors stop rising of their own accord is when they were not caused by the drive, such as bad SATA cable or memory errors.

 

[Mussels]

These drives will have revisions and different internals
They can have a variety of flash memory and controllers with revisions, but could also be made in different factories or countries

Of the screenshots on the last page for example, they have a bunch of various numbers - some are a little fuzzy but:

The PN "product name" is the same
the model is the same
The SN (serial number) is obviously different
WWN is the same until near the end - they both have 5002538F31 but then differ - is this a batch code? revisions?
PSID is totally different after first digit

If you can find any other differences between them from any programs, it'll be interesting what they are

If you ever end up opening them physical inspection would be fascinating too, for all we know it's like the covid GPU's and we'll find fingernails gloves and bandaids instead of thermal pads

 

[DZMBA]

Of the screenshots on the last page for example, they have a bunch of various numbers - some are a little fuzzy but:

Here's a clearer picture with the entire gang, if it helps.
I would open, take pictures, and inspect the internals, but the sticker states "Warranty void if label removed or case opened.". However, there's a law that states these stickers are unenforceable and completely illegal thanks to the Magnuson-Moss Warranty Act of 1975.
[Warranty-Voiding Stickers Are Illegal. | iFixit News]
But I don't want to have to jump through hoops or argue with them to get them to honor their warranty when I RMA eventually.
If someone knows this won't be a problem, then I'd be happy to open them up and take pictures to post here.

Both drives passed all Samsung Magician tests. But it does identify CRITICAL smart issues, even though they pass the SMART tests just fine.

---------------------------
UPDATE:

I've been running the drives through a gauntlet but the error counters haven't increased since fully wiping and formatting them. Makes me wonder if there's some kind of bit rot going on with infrequently accessed data. When I moved the files off originally, it did seem like all the CRC errors occurred with older files.
Since no new errors were happening, I decided to finally upgrade the firmware to SVT02B6Q and figured maybe I'll add one back to the pool and see what happens. But after the upgrade and before adding it to the pool, SAMSUNG Magician is finally saying "Failing LBA" and an error counter has increased, Though, a different one this time, CRC Error Count from 0 to now 4.

I can't quite figure out how to file a warranty claim. I thought maybe Magician would suggest something at this point. The website just runs me in circles and always leads me to the Download page for Magician - which is why I thought I needed it to report "Failing LBA" to get anywhere.
1. The "Request Service", "Schedule a Repair", & "Get Support" buttons for my registered products randomly shuffle around, and for products that are still in warranty, the "Request Service" & "Schedule a Repair" buttons never appear and it only presents the "Get Support" button.
2. Clicking the "Get Support" button takes you to "Recommended solutions", but only presents you with one option, "Get Information -> Get Started".
3. Clicking "Get Started" takes you to the Download page for Samsung Magician.

 

[chewie198]

I was able to pull some of my drives from the server today and found that the two drives manufactured in Jan 22 were already running the SVT02B6Q firmware, so Samsung must have started applying this from the factory in November or December. Nonetheless, the SMART characteristics from those drives were still reporting some CRC errors, although everything else error-related was at zero. TrueNAS had also been reporting errors on one of them as well, so I might need to do a little more digging on those two to figure out what's happening there. The one drive I managed to test from July 21 is displaying the expected set of SMART errors and high error counts, so I'll likely end up RMA'ing that batch of four from July 21 to start and proceeding from there. Has anyone had a RMA replacement or firmware-updated drive in service long enough to verify whether the problem has indeed been resolved? I'm curious whether anyone has had a drive last over 12 months without issue yet.

 

[80251]

It would be interesting to find out just what the problem is with the 870 EVO series. Does it have a unique controller that is only used by the 870 EVO series? Is the V-NAND used unique to only the 870 EVO series? Unfortunately I get the feeling no one outside of Samsung's engineering and senior management will ever know.

 

[Mussels]

Samsung will just shrug and say that's what warranties are for and for hardware revisions, these drives have no guarantees of no data loss and they'd just tell you to make backups

 

[chewie198]

It appears that the CRC errors I was seeing on the SVT02B6Q firmware may be due to buggy support for queued TRIM under Linux. According to https://www.phoronix.com/news/Samsung-860-870-More-Quirks, the 5.15 kernel blacklists that function on these drives, so I'll have to wait until I can update my TrueNAS-SCALE system to the 22.12.1 release to test that theory. Just another fun bug to deal with on these drives in the meantime.

 

[Mussels]

If you guys are running linux based OS's (the comments indicate several of you are) that might explain a lot of this, especially since they say it's worst for linux users with AMD chipsets


statistics would say most people are running them on intel chipsets in windows, and explain why they fail commonly for some people but not for others

 

[Potomac]

I received a SSD 870 EVO 500 GB replacement for the RMA I made last week,

I don't know if it's a refurbished SSD, it comes with a sealed package, like a new SSD,

production date : 2022-11-22, made in Vietnam
firmware : SVT02B6Q

SMART infos (the serial number is intentionally hidden) :

It seems a new SSD, no LBAs written, just "4" for the value of power on count, and "2" for "POR Recovery Count",

Do you think it's a new SSD or a refurbished SSD ?

 

[chewie198]

I received a SSD 870 EVO 500 GB replacement for the RMA I made last week,

I don't know if it's a refurbished SSD, it comes with a sealed package, like a new SSD,

production date : 2022-11-22, made in Vietnam
firmware : SVT02B6Q

SMART infos (the serial number is intentionally hidden) :

View attachment 274694

It seems a new SSD, no LBAs written, just "4" for the value of power on count, and "2" for "POR Recovery Count",

Do you think it's a new SSD or a refurbished SSD ?

Well with a Power-on Hours count of 0 and 0 LBA's written, and a manufacturing date less of last month, that's about as new as you can get. The non-zero power-on count might just be due to factory testing of the drive, but even if it's not, what difference would it make, practically speaking? If it were me I wouldn't be concerned about that unless it was showing other signs of use or throwing errors. Not to mention that it would be difficult, albeit not impossible, for Samsung to receive a failed drive, refurbish it, and send it back out in under a month unless they're either falsifying the manufacturing dates or simply zeroing out the error counts, replacing the label, and sending old drives back out. Not saying that I have evidence one way or the other, but no one thus far has suggested that anything nefarious is going on with the replacement process, only that the original firmware seems to have major issues.

If you guys are running linux based OS's (the comments indicate several of you are) that might explain a lot of this, especially since they say it's worst for linux users with AMD chipsets


statistics would say most people are running them on intel chipsets in windows, and explain why they fail commonly for some people but not for others

I think there are two separate issues here and they appear to be mostly unrelated, and I say that for several reasons. First, the drive failures indicated by the URE and ECC errors seem to be happening universally across both Windows and Linux, and in every case that I've seen have happened on drives manufactured prior to Dec 2021 running the SVT01B6Q firmware.

Second, the queued trim issues manifest as SMART CRC errors, while the URE and ECC counts remain at zero. This is the case on both of my drives running the updated SVT02B6Q firmware. Supposedly this only happens on Linux kernels prior to 5.15, although I haven't had a chance to verify that 5.15 resolves this issue. Lastly, there may be even worse performance and reliability issues associated with NCQ support on AMD SATA controllers under Linux, and kernel 5.15 supposedly resolves that as well. In my case, however, while these drives are running in an AMD EPYC server, they're all connected to LSI SAS HBAs which are pretty much the industry standard for NAS HBAs and do not appear to be affected by NCQ issues. Once I've had a chance to update to 5.15 and have either RMA'd or updated all of my 2021 drives I'll try to provide a follow up if that resolves the issue. Ideally it'd be nice to get some feedback from someone who's been running the latest firmware release for at least 6-12 months to see if there any residual long-term issues or if the problems have been completely resolved.

One positive observation I have regarding the SVT02B6Q firmware was that I was able to take a failed 4TB drive, updated the firmware first, then ran a full diagnostic scan. Despite finding hundreds of errors, the recovery process was able to complete successfully. If I'm able to resilver these drives into the ZRAID pool successfully and can run them error-free while I repair/RMA the remainder, that would be an indicator that this may be primarily a firmware issue. In which case the RMA process may be simply a way to zero the SMART error counts and clear out any bad block mappings (or maybe the firmware update takes care of that itself). Just speculating here.

 

[Daytrader]

I received a SSD 870 EVO 500 GB replacement for the RMA I made last week,

I don't know if it's a refurbished SSD, it comes with a sealed package, like a new SSD,

production date : 2022-11-22, made in Vietnam
firmware : SVT02B6Q

SMART infos (the serial number is intentionally hidden) :

View attachment 274694

It seems a new SSD, no LBAs written, just "4" for the value of power on count, and "2" for "POR Recovery Count",

Do you think it's a new SSD or a refurbished SSD ?

Mine came sealed and looked brand new, i think they are new myself as the date manufactured suggests.

 

[floogulinc]

A few days ago I came across a comment on reddit mentioning this issue, & I have four 2TB Samsung 870 Evos, two purchased in Feb & two in June. So I looked....
At the time I hadn't notice anything off yet, but apparently the batch from Feb had been blowing up the error logs for several months now. It wasn't until I actually tried getting data off them did the "Can't read from source file or disk" & "Cyclic Redundancy Check" user facing errors start.

The first image is what I posted in that reddit thread just 5 days ago. Those error rates will jump up quite a bit as I try to get data off. The second shows my current situation after diving in. I offloaded nearly all the data I could from the storage pool, but I still don't have enough redundancy to handle a double failure occurring in my largest drives, that are Samsung no less! I replaced old Crucial M4s that I thought weren't as reliable but were nothing but reliable for 7-8yrs. Samsung.... I don't know what to say. And I have 2 more of them that are just 2 months away from 7 months since purchase!

Here is the current state.

Manufactured October & November of 2021

My suggestion would be to just copy everything off that pool onto completely separate storage. Then you can just destroy the pool and make a new one without the bad drives. Windows should be able to recover most of all bad blocks when copying data off, or at least that's what happened with my storage spaces mirror of 3 870 evos.

 

[Mussels]

Warranty drives will always be new drives, they cant DO a repair to an SSD

 

[Potomac]

Not sure for 870 EVO series, I saw this message on reddit, he received a refurbished SSD for 860 EVO series :

https://www.reddit.com/r/buildapc/comments/rlmr5k

 

[Bandi]

First I had an 840 Evo, that had serious issues until Samsung fixed it with a firmware update.
Then I had an 970 Evo NVMe, that only works error-free on my AMD system with a special driver from Samsung.
Lastly I bought an 970 EVO, instead of the cheaper QVO, because I finally wanted a fast and reliable drive. And now this. I am sick of Samsung at this point.

 

[kevin335200]

SMART ID "FC" (252) is described on wikipedia as follows:

252 0xFC

Newly Added Bad Flash Block

The Newly Added Bad Flash Block attribute indicates the total number of bad flash blocks the drive detected since it was first initialized in manufacturing.[81]

Given this explanation, increasing this value seems like a problem.
I've been using the 870evo for over a year now and the "FC" value has been increasing recently.

I may have found what "FC", also ID 252 actually means for Samsung SSD: Read ECC Count
Source: PM863a datasheet

 

[DZMBA]

I may have found what "FC", also ID 252 actually means for Samsung SSD: Read ECC Count
Source: PM863a datasheet

I sure hope not.
That attribute is new as of SVT02B6Q firmware. My 2 "Good" 870Evos manufactured 2022.04 that came with SVT02B6Q both have a value of 1 there. The 2 bad 870Evos that I flashed to SVT02B6Q the other day show zero there, but perhaps it wasn't being tracked before.

 

[E.S]

I may have found what "FC", also ID 252 actually means for Samsung SSD: Read ECC Count
Source: PM863a datasheet

Thanks for the information.
If it's really ECC counts instead of bad blocks then I'm a bit relieved.

 

[kevin335200]

https://www.farnell.com/datasheets/2213316.pdf

Thanks for the information.
If it's really ECC counts instead of bad blocks then I'm a bit relieved.

Although not as bad as the bad block, this count is still problematic and it could still contain UECC.
If you do some Google search, you can find someone has uploaded PM9A3 firmware including its release note, which includes a situation that interval read may cause UECC.
Of course DC-class SSD firmware is much more complex than consumer-class, it may not be the same issue, but at least it shows that firmware defects can lead to read interference.
I will not put the link directly because it is confidental, just like the datasheet above, but it's easy to google it for now.

I sure hope not.
That attribute is new as of SVT02B6Q firmware. My 2 "Good" 870Evos manufactured 2022.04 that came with SVT02B6Q both have a value of 1 there. The 2 bad 870Evos that I flashed to SVT02B6Q the other day show zero there, but perhaps it wasn't being tracked before.

Although I don't have an 870 EVO to verify, based on its relatively small value, I think it should have been uncounted in previous firmware versions.

 

[E.S]

I'm worried if it's UECC.
If there is UECC, will that block be retired and consume spare blocks?

 

[chrcoluk]

First I had an 840 Evo, that had serious issues until Samsung fixed it with a firmware update.
Then I had an 970 Evo NVMe, that only works error-free on my AMD system with a special driver from Samsung.
Lastly I bought an 970 EVO, instead of the cheaper QVO, because I finally wanted a fast and reliable drive. And now this. I am sick of Samsung at this point.

View attachment 274917

Samsungs silence has them on the path to junk vendor for me. 3 Skus with major issues, 2 of them only acknowledged after pcper put on media pressure, silence on this once, no one wants to lose their review samples, press access etc.

 

[kevin335200]

Samsungs silence has them on the path to junk vendor for me. 3 Skus with major issues, 2 of them only acknowledged after pcper put on media pressure, silence on this once, no one wants to lose their review samples, press access etc.

At least there is still one key opinion leader who has cut through this issue from a data security perspective:
SSD in danger: Grinding down a Samsung 980 PRO 2TB until the RMA takes effect? Data security has priority! | igor'sLAB (igorslab.de)

I'm worried if it's UECC.
If there is UECC, will that block be retired and consume spare blocks?

It depends. If UECC is caused by read disturb, bit-flipping could be a non-permanent damage (e.g. the increased positive voltage during reading may interfere with the neighboring page, which can be eliminated), so after erasing, the block can be used again. Of course, if the ECC error rate of this block is very high (e.g. because the cell has low threshold voltage), the block may still be flagged because of threshold voltage degradation, which depends on the implementation of FTL in the firmware.

 

[Mussels]

I sure hope not.
That attribute is new as of SVT02B6Q firmware. My 2 "Good" 870Evos manufactured 2022.04 that came with SVT02B6Q both have a value of 1 there. The 2 bad 870Evos that I flashed to SVT02B6Q the other day show zero there, but perhaps it wasn't being tracked before.

ECC is a good thing, and far better than bad blocks

 

[DeusExMachina]

Unfortunately, I have similar experience with the EVO 870 4TB drives, with two failing consecutively. I used one as a backup drive but after a year started having issues. It was not clear to me then that it was the drive failing. I just saw intermittent device disconnection notifications on Windows but could not identify the cause. Then when I was upgrading to Windows 11, I noticed the 870 EVO kept disappearing/appearing on my Drives list. File transfers also kept getting interrupted or failing.

Incidentally I sent my system for repairs for other issues and the staff replaced the drive with another EVO 870 as it was identified as faulty. It's been less than a year since, and I wrote about 1 GB worth of data to the drive, but now it looks like it has issues too. Some files could not be moved nor opened. I ran Samsung Magician which said the drive was in "Good" health. The short scan ran fine, but the Smart tests kept failing/aborting with no recovery options. I tried firmware updates, but that also kept failing for the EVO. Did some Googling and it looks like the EVO 4TB has some bad batches.

I think my next replacement backup drive will be the Crucial MX500; rather not gamble on another EVO bad batch...