Samsung SSD 860 EVO 250GB - major sudden corruption of random files - advice needed

[eternalsadness]

I got an sudden file corruption on my Samsung 860 EVO 250GB SATA SSD.

While in Windows, I received a BSOD, I think this one: Bug Check 0x1A MEMORY_MANAGEMENT, then at reboot it said something about boot bcd 024 and it didn't want to boot. System repair couldn't do anything either.
Luckly, no ScanDisk was able to run over it(it would have botched the corrupted files forever).

1. I cloned the SSD using Easeus Partition Manager to a HDD. I connected the HDD to a 2nd computer, ScanDisk fired up and it was discovering lots of corrupted files, tens of thousands, probably, the screen was pouring.
After ScanDisk finished, and the 2nd computer booted, I could see my directories structures and filenames intact, I couldn't spot a single botched name, but many files were corrupted(filled with zeroes/other characters by ScanDisk), although not all of them. Probably more files are not corrupted than the ones that are corrupted.

2. I cloned the SSD again using Easeus Partition Manager to a new SSD. I connected this SSD to the 2nd computer, ScanDisk fired up "fixing" corrupted files. Of course there was nothing to fix, it filled portions of them with a single repeated symbol.

3. I connected both the SSD clone and the HDD clone and started comparing files using binary comparison, in BeyondCompare(I wanted to see how much extra corruption the second clone would bring). I was checking through the differences between the 1st clone and the 2nd clone and I noticed that the filling patterns for corrupted files were different, a different character was used for the 2nd clone.

4. But I also noticed that one picture is INTACT(viewable) in the 2nd clone and corrupt in the 1st clone. How could this be possible?
It means that the image is still stored correctly on the SSD, but readable correctly only sometimes.

Is there any hope for me to recover the files? Some firmware issue?
Are you guys aware of such issues with this model of SSD?

Right now I'm just cloning again and again, doing CRC32 hashes, trying to see if there are any other changes from one clone to the other(either some new files being read correctly, or even more corruption happening).
Any ideas how to proceed further?

This is the culprit, bought brand new from one of the largest retailers, so it's original 100%. Started using it 1 year ago, no firmware update, probably around 10TB of writes(I had pagefile and hiberfil active).

 

[Rock N Roll Rebel]

open cmd as admin then run these commands

Dism.exe /online /cleanup-image /restorehealth
SFC /scannow
chkdsk /scan
chkdsk /spotfix

To check Windows image for issues (Doesn't do repairs)
DISM /Online /Cleanup-Image /CheckHealth

more advanced scan
DISM /Online /Cleanup-Image /ScanHealth

Fix problems
DISM /Online /Cleanup-Image /RestoreHealth
- This might get stuck sometimes. Just wait for it to finish.

use offline image
DISM /Online /Cleanup-Image /RestoreHealth /Source:\Sources\install.esd

to disable windows update in the scan add (/LimitAccess)

SFC

to just verify run
sfc /verifyonly

to fix run
SFC /scannow

 

[eternalsadness]

I cannot run chkdsk on that SSD in this stage, because I will corrupt the data.
I ran chkdsk on low level clones of it(easeus partition master - copy disk), and it made folders become accessible, files being visible, but lots of them are "repaired aka filled with zeroes/dots/same-char strings.

Any ideas what could have happened to this SSD to corrupt so many random files scattered all over the directory hierarchy?
I looked a bit at the specs and it uses a top-notch TLC flash/controller, compared to other brands.
It doesn't have PLP(power loss protection) though. Could this missing feature cause this disaster?

Has anyone researched a bit what usually causes Samsung EVO's to fail or get corrupted?

 

[K²mix]

Post a full CrystalDiskInfo screenshot of this drive.

 

[eternalsadness]

How should I obtain that screenshot without using Windows?
I don't want to risk running ScanDisk over it at boot time.

Is there some linux based liveCD option that doesn't touch the SSD?

 

[wNotyarD]

Is there some linux based liveCD option that doesn't touch the SSD?

No live (CD or USB stick) Linux distro should do anything to the SSD during boot, only when you were to mount it or get its S.M.A.R.T. data with the OS running

 

[PassportBro]

Τhis has happened to me b4 while in the process of overclocking and mem tuning. My Samsung 840 became unbootable & corrupted and so i had to reinstall windows and lose some of my files.

This was the last time though that something like this happening would inconvenience me as I bought Acronis True Image and started systematically backing up my 2 SSD's (Samsung 750 boot drive & 870 games only drive) weekly onto my HDD Raid 1 array.

 

[eternalsadness]

How to get the SMART data in the most harmless way?

 

[nageme]

Annoying. Did you have at least partial backups?

It's a relatively old drive, manufactured in 2018. Was it formatted a year ago?

Difficult to say what went wrong. SMART might indeed shed some light.

Is there some linux based liveCD option that doesn't touch the SSD?

I don't think any would write to it unless explicitly told to.
Ubuntu should do. Maybe the stock Disks app is enough. If not, install smartmontools.

 

[eternalsadness]

This is how it looks. What do you see guys? There's nothing suspicious in my opinion, but I don't have experience.
Sorry for the quality, I had to take photos of the monitor.

 

[Rock N Roll Rebel]

I was visiting my sister back in July her system had a 500 GB m.2 and it was full of data she was having trouble with it so I removed one app that she did;t care about and got enough space to install Acronis
then I cloned her drive onto a 1 TB drive now she has more space and her system works perfectly. so maybe your m.2 is full

 

[eidairaman1]

Stop what you are doing, back up your data and get a MX 500 or Samsung Pro Drive or a HDD at this rate

 

[eternalsadness]

I posted above the SMART screenshots.

 

[efikkan]

Your BSOD indicates memory errors, and your SMART data indicates no particular issue other than a drive approaching the end of its life. (which is deserved after 584542 years and 16 days!)

I would put it into another machine and dump it again, then run a full batch of memory tests on your system.
Edit: Your Ubuntu boot medium includes Memtest86+, which is much better than the built in test for Windows)

 

[eternalsadness]

I ran Memtest86 for 1 pass, over an hour(32GB of RAM), zero errors.
I know that usually you should run it for a day or so, if you really want to be thorough, but I didn't have the nerves at that point.

Plus, the corruption has happened across lots of files, and various directories that were written in different periods over the past year.

 

[nageme]

I'm not sure why SMART 235, and especially 241 (total sectors written), say "N/A" instead of showing data. Though the normalized value is 99, which should be good.
I'm not familiar with "Disks". Maybe smartmontools (smartctl) would be more informative.

Other than that, nothing seems remarkable in the SMART stats.
I'm guessing the PC was powered off/on, or possible just sleep, multiple times per day?

The only thing I can think of, if the OS was installed 6 years ago, then the drive sat in a drawer for 5 years, and it was put into actual use only a year ago, maybe it's a data retention problem.

Also research if there were any reported firmware problems, depending on revision.

your SMART data indicates ... a drive approaching the end of its life.

I don't see EoL indications.
It reports 10 months of power-on hours, and a Wear Level Count at ~87% (raw value of 230, which is supposedly the average number of P/E cycled used up).

(which is deserved after 584542 years and 16 days!)

Probably a bug in the software, related to running from a LiveCD.

Your BSOD indicates memory errors

I don't think there's a way to know what it meant, based on the findings so far. Anything can cause it.
A theory: if the immediate trigger was some memory-related fault, it could also have been caused by corrupt data on the SSD leading to unexpected kernel behavior.

 

[eidairaman1]

Tried a different sata cable?

 

[nageme]

@eidairaman1
Wouldn't hurt to check, though SMART 199/0xC7 reports no interface CRC errors.

More interesting: a different SATA port, or controller. Or through a USB enclosure.

 

[eternalsadness]

I changed the cables first thing, before started cloning.
For sure it ended up on a different port than initially, I disconnected it multiple times.
I cloned it 3 times, the directories that are affected and "fixed" by scandisk to be at least accessible, are the same for all clones.
Other SSDs work fine on this computer, with their own Windows installation.

I could try to use another computer.

However, this intrigues me:

I noticed that one picture is INTACT(viewable) in the 2nd clone and corrupt in the 1st clone[both clones had ScanDisk fixes, otherwise the picture it was in an inaccessible directory]. How could this be possible?
It means that the image is still stored correctly on the SSD, but readable correctly only sometimes.
Sadly, I haven't noticed this for other files. Maybe I need to do more cloning...

 

[efikkan]

The only thing I can think of, if the OS was installed 6 years ago, then the drive sat in a drawer for 5 years, and it was put into actual use only a year ago, maybe it's a data retention problem.

Data rot should show up as errors, when the files are accessed. (Edit: unless the drive is so corrupted that it doesn't keep the statistics correctly)

I don't think there's a way to know what it meant, based on the findings so far. Anything can cause it.
A theory: if the immediate trigger was some memory-related fault, it could also have been caused by corrupt data on the SSD leading to unexpected kernel behavior.

When the drive reports no signs of errors, yet the files are read with errors, that's a good indicator that something outside the drive is the problem. With Windows reporting memory corruption, it's usually a fairly safe bet that something is causing the memory to corrupt, which usually means the memory itself, although it can also be the CPU or the motherboard (sometimes just reseating the memory might be enough). The only other explanation I could come up with would be corrupted system files from an update or malware, but that's fairly unlikely and not where to start.

I would start with a solid round of memory tests, and then (if needed) CPU tests. (I try to do this periodically on my systems)

Only after I've done every attempt to recover files from the drive, I would execute an "extended" SMART test (you can find that in the Ubuntu disk utility). It will take a while, but it should trigger any errors if there are bad blocks etc., just make sure you run this on a stable system.

 

[nageme]

I noticed that one picture is INTACT(viewable) in the 2nd clone and corrupt in the 1st clone

If both clones were created on the same PC (and with the same OS install), maybe try a different one.

Also, is the bad data file always full files, or sometimes just parts of them?

Data rot should show up as errors, when the files are accessed. (Edit: unless the drive is so corrupted that it doesn't keep the statistics correctly)

Makes sense. Drive-reported CRC errors.
But it seemed bad data may have variably reached Windows, if it was non-0: "zeroes/other characters".

 

[efikkan]

@eternalsadness:
Have you tried cloning the drive on the second computer?

Also, it is possible from your Ubuntu drive to mount the file system as read only, let us know if you want know how to do that (some terminal commands).

Makes sense. Drive-reported CRC errors.
But it seemed bad data may have variably reached Windows, if it was non-0: "zeroes/other characters".

A working drive will know whether a sector read is bad or not (the OS doesn't), and it should keep statistics of this in the SMART data, unless we're talking about a terribly broken SSD, beyond anything that I've ever seen.

According to the indicators we've seen so far, the data is corrupted in memory, which would also explain why some files are read fine one time, and not another.

 

[eternalsadness]

I need the commands how to mount it as readonly, if you know them. It's ntfs.
I will try it on the 2nd computer, but i don't think it will make a difference.

Also, as i told you, the corruption is in files all over the place, not in a single directory. It couldn't come from the computer's RAM, it would make more sense if the SSD controller tried some operation on those V-NAND cells that have to be operated in bulk, and somehow it botched the data.

Some enterprise SSD's have Power loss protection, this being a consume one, it doesn't have. Could it be affected somehow by missing PLP?
I had a power outage 1-2 weeks ago. And some BSOD's once in 1-4 weeks. But I had BSODs with all the SSD's that I had, over the years, over multiple PCs, but this would be my first SSD with data loss.

 

[nageme]

I don't think it's about losing power. Many people lose power occasionally, so SSDs, HDDs, controllers, operating systems are designed with that in mind.
I'd expect the only loss to be what's being actively written when power goes out.

NTFS core structures should be updated in a way that's resilient too, in theory.
I did encounter losing a partition due to a strange localized logical corruption (not power loss).
But it didn't ruin data all over the drive, and it was mostly recoverable, and the data didn't "flicker" like what you're reporting.

Multiple BSODs per month is definitely not normal.

To mount readonly:

How to mount a hard disk as read-only from the terminal

How do you mount a hard disk from the command line as read-only? I don't want or need a link to the man page, I want the exact thing I will have to type in if the following is true: disk to mount...

askubuntu.com

 

[efikkan]

Also, as i told you, the corruption is in files all over the place, not in a single directory.

Just for clarity, in your first clone, did you notice corrupted file names or simply missing files/folders?

It couldn't come from the computer's RAM, it would make more sense if the SSD controller tried some operation on those V-NAND cells that have to be operated in bulk, and somehow it botched the data.

If that's the case, then the controller must be defective. It certainly will, and should move things around to avoid data rot and for wear leveling.
But since every sector have parity bits, when you then try to read the sector, the SSD will detect the error and report it to the OS, as well as showing the count in the SMART data.

It sounds to me that this corruption happened for some reason in software, and the corrupted data were written back to the SSD. If the OS for any reason (faulty software or HW) writes garbage data, the SSD will not know the difference.

Some enterprise SSD's have Power loss protection, this being a consume one, it doesn't have. Could it be affected somehow by missing PLP?

PLP is certainly something that prevents file system corruption.
The damage from power loss is usually limited to the files written to in the last few minutes(?) before a power loss, and considering NTFS is a "journaled" file system, in theory the rest shouldn't be impacted. So, of all the files corrupted, were they modified right before your power outage? (or is it any random file up to several years old?)

(I absolutely think PLP is important, but we'll get back to that if relevant)

I had a power outage 1-2 weeks ago. And some BSOD's once in 1-4 weeks. But I had BSODs with all the SSD's that I had, over the years, over multiple PCs, but this would be my first SSD with data loss.

The one BSOD you mentioned "Bug Check 0x1A MEMORY_MANAGEMENT", and if you remembered correctly, that one only happens if there are severe memory issues, no SSD can cause that.

It would be very useful for diagnostics to know whether the other BSODs, both recent and historical, have shown random messages or a pattern of specific messages, as they do in fact point you in the right direction.

Generally speaking, BSODs can be caused by a number of reasons, including bad hardware, bad drivers, bad connections (cables or plugs), overclocking, underpowered PSUs, overheating, malware, Windows bugs, etc. It can even be multiple unrelated issues. Without more information about the crashes, and preferably your hardware config, it will be hard to offer specific guidance.

But I'm a firm believer that PCs shouldn't crash, and you should never settle for an unstable computer. If something keeps crashing, you should get to the bottom of it and resolve it.

I need the commands how to mount it as readonly, if you know them. It's ntfs.
I will try it on the 2nd computer, but i don't think it will make a difference.

If you get stuck, let me know at which step.
Step 1 - find the device name for the partition.
In the "disks" utility, right before you opened the SMART window, it will show you the partitions and their names.
There will probably be several tiny partitions and one large. Click the big NTFS partition, and locate the name of the partition below. (It will not be "C:\", but probably /dev/sda1, /dev/sda2, etc.)
Look at this picture for an example:

Step 2 - open a terminal (usually Ctrl + Alt + T)

Step 3 - make a temporary mount point

sudo mkdir -p /mnt/ntfs

Step 4 - mount the partition read only:

sudo mount -o ro -t ntfs /dev/sda1 /mnt/ntfs

Important: replace /dev/sda1 with whatever you found in step 1.

Step 5 - open the file manager and browse the drive