SSD Defragging: The safe way

[evernessince]

I think I should patent "Continuous Defragmentation". Method and apparatus and all. But, as always, too many people have thought of useful things before I did. Such as these guys.

What I mean is: the file system driver should be gathering some statistics about the files being read. It would detect and count situations where a sequential read has to jump all over the disk because of fragmentation, and temporarily store (in RAM) some metadata about these problematic parts of files. That operation would consume very little time. Some time later, perhaps when the disk were idle, a process akin to a garbage collector would defragment only those specific parts of specific files, putting priority on fragments that were read most often, and also trying to reduce the fragmentation of free space along the way.

I would of course throw in some advanced (but still dumb) statistics, to make the whole process somewhat adaptive depending on what's going on on the disk. Violà, artificial intelligence!

Diskeeper used to outright prevent fragmentation completely, it made sure files were always written to disk in one continuous piece. Unfortunately they no longer offer it anymore and all it's features are now rolled up into their enterprise products.

 

[freeagent]

For me.. I dont use a swap file, I let GC do its thing, and I will TRIM it manually if I am about to run an SSD bench.. but I let windows TRIM on its own. Seems to work ok?

The only time I wipe the drive is to delete the install partitions to start fresh..

 

[A Computer Guy]

Well they all get worse as the disk gets fuller.

So basically we are back to file system overhead performance issues and this...

Thanks @Mussels for introducing me to defraggler. It's nice to know you can choose defrag individual files I never knew that.
This also allowed me to see how VMWare snapshots create a huge amount of fragmentation that I didn't know was happening before.
It will be interesting to see at the end of the month what it looks like after many snapshots.

 

[Wirko]

So basically we are back to file system overhead performance issues and this...

It happens with all kinds of storage, my garage included. I'm sure there's room for "a car" inside it ... but this doesn't mean there's room for a contiguous car.

 

[goms]

The scheduled defrag task is set to run the "optimize" job on every drive when it is triggered. Do defrag.exe -? in a command prompt and you can see for yourself that the optimize operation will do the appropriate defrag OR re-trim operation depending what kind of drive it is run on. Meaning that unless your ssd is wrongfully flagged as a hdd it will never be automatically defragmented by Windows.

You're completely wrong. Please read the Microsoft documentation. I posted the link a few posts above yours:

defrag

Reference article for the defrag command, which locates and consolidates fragmented files on local volumes to improve system performance.

learn.microsoft.com

Windows does in fact defragment SSDs sometimes, just like it does with hard drives.
Yes, it is true that Windows has no actual control over the physical location of data inside the NAND chips. Windows is moving files around so that they look continuous to the file system.

 

[claes]

Just to add, iirc, it only defrags if snapshots are enabled

 

[goms]

Interesting observation, Window's will defrag SSD's, but what you have observed from the timestamps of the logs makes sense as it should be only defragging heavily fragmented drives that utilise volume shadow copies, not every single SSD.

In addition I think it isnt a unconditional once a month thing, it actually checks the level of fragmentation.

Just to add, iirc, it only defrags if snapshots are enabled

Nope, I don't think so. Defrag happens even if snapshots are disabled.

The wording in the Microsoft article indicates that defragmentation is not limited to volumes with shadow copies enabled. In fact, Microsoft does not cite "shadow copies" at all. And that seems to agree with my own tests. Even an external USB drive (SSD + enclosure) was defragged once. What Microsoft says is that Windows is supposed to retrim drives once a week and defrag once a month. I suppose it works similar to hard drives, where it checks the defragmentation level and then acts upon it.
But the key word is "supposed to". In reality, Windows often skips the optimization routine. There were months where I didn't see any defrag and even no retrims. Apparently the scheduled task runs with low priority and there are scenarios where the task will not run:

 

[claes]

I’m basing that on the Hanselman blog post

The real and complete story - Does Windows defragment your SSD?

There has been a LOT of confusion around Windows, SSDs (hard drives), and ...

www.hanselman.com

Unfortunately I can’t find the source for this post or the relevant forum thread, nor is it clear if defrag will run when snapshots are enabled on all or some volumes

Edit: tbc, mussels is continuing a conversation that we began last year. I’m pretty convinced that windows is only defragging metadata but have no real reason to believe this

Are there wear problems from partitioning a SSD?

6% wear for ~46TB over 7-8 years. ---- In terms of unallocated, I did some research before making my partitions, and I have 140mb unallocated MSR, not sure if that's relevant at all. My original partition layout for the SSD and Windows was the following: GPT 480mb - NTFS, label: Recovery...

www.techpowerup.com

 

[A Computer Guy]

Nope, I don't think so. Defrag happens even if snapshots are disabled.

The wording in the Microsoft article indicates that defragmentation is not limited to volumes with shadow copies enabled. In fact, Microsoft does not cite "shadow copies" at all. And that seems to agree with my own tests. Even an external USB drive (SSD + enclosure) was defragged once. What Microsoft says is that Windows is supposed to retrim drives once a week and defrag once a month. I suppose it works similar to hard drives, where it checks the defragmentation level and then acts upon it.
But the key word is "supposed to". In reality, Windows often skips the optimization routine. There were months where I didn't see any defrag and even no retrims. Apparently the scheduled task runs with low priority and there are scenarios where the task will not run:

On my machines I have gotten into the habit of creating restore points to guard against bad driver updates as well as enabling the optimizer (it says weekly) and I can say I have observed instances where I thought the optimizer should have trimmed my drives but for whatever reason did not and I didn't see the expected 0 days since last retrim but things like 14 days ago , 28 days ago, 48 days ago, 72 days ago etc... usually then I just manually trim them.

Now I got to thinking this might be because, for example on my home server where I have actively checked for this for awhile, I only run it a couple of days a month for some tasks. So perhaps Windows isn't really relying on a weekly schedule as much as it is counting actual days of operation - this is/was my theory. In other words if I only used my computer twice a month it might take the OS 7 months before the scheduled trim would actually kick off even though is says weekly. Now that I know the activity is logged from reading this thread I could just go check the logs to see what it's doing.

 

[Mussels]

This made me curious if the differences you’re seeing in performance are due to trim or actual defragmentation. Does trim defragment? When defraggler says “optimize,” do they mean defragment? Suppose we may not know until Microsoft tells us.

Considering the number of fragments gets removed, yes it defragments.
Deleted files aren't deleted, but just removed from the file table - deleted from the index of a book while the pages are still full.
TRIM is part of garbage collection to reduce writes.

TRIM allows the OS to inform the drive about upcoming deletes in advance, so it can queue them up and save an erase cycle.

This a good quote on how that works

The SSD TRIM command simply marks the invalid data and tells the SSD to ignore it during the garbage collection process. The SSD then has fewer pages to move during garbage collection, which reduces the total number of program/erase cycles (P/E cycles) to the NAND flash media and prolongs the life of the SSD. NAND flash wears out due to the long-term effects of the P/E cycle, so reducing the number of erases can lengthen the endurance of the SSD.

TRIM doesn't defrag or delete files - it's actually a delay for deleting until a threshold is met (X% of NAND chip marked ready for deletion) so that the 'dirty' ones can wait to be erased and used for writes at the same time.


Garbage collection then comes along and writes to the ones with the most 'ready to trim' blocks so that you don't waste an erase cycle and then a write cycle later - it can wait and do them together.

When a user deletes a file, the OS sends a TRIM command to the SSD controller to tell it which data pages can be erased when the garbage collection process takes places. The TRIM command and the write command operate independently of each other.

I’m basing that on the Hanselman blog post

The real and complete story - Does Windows defragment your SSD?

There has been a LOT of confusion around Windows, SSDs (hard drives), and ...

www.hanselman.com

Unfortunately I can’t find the source for this post or the relevant forum thread, nor is it clear if defrag will run when snapshots are enabled on all or some volumes

Edit: tbc, mussels is continuing a conversation that we began last year. I’m pretty convinced that windows is only defragging metadata but have no real reason to believe this

Are there wear problems from partitioning a SSD?

6% wear for ~46TB over 7-8 years. ---- In terms of unallocated, I did some research before making my partitions, and I have 140mb unallocated MSR, not sure if that's relevant at all. My original partition layout for the SSD and Windows was the following: GPT 480mb - NTFS, label: Recovery...

www.techpowerup.com

And yep that's the thread the info came from: Windows defrags once every 30 days, but it has minimum criteria for what it will actually defrag.

Again, it's all about making people aware that SSD's can and do fragment and theres a serious performance loss when it happens.

Look at Starfield and how it cant even be played of a mech drive - it's files being fragmented could be the difference between smooth gameplay and massive stutter just because your drive was 80% full when you installed it.

 

[Wirko]

On my machines I have gotten into the habit of creating restore points to guard against bad driver updates as well as enabling the optimizer (it says weekly) and I can say I have observed instances where I thought the optimizer should have trimmed my drives but for whatever reason did not and I didn't see the expected 0 days since last retrim but things like 14 days ago , 28 days ago, 48 days ago, 72 days ago etc... usually then I just manually trim them.

Now I got to thinking this might be because, for example on my home server where I have actively checked for this for awhile, I only run it a couple of days a month for some tasks. So perhaps Windows isn't really relying on a weekly schedule as much as it is counting actual days of operation - this is/was my theory. In other words if I only used my computer twice a month it might take the OS 7 months before the scheduled trim would actually kick off even though is says weekly. Now that I know the activity is logged from reading this thread I could just go check the logs to see what it's doing.

Why do you think that frequent manual trim is necessary?

The most important mechanism is sending TRIM commands immediately after the deletion of a file. Everything else is done just as a backup measure, in case the SSD was not able to process those.

 

[Mussels]

Why do you think that frequent manual trim is necessary?

The most important mechanism is sending TRIM commands immediately after the deletion of a file. Everything else is done just as a backup measure, in case the SSD was not able to process those.

Early information on TRIM was technical, and then misunderstood and explained badly by youtubers/tech magazines as "defrag for SSD's"
TRIM extends life of an SSD while excessive writes harms them - and somehow they got lumped into 'TRIM is defrag for ssd' in common misunderstandings


Manual TRIM just tells it to get cracking on that write earlier, rather than when it's needed - manual trim is for when the OS isn't aware it's an SSD, and only really mattered in 7.

 

[Wirko]

Manual TRIM just tells it to get cracking on that write earlier, rather than when it's needed - manual trim is for when the OS isn't aware it's an SSD, and only really mattered in 7.

Windows 7 knows a thing or two about SSDs. It sends TRIM when deleting files. Whatever else it does, stupid or smart, related to re-trim and defrag, I don't care as long as my 850 EVO stays at 99% health (that's after 7 years and 28k power-on hours).

I don't know about Vista but XP of course is totally unaware of SSDs. I used to use Intel's utility to trim manually on my old XP PC. These days, it gets turned on once or twice a year. My right hemisphere wastes energy worrying about power-off retention, and left hemisphere knows that a MLC SSD can not ever possibly lose a single bit.

 

[Mussels]

Windows 7 knows a thing or two about SSDs. It sends TRIM when deleting files.

Which makes the file 'deleted' to the OS, while not yet deleted on the SSD.
Win7 did not support TRIM easily, lotta trouble with drivers and updates that killed the early SSD's before it was sorted out.

Peoples information on how TRIM works came from that era, and there was a LOT of confusion that's been repeated ever since

 

[Wirko]

Which makes the file 'deleted' to the OS, while not yet deleted on the SSD.

Hm, I'm sure we both understand well how it works, we just use different words to describe the same thing. By sending the TRIM command, the OS informs the SSD that specific sectors are no longer in use by the file system. Retrim just sends that same information again, which doesn't do any damage. How SSD controller uses that information is a secret, we can just make some reasonable assumptions. It maps sectors to blocks, gathers statistics for each block (% written, % trimmed, etc), and at a later time, as a part of GC, erases the blocks. The ideal case would be to erase blocks that were previously 100% written, then 100% trimmed (files deleted). Everything else is less than optimal, may require relocation of data, and increases write amplification.
What wears down the flash cells is erasing, not writing, because it occurs at a higher voltage and also takes longer (milliseconds), and of course can only be done on whole blocks, which are about 32 MB in size.

Win7 did not support TRIM easily, lotta trouble with drivers and updates that killed the early SSD's before it was sorted out.

I can believe that, SSDs were uncommon in 2009. Apart from that, many enterprise SSDs were PCIe cards, and Wikipedia does mention that Windows 7 "did not support this command for any other devices [except ATA/SATA] including Storport PCI-Express SSDs even if the device itself would accept the command". If the same was true of Server 2008 then it was a serious limitation.

 

[A Computer Guy]

Why do you think that frequent manual trim is necessary?

The most important mechanism is sending TRIM commands immediately after the deletion of a file. Everything else is done just as a backup measure, in case the SSD was not able to process those.

These days I might to manual trim after doing a lot of file I/O. Years ago I was noticing performance loss after moving around large Virtual Machine files, VM cloning, or DVD rips.

 

[Mussels]

Hm, I'm sure we both understand well how it works, we just use different words to describe the same thing. By sending the TRIM command, the OS informs the SSD that specific sectors are no longer in use by the file system. Retrim just sends that same information again, which doesn't do any damage. How SSD controller uses that information is a secret, we can just make some reasonable assumptions. It maps sectors to blocks, gathers statistics for each block (% written, % trimmed, etc), and at a later time, as a part of GC, erases the blocks. The ideal case would be to erase blocks that were previously 100% written, then 100% trimmed (files deleted). Everything else is less than optimal, may require relocation of data, and increases write amplification.
What wears down the flash cells is erasing, not writing, because it occurs at a higher voltage and also takes longer (milliseconds), and of course can only be done on whole blocks, which are about 32 MB in size.

I can believe that, SSDs were uncommon in 2009. Apart from that, many enterprise SSDs were PCIe cards, and Wikipedia does mention that Windows 7 "did not support this command for any other devices [except ATA/SATA] including Storport PCI-Express SSDs even if the device itself would accept the command". If the same was true of Server 2008 then it was a serious limitation.

I forgot the important part: WIndows 7 *Defrag tool* was the big issue back then.

A fast defrag only defragmented the worst files, while a full defrag would re-organise every single file on the drive - and THAT is what killed disks and caused the whole 'Do not defrag your SSD' misunderstanding in the first place.

TRIM is a side story to extend lifespan, not part of defragmenting at all.

 

[Mussels]

Found some better tools to demonstrate this in a simpler way for people.

I used a file naturally fragmented by a steam update, but you can artificially test this yourself with two programs:

1: Fragger
PassMark Fragger - File Fragmentation Utility

2. Disk Bench
Download Disk Bench - free - latest version (softonic.com)

This drive below (my gamines drive) has 64KB clusters to reduce fragmentation, on the default 4KB cluster you'll get more fragments harming performance more.

Fragments:
Test one is 1,680 fragments.

Test two is 1 fragment, after copying the file to another location on the same drive, defragmenting the new file and restarting the PC, to avoid any caching.

17% faster, with the file tested averaging around one fragment per 1.4MB of disk space used.



This should let people figure out around the point the fragmentation will become an issue.
We can average these results, but actual fragmentation could have hundreds of 4KB pieces rather than 1MB+ chunks.

Test this yourselves with the most fragmented file on your drives, read-only wont harm anything for the paranoid.

 

[pavle]

Fragmentation has always been a problem, SSD's aren't exempt; I observed speed of my system SSD drive go to one half of what it used to be without de-fragmenting.
Diskeeper is fine still, but they change their name as their underwear and now they want subscriptions to their ware...

 

[Klemc]

Disktrix Ultimate Defrag has an option for SSD :

 

[Deleted member 57642]

Disabled Defrag/Optimize shortly after installing Windows. Once every 7 - 12 days i select the SSD partitions and Trim/Optimize it on call. Also do that whenever i install a major update or a big app/game.

Bought an external HDD - for storage or downloading (anything) - for the sake of prolonging its life. Other than that - Browser caching and Windows Updates are left as the main culprits for wearing down the SSD (with time). Currently shown at 98% life status - but i don't know how accurate that is - since it's still fairly new tech (which makes those values more of logical estimation) and not all SSD are the same.

 

[CheapMeat]

I was just about to post the same Scott Hanselman article "The real and complete story - Does Windows defragment your SSD?" from a while back. But it was already mentioned. But I believe it even for newer drives and Win11 too, in terms of optimizing the metadata and how it's mostly happening in the background. The one thing I'm super curious about is how Optane drives are handled or how metadata is handled by the controller.

 

[Mussels]

Disktrix Ultimate Defrag has an option for SSD :

View attachment 324556

Thanks!

I'll look into that, it should be similar to what windows does - only touch files past a specific threshold (X fragments per GB)

I was just about to post the same Scott Hanselman article "The real and complete story - Does Windows defragment your SSD?" from a while back. But it was already mentioned. But I believe it even for newer drives and Win11 too, in terms of optimizing the metadata and how it's mostly happening in the background. The one thing I'm super curious about is how Optane drives are handled or how metadata is handled by the controller.

File systems are file systems.

Optane was intended only as cache so it was never going to run into fragmentation, rather it would be a a method to counter the slowdown by having already cached content.

Disabled Defrag/Optimize shortly after installing Windows. Once every 7 - 12 days i select the SSD partitions and Trim/Optimize it on call. Also do that whenever i install a major update or a big app/game.

Bought an external HDD - for storage or downloading (anything) - for the sake of prolonging its life. Other than that - Browser caching and Windows Updates are left as the main culprits for wearing down the SSD (with time). Currently shown at 98% life status - but i don't know how accurate that is - since it's still fairly new tech (which makes those values more of logical estimation) and not all SSD are the same.

That method works, but they'll only touch files every 30? days as some of this information has uncovered. It's poorly documented.

Seeing my Baldurs gate 3 load times shoot from ~10 seconds to ~30 seconds every patch til i defrag that one 10GB file every single update made it super clear how annoying this issue is.
And i'm on a system with 32GB of spare ram at any time on a PCI-E 4.0 NVME drive - that value would add zeroes on some poor bastard with a mech drive.

Seeing what i posted above: 540ms at 4,200MB/s

at 42MB/s that'd take 100x longer to load, at 54 seconds.
My Intel 6000P 1TB NVME drive can slow down to under 15MB/s when it's filled its cache, or just decides to hate me.

Fragmented files would make that even worse than that estimate, let alone on a system that doesnt have enough RAM to unpack a 12GB file and keep it there - imagine loading, discarding and loading assets non-stop in a RAM and VRAM starved environment. Shit would be so bad.



I've seen reports every single update from people on mech drives, full SSDs and budget SSDs where their game loads like this post-update after EVERY update

The files take so long to load the game engine gives up and doesn't render them.
These users find that reinstalling helps, but they've also learned that 'moving' the game from one drive to another magically fixes the issue - almost like doing so (partially) defrags the files.

This happens on the consoles too, they had to patch the Series S specifically because of it with lower quality textures for most clothing items.

 

[Deleted member 57642]

That method works, but they'll only touch files every 30? days as some of this information has uncovered. It's poorly documented.

Seeing my Baldurs gate 3 load times shoot from ~10 seconds to ~30 seconds every patch til i defrag that one 10GB file every single update made it super clear how annoying this issue is.
And i'm on a system with 32GB of spare ram at any time on a PCI-E 4.0 NVME drive - that value would add zeroes on some poor bastard with a mech drive.

Seeing what i posted above: 540ms at 4,200MB/s

at 42MB/s that'd take 100x longer to load, at 54 seconds.
My Intel 6000P 1TB NVME drive can slow down to under 15MB/s when it's filled its cache, or just decides to hate me.

Fragmented files would make that even worse than that estimate, let alone on a system that doesnt have enough RAM to unpack a 12GB file and keep it there - imagine loading, discarding and loading assets non-stop in a RAM and VRAM starved environment. Shit would be so bad.

It's still new tech (and keeps evolving/improving - bot speed and reliability wise) - and only recently (past 10 year or so) - turned mainstream (used as a standard storage unit in most laptops available on market - also many many times cheaper - thus, adapted as a standard at a large scale even among desktop users). Last time i checked - derangementing an SSD was deemed as a bad thing. Checked again and still is (deemed a bad thing / not recommended). As for Microsoft having a scheduled defragmentation even for an SSD - i don't trust it. Microsoft doesn't make SSDs nor it it specialized on storage maintenance (more focused on the average Windows user - thus features like - SuperFetch/SysMain and Indexing are activated by default - which can actually be useful for the average user using small apps frequently - yet, quite catastrophic for a power user or even a gamer dabbling with multiple apps quite large in size - which translated to 100% disk usage). Even the TRIM command - which, despite proven to be beneficial (if used the correct way) - was reworked in functionality - because it also proved to be quite bad for some SDDs (Samsung 840 and 860 in particular - didn't work so well with auto-trim - thus it was disabled by default) - if used to often or while actively using the SSD. That being said - if Samsung will recommend to defrag their SSD (maybe for some future model - that can be beneficial), sure - might give that a 2nd tought, bu so far:

No, Solid State Drives do not need defragmentation because they have no moving parts and can access any location on the drive equally fast. Please disable any defragmentation utilities on your computer because they will only wear down the performance of your SSD. Visit the OS Optimization section of Samsung SSD Magician for help doing this.

Same goes for Western Digital, which deems read intensive tasks as safe - while write intensive tasks like Defrag is also flagged as doing more harm than good in the long run (by shorting the lifespan).

DEFRAG is write-intensive. It can cause the premature wearing out or degradation of the NAND (flash memory) on SSDs as data is moved and re-written elsewhere in the NAND.

CHKDSK is read-intensive. It reads or scans the disks for errors and depending on your settings, can correct it using a very small write operation that does not negatively affect the NAND in the same way that 'defrag' might.

So hey, doing a windows update or while removing/installing some major apps (that's specially the case for every modern game - while taking their size into account) - i'll issue the TRIM command manually. After decades of using HDDs - the SSD tech is still a piece of modem tech which i find highly impressive (minutes even hours - turned into secconds).

As for Baldur Gate 3, great game - but honestly - does seem ratter poorly optimized at managing storage resources. Not to mention an SSD is a main requirment - even a Minimum requirment:

MINIMUM:

• Requires a 64-bit processor and operating system
• OS: Windows 10 64-bit
• Processor: Intel i5-4690 / AMD FX 8350
• Memory: 8 GB RAM
• Graphics: Nvidia GTX 970 / AMD RX 480 (4GB+ of VRAM)
• DirectX: Version 11
• Storage: 150 GB available space on an SSD
• Additional Notes: Default API is Directx11. Vulkan 1.1 API also provided.

This being one of the reasons why I opted for the GoG version (saw a streamer with the Steam version - and that comes with Cloud Savings - which in one of his streams took at least 10 minutes to save). Tho, not entirly satisfied with the GoG versions either for somehow similar reasons. As in - instead of a patch (even big one like 10 gb or so), there's times when a new version seem to download/install the whole game all over again. Stuff like this it's why i bought an HDD, which helps only partially - cause i still have to install the whole thing.

Anyway, can't say that i ever encountered that type of issues (as shown in above images), and this is an OEM nVME SSD (quite mediocre compared to latest models). Tho, satisfies my needs and still impressed with its performance.

 

[SPDIF]

Just use your SSD as it is meant to, you don't need any special tools for it. Just use it and don't look back! There's many things to say about it, but defragging means unnecessary wear, so don't do it. But it's your SSD, and your money, it's just advice.