SSD, compact.exe saves filesize %, what about over-provision ?

[Klemc]

If compact.exe can compress a file(s) to 15/45% less his original size (when simply copied), my question is what about this gained space, did it count as written space to the hardware, and SSD things it's full ?..

... so if an SSD is 99% full, using compact.exe if lucky can get back 40% of the total space (so it's only 60% full after), then the question is about the empty space after compression why can't it be found/used by the SSD (overprovision...) ?

//

What's in all those empty data, i mean, a 50 gb file is given... then compact.exe compresses it to 35% less written space being now 35 gb, why the file is 50 gb in the first place, what's an empty data doing there ?.. why does empty thing

takes place, why the non compressed file is huge, is it emptyness ?



I'd like to understand a bit, thanks.

 

[DirtyDingusMcgee]

Data Compression

LZW

Huffman Compression

how file compression works

This is an over-simplification.

60gb hdd (or ssd) with a 50gb data file, so 10gb of free space. Data file compressed to 40gb, so now you have 20gb free space.

You would be able to use that 20gb of free space, and it would be 20gb of free space, since the 50gb file is only 40gb now.

If you added another file to the drive that was 15 gb (so only 5gb free space now), then you would be unable to decompress the large file until sufficient space was available.

... so if an SSD is 99% full, using compact.exe if lucky can get back 40% of the total space (so it's only 60% full after), then the question is about the empty space after compression why can't it be found/used by the SSD (overprovision...) ?

It wasn't empty space until you compressed it. Overprovisioning is a totally different thing.

Overprovisioning

Perhaps you are asking "why isn't it compressed as much as it can be from the start?", and that can be done, but there is some serious system overhead involved in real-time compresssion/decompression.

Real time compression - this article is specifically about IBM real time compression, but has it great info about the subject in general and storage footprint reduction.

 

[nageme]

As far as I know, only SandForce's SSD controllers used compression, years ago:

SandForce Announces Next-Gen SSDs, SF-2000 Capable of 500MB/s and 60K IOPS

www.anandtech.com

Files that are compressible aren't empty, usually. They're just compressible.
Compression is used in a lot of data formats, and in some networking protocols, but it's not always practical because of various factors.

For example, no compression means using less processing power, or less electric power, or having the ability to read or write data randomly and in small chunks (possibly 1-byte precision), or less latency, or better recoverability in case of corruption. So it's a compromise, used when saving space or bandwidth is more important.

Some formats that use compression:
JPEG, PNG, GIF, WebP, MPEG* video and audio, Vorbis, Opus, FLAC, RAR, 7Z, ZIP and many formats that use ZIP like DOCX, APK, JAR...

 

[R0H1T]

so if an SSD is 99% full, using compact.exe if lucky can get back 40% of the total space

You won't save that much even if you're lucky, the biggest savings are from executables/binary files & you can't compress most of them on a System drive. The compression for regular media files won't work as you have to transcode or convert them, regular NTFS compression will only save you a very small amount.

takes place, why the non compressed file is huge, is it emptyness ?

The difference mainly comes from the default allocation size vs actual amount used by files, also a lot of space used to be wasted because of default sector size back in the day for HDD's ~

Advanced Format - Wikipedia

en.wikipedia.org

 

[Wirko]

Are you just experimenting, or do you have a real use case for NTFS compression?

Most large files are compressed internally, as others have explained already. There are exceptions such as some video/audio/photo raw files and text log files - in general, anything that is meant to be changed, updated or appended. Those should NOT be compressed in the file system because any update will lead to very bad fragmentation. The main reason for fragmentation is that the compressed file size is not constant but depends on the content.

So if you identify any files that are large, never change, and are not internally compressed, those would be good candidates for NTFS compression. But I have no idea what those could be.

Apart from compact.exe, you can also turn the compression on or off in Windows GUI. It does the same thing. Right click on the file or folder, Properties, General tab, Advanced button, Compress contents to save disk space checkbox.

 

[Klemc]

Video games compress well (i use compression on storage that have video games on it, nothing else), the max i seen was 60%, in the end on the 500 GB Steam-Deck's SSD i won 30% more free space compared too raw data.

As an exemple (not on my SD btw) "The Sinking City" needed an update to see the game size near halved (year after elease 2019-6 > 2021-1).

Depot 750132
TSCGame/Content/Paks/TSCGame-WindowsNoEditor.pak (-11.52 GiB) = this file is the full game, remaining files/binaries are about 100-150 MB

That proves the compressibles games are bad encoded data (right ?), and should always be compressed on NTFS (compress or compact.exe btw) to not loose unnecessarily storage space.

//

I read your comments, there is a lot technical stuff, i was aware of format allocation size but always used default...
... i'd never know how to know how much big/tiny files there will be stored on my disks in advance.

So, the data written, if compressible doesn't count as free space for the SSD, like it's always told to keep 10-15% free !.. that was what i wanted to understand, and it looks the answer has been no.

Thanks to all for all.

 

[nageme]

That proves the compressibles games are bad encoded data (right ?)

That's interesting. Could you post a list of some games and their pre/post NTFS-compressed sizes?
I haven't been following really, but I do recall that unlike the past, nowadays it's common for games to be wasteful, with uncompressed speech and audio, maybe textures, and more.

BTW, for files that do actually include a lot of empty regions in the middle, NTFS supports sparse files, which is distinct from compression.
Not sure there's a GUI for that because I don't think it's an everyday feature. Maybe for special uses, like scientific data, or some database types.

the biggest savings are from executables/binary files & you can't compress most of them on a System drive.

The most compressible would usually be text files (including real text, logs, HTML, XML, SQL), and after that maybe uncompressed images (BMP, PSD, and others), and uncompressed audio (like WAV).
AFAIK there shouldn't be a problem to compress Windows system files, but I haven't tried overriding what Windows does by default.

The difference mainly comes from the default allocation size vs actual amount used by files

Slack is quite minimal. The common size for clusters on NTFS is 4K, but even if it were, let's say, 16K, that's not too bad.

I checked the Windows directory here. The average file size is 140K. With the default 4K cluster size, which means an average slack of 2K per file, the waste is 1.4%.
(In actuality some files are NTFS-compressed, so the on-disk size is less than the logical size.)
And other collections of files are typically larger. For example, my main software directory averages 227K/file. And data files would be much larger.

because of default sector size

Slack is governed by cluster sizes. The change in physical sector size hasn't really affected it.
Actually, clusters are usually smaller now than in the old days, because of the way NTFS works compared to FAT(32).

Those should NOT be compressed in the file system because any update will lead to very bad fragmentation.

I'm not sure about it, assuming compression is enabled on a directory tree before writing the initial files.
Appending is the same, compressed or not.
Overwriting in the middle, I'm guessing normally it wouldn't radically change the compressibility of the block. Though that indeed depends on the nature of the file.

But anyway, fragmentation is less of an issue on SSDs (although apparently it's still beneficial to minimize it).

 

[R-T-B]

There's also the core reason games generally do not use (much) compression: it incurs cpu overhead on read and this does NOT help fps or latency. So no, not everyone should do it.

 

[nageme]

Audio decompression is cheap anyway. And you're not playing/mixing too many tracks at the same time. And if stored in RAM, you want to minimize it.
Textures might be faster to send to the GPU compressed.

It may have changed with SSDs, but on HDDs there's also a benefit to minimizing the size of the data you need to stream-read.

 

[DirtyDingusMcgee]

compression is a neat topic. lots of fancy math be mathing.

 

[nageme]

Though filesystem-style compression is quite limited in innovation or progress, because it must be lossless, all-purpose, and needs to still provide random access.
Also, storage and bandwidth are getting cheaper.

(Still, it would be nice for games not be so wasteful. They don't have to be 96KB like .kkrieger, but at least delete before release the 10GB of unused resources, and maybe make updates include just the delta?)

 

[lexluthermiester]

There's also the core reason games generally do not use (much) compression: it incurs cpu overhead on read and this does NOT help fps or latency. So no, not everyone should do it.

While true, it's almost trivial at this point and many people would rather have compression and smaller install sizes.

 

[Klemc]

That's interesting. Could you post a list of some games and their pre/post NTFS-compressed sizes?
I haven't been following really, but I do recall that unlike the past, nowadays it's common for games to be wasteful, with uncompressed speech and audio, maybe textures, and more.

CompactGUI has a DB:

CompactGUI Results

docs.google.com

Git

Community Compression Results

Transparently compress active games and programs using Windows 10/11 APIs - IridiumIO/CompactGUI

github.com

 

[chrcoluk]

When I last tried NTFS compression, it had a pretty bad flaw, maybe its fixed in 11, dont know. But basically it will write uncompressed first, then compress and rewrite, so you increase your writes.

 

[Klemc]

When I last tried NTFS compression, it had a pretty bad flaw, maybe its fixed in 11, dont know. But basically it will write uncompressed first, then compress and rewrite, so you increase your writes.

Yes, writes, in my case the games i play on S-Deck are up-to-date and won't update, so it's good, in cases files are rewitten, it's not the best idea, also it's compact.exe i use so i compress only folders i want when/if i want.

 

[R0H1T]

The most compressible would usually be text files (including real text, logs, HTML, XML, SQL), and after that maybe uncompressed images (BMP, PSD, and others), and uncompressed audio (like WAV).
AFAIK there shouldn't be a problem to compress Windows system files, but I haven't tried overriding what Windows does by default.

Yes, on average probably, but you rarely get massive log/text files on consumer systems. I'm talking about probably a typical home system drive; the biggest savings I've seen are on binary files in my experience.

You can't compress them while they're in use; even if you somehow manage to boot into another WinPE environment, there could be issues if those files aren't supposed to be compressed. Like pagefile & hibernation files are compressed when you do run compact on the entire C:\ drive, but the system will either reinitialize/recreate the pagefile without compression or create a new one, & hibernation won't work as long as that file is compressed. Just an example with respect to system files, this also applies to anything in the Windows folder.

Just to be clear, I'm talking about the additional flag ~

compact /exe

That will save you (lot) more than regular NTFS compression.

 

[lexluthermiester]

When I last tried NTFS compression, it had a pretty bad flaw, maybe its fixed in 11, dont know. But basically it will write uncompressed first, then compress and rewrite, so you increase your writes.

Um, no. It's not fixed. Don't use NTFS compression.