SSD, compact.exe saves filesize %, what about over-provision ?

[Klemc]

If compact.exe can compress a file(s) to 15/45% less his original size (when simply copied), my question is what about this gained space, did it count as written space to the hardware, and SSD things it's full ?..

... so if an SSD is 99% full, using compact.exe if lucky can get back 40% of the total space (so it's only 60% full after), then the question is about the empty space after compression why can't it be found/used by the SSD (overprovision...) ?

//

What's in all those empty data, i mean, a 50 gb file is given... then compact.exe compresses it to 35% less written space being now 35 gb, why the file is 50 gb in the first place, what's an empty data doing there ?.. why does empty thing

takes place, why the non compressed file is huge, is it emptyness ?



I'd like to understand a bit, thanks.

 

[DirtyDingusMcgee]

Data Compression

LZW

Huffman Compression

how file compression works

This is an over-simplification.

60gb hdd (or ssd) with a 50gb data file, so 10gb of free space. Data file compressed to 40gb, so now you have 20gb free space.

You would be able to use that 20gb of free space, and it would be 20gb of free space, since the 50gb file is only 40gb now.

If you added another file to the drive that was 15 gb (so only 5gb free space now), then you would be unable to decompress the large file until sufficient space was available.

... so if an SSD is 99% full, using compact.exe if lucky can get back 40% of the total space (so it's only 60% full after), then the question is about the empty space after compression why can't it be found/used by the SSD (overprovision...) ?

It wasn't empty space until you compressed it. Overprovisioning is a totally different thing.

Overprovisioning

Perhaps you are asking "why isn't it compressed as much as it can be from the start?", and that can be done, but there is some serious system overhead involved in real-time compresssion/decompression.

Real time compression - this article is specifically about IBM real time compression, but has it great info about the subject in general and storage footprint reduction.

 

[nageme]

As far as I know, only SandForce's SSD controllers used compression, years ago:

SandForce Announces Next-Gen SSDs, SF-2000 Capable of 500MB/s and 60K IOPS

www.anandtech.com

Files that are compressible aren't empty, usually. They're just compressible.
Compression is used in a lot of data formats, and in some networking protocols, but it's not always practical because of various factors.

For example, no compression means using less processing power, or less electric power, or having the ability to read or write data randomly and in small chunks (possibly 1-byte precision), or less latency, or better recoverability in case of corruption. So it's a compromise, used when saving space or bandwidth is more important.

Some formats that use compression:
JPEG, PNG, GIF, WebP, MPEG* video and audio, Vorbis, Opus, FLAC, RAR, 7Z, ZIP and many formats that use ZIP like DOCX, APK, JAR...

 

[R0H1T]

so if an SSD is 99% full, using compact.exe if lucky can get back 40% of the total space

You won't save that much even if you're lucky, the biggest savings are from executables/binary files & you can't compress most of them on a System drive. The compression for regular media files won't work as you have to transcode or convert them, regular NTFS compression will only save you a very small amount.

takes place, why the non compressed file is huge, is it emptyness ?

The difference mainly comes from the default allocation size vs actual amount used by files, also a lot of space used to be wasted because of default sector size back in the day for HDD's ~

Advanced Format - Wikipedia

en.wikipedia.org

 

[Wirko]

Are you just experimenting, or do you have a real use case for NTFS compression?

Most large files are compressed internally, as others have explained already. There are exceptions such as some video/audio/photo raw files and text log files - in general, anything that is meant to be changed, updated or appended. Those should NOT be compressed in the file system because any update will lead to very bad fragmentation. The main reason for fragmentation is that the compressed file size is not constant but depends on the content.

So if you identify any files that are large, never change, and are not internally compressed, those would be good candidates for NTFS compression. But I have no idea what those could be.

Apart from compact.exe, you can also turn the compression on or off in Windows GUI. It does the same thing. Right click on the file or folder, Properties, General tab, Advanced button, Compress contents to save disk space checkbox.

 

[Klemc]

Video games compress well (i use compression on storage that have video games on it, nothing else), the max i seen was 60%, in the end on the 500 GB Steam-Deck's SSD i won 30% more free space compared too raw data.

As an exemple (not on my SD btw) "The Sinking City" needed an update to see the game size near halved (year after elease 2019-6 > 2021-1).

Depot 750132
TSCGame/Content/Paks/TSCGame-WindowsNoEditor.pak (-11.52 GiB) = this file is the full game, remaining files/binaries are about 100-150 MB

That proves the compressibles games are bad encoded data (right ?), and should always be compressed on NTFS (compress or compact.exe btw) to not loose unnecessarily storage space.

//

I read your comments, there is a lot technical stuff, i was aware of format allocation size but always used default...
... i'd never know how to know how much big/tiny files there will be stored on my disks in advance.

So, the data written, if compressible doesn't count as free space for the SSD, like it's always told to keep 10-15% free !.. that was what i wanted to understand, and it looks the answer has been no.

Thanks to all for all.