Setting 4k sector size on NVMe SSDs: does performance actually change?

[Solid State Brain]

NVMe specifications allow the host to send specific low-level commands to the SSD in order to permanently format the drive to 4096 bytes logical sector size (it is possible to go back to a 512 bytes size in the same way). Not all NVMe SSDs have this capability.

Most client-oriented storage operates by default in "512-bytes emulation" mode, where although the logical sector size is 512 byes/sector, internally the firmware uses 4096 bytes/sector. Storage with a 4096 byte size for both logical and physical sectors operates in what is commonly called "4K native" mode or "4Kn". Due to possible software compatibility issues that have still not been completely solved yet (for instance, cloning partitions from a 512B drive to a 4096B drive is not directly possible), these drives tend to be quite rare in the client space and it is mostly enterprise class drives that employ it.

Some background information on this subject on Wikipedia:

Advanced Format - Wikipedia

en.wikipedia.org

Why change this setting? In theory, the 4K native LBA mode would get away with the "translation" the firmware has to do with 512-bytes logical sectors to map them to the underlying 4K "physical" arrangement (if a physical/logical distinction makes sense for SSDs) and may offer somewhat higher performance in this way.

This is possibly true for fast NVMe SSDs and high-performance (non-Windows) file systems in high-I/O environments, but it is unclear whether Windows performance with ordinary NTFS partitions would be improved, and the subject is sort of obscure and somewhat confusing. Some people for instance may think that the logical sector size is the same as the partition's cluster size (which defaults to 4 kB on Windows), but they are unrelated with each other. Furthermore, changing the logical sector size requires to delete everything on the SSD and basically reinstall the OS from scratch, which makes it even more unlikely for users to attempt it and see if differences arise. This is better tested with brand-new, empty drives.

I have a WD SN850 which allows changing this setting, but after doing it and reinstalling the OS I cannot say I have really observed significant differences on Windows 11 (also, Windows 11 currently has performance consistency issues with NVMe SSDs under certain configurations, but I didn't know this at the time).

I did it after following this blogpost where a WD SN850 user on Linux reportedly measured 10% higher performance on EXT4 partitions with basic benchmarks:

Switching your NVME ssd to 4k - Bjonnh.net

I recently got a WD SN850. There is a little trick to do when you receive it to switch it to 4k LBA and thus getting better performance by using native block size.

www.bjonnh.net

Some sources claim that the 4 kB LBA format provides "the best performance and endurance". This is also what appears to be hinted by low-level NVMe utilities listing the supported sector size format by the SSD. This source below however did not provide any benchmark to back this claim:

How to format an NVMe drive

A blog about computer storage tips.

filers.blogspot.com

In this website a user saw very marginal improvements on a Samsung Pm1725a 800GB (in Chinese):

教你把 NVMe SSD 切换到原生4K模式

虽然 SSD 已经普及很久，机械硬盘用4K扇区模拟512的时间更久，久到硬盘工具都能很好的处理4K对齐，没人到处问4K对齐检查了。 不过尴尬的是， 至今为止大部分 SSD 为了保证兼容性，仍在使用512扇区，比如我的这块西…

zhuanlan.zhihu.com

Here a Micron 9300 apparently gets better performance on ZFS filesystem (Linux) with 4K native LBA:

Proxmox VE ZFS Benchmark with NVMe

To optimize performance in hyper-converged deployments with Proxmox VE and ZFS storage, the appropriate hardware setup is essential. This benchmark presents a possible setup and its resulting performance, with the intention of supporting Proxmox users in making better decisions. Download PDF...

forum.proxmox.com

However, a user with a WD SN550 saw no differences in basic benchmarks:
https://www.reddit.com/r/hardware/comments/m18a5c/_/gqcyjt0
There is some information on the Linux utilities used to change this setting on a blogpost of NVM Express organization:

Open Source NVMe® SSD Management Utility – NVMe Command Line Interface (NVMe-CLI) - NVM Express

Jonmichael Hands, VP Storage, Chia Network NVM Express® (NVMe®) technology has enabled a robust set of industry-standard software, drivers, and management tools that have been developed for storage. The tool to manage NVMe SSDs in Linux is called NVMe Command Line Interface (NVMe-CLI). Overview...

nvmexpress.org

Some vendors may provide tools to change this from Windows effortlessly, for example like Sabrent for its Rocket 4 SSDs with a "Sector Size Converter" utility. However, there is no mention about possible performance differences, only issues pertaining to "data cloning scenarios":

SSC | Sabrent

www.sabrent.com

Has anybody else tried and benchmarked any performance difference (possibly also in mixed read/writes, high-I/O scenarios) in a more controlled environment?

 

[cfelicio]

I ended up using Anvil Benchmark, as it checks both IOPS and Throughput, and found no measurable difference in performance, at least using Windows / NTFS / Sabrent Rocket NVME. From reading Seagate's paper, it seems like using 4096 has some advantages on the error correction side of things, as well as potentially avoid the controller having to emulate / convert from 512 to 4096. I will keep using 512 for my existing boxes, but will certainly add 4096 whenever I provision something new.

 

[plastiscɧ]

i have samsung drives; when i -- secure erase -- my NVMe its erased with 512, for sure. i have no other option in BIOS.
after clean installing the OS everything is automatically converted to 4096 cus u start with a raw drive. Win11 gives no choice.

but that information about error correction has value for me! thanks

4096 has some advantages on the error correction side of things

 

[cfelicio]

i have samsung drives; when i -- secure erase -- my NVMe its erased with 512, for sure. i have no other option in BIOS.
after clean installing the OS everything is automatically converted to 4096 cus u start with a raw drive. Win11 gives no choice.

View attachment 232562

but that information about error correction has value for me! thanks

To double check, you can run on the command line fsutil fsinfo sectorinfo C:, and it should return something like this:

If both physical and logical sectors are showing 4096, you are running on 4KN.

 

[eidairaman1]

I just recalled that after Windows XP SP3 launched, anything larger than 4KB sectors didn't work for NTFS. SSDs have made it where setting sector size greater than 4KB is pointless due to firmware management algorythims.

I do recall 8KB sectors were faster but they used up space quicker.

 

[Solid State Brain]

I ended up using Anvil Benchmark, as it checks both IOPS and Throughput, and found no measurable difference in performance, at least using Windows / NTFS / Sabrent Rocket NVME. From reading Seagate's paper, it seems like using 4096 has some advantages on the error correction side of things, as well as potentially avoid the controller having to emulate / convert from 512 to 4096. I will keep using 512 for my existing boxes, but will certainly add 4096 whenever I provision something new.

I didn't find significant performance differences either on Windows with my WD Black SN850, but I haven't been able yet to to do better testing on Linux with other filesystems. At some point I also had to revert 512b sectors due to having to clone partitions to/from a different SATA SSD that does not support changing the physical sector size.

i have samsung drives; when i -- secure erase -- my NVMe its erased with 512, for sure. i have no other option in BIOS.
after clean installing the OS everything is automatically converted to 4096 cus u start with a raw drive. Win11 gives no choice.

I just recalled that after Windows XP SP3 launched, anything larger than 4KB sectors didn't work for NTFS. SSDs have made it where setting sector size greater than 4KB is pointless due to firmware management algorythims.

I do recall 8KB sectors were faster but they used up space quicker.

Keep in mind that the post is about the "physical" sector size, not the cluster size set when formatting the SSD with Windows. Windows uses a 4096-bytes cluster size by default with NTFS size regardless if the drive has 512-bytes or 4096-bytes physical sectors.

 

[AsRock]

I didn't find significant performance differences either on Windows with my WD Black SN850, but I haven't been able yet to to do better testing on Linux with other filesystems. At some point I also had to revert 512b sectors due to having to clone partitions to/from a different SATA SSD that does not support changing the physical sector size.




Keep in mind that the post is about the "physical" sector size, not the cluster size set when formatting the SSD with Windows. Windows uses a 4096-bytes cluster size by default with NTFS size regardless if the drive has 512-bytes or 4096-bytes physical sectors.

You will still lose space, i did when i changed mine 4k and put the files back on the drive and ended up losing space because of it.

 

[R-T-B]

You shouldn't be losing any disk space as the ntfs default cluster size is already 4kb.

 

[AsRock]

You shouldn't be losing any disk space as the ntfs default cluster size is already 4kb.

I'd re-check but it takes some time to do.

Just checked and do believe you are 100% right.

 

[plastiscɧ]

i was unsure as well. but after research it turned out that my NVMe stays like it is. @eidairaman1 was right regarding the firmware

Samsung 980 pro

greater than 4KB is pointless due to firmware management

 

[chrcoluk]

I only use 4k now on my OS drive for compatibility reasons, I am in the process of migrating most of my other partitions/drives to 64k clusters.

The loss of space is small compared to the hit on performance, especially for spindles which benefit a lot from 64k.

My gaming partition on my NVME SSD might get changed to 8k or 16k. From 4k, to reduce io/sec loads. Samsung SSD's are known to perform very well with 8k, and a new micron SSD has launched with a native 16k storage block.

512 bytes in my opinion shouldnt even be considered at this point.

 

[Ferrum Master]

I do not think it matters for SSD's really, as those do not write in sequential manner. They write in rather random behavior based on free space and wear level mitigation techniques.

Yes it applied to spindles, hands down 4K is a default... 512 were for floppy magnetics and super early five incher HDD drives?

The thing it matters more are in removable media, where you need to decrease the overhead thus making large sectors makes sense due to performance reasons only.

 

[gahabana]

on Kingston KC3000 temps are MUCH lower with 4KB vs 512B sector/LBA size​

hi all,
this thread motivated me to experiment. got Kingston KC3000 which supports both 512 and 4k LBA (sector?) size.
Did not do enough experiments - performance with 2 apps that scan music library and read tags (on average 10-120kB of each file must be read though files are on average 20-40MB each) are simmilar.
HOWEVER - when loading SSD (lot of these files get copied 1.5TB) - with same performance i get MUCH lower temperatures on NVME drives. in 50C+ range rather then even throttling (72C+) ( in spite of passive heatsinks) ... my guess translation of amount of operations causes controler to use lot of power.
Anyone else care to check ?
Will check with Samsung 980 Pro as well and let you all know !

 

[Solid State Brain]

Unfortunately it is not possible to change the LBA size on the Samsung 980 Pro, which I currently have (I moved the previous WD Black 850 onto a different PC). The usual NVMe utilities show 512-bytes as the only supported format.

Thanks for your report on the KC3000, though. Interesting that you get lower temperatures with 4KB sectors; I didn't pay attention to them last time with the WD850.

#> sudo smartctl -x /dev/nvme0n1
smartctl 7.2 2021-09-14 r5237 [x86_64-linux-5.17.1-1-default] (SUSE RPM)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number: Samsung SSD 980 PRO 1TB
Serial Number: xxxxxxxxxxxxxxxxxxx
Firmware Version: 5B2QGXA7
PCI Vendor/Subsystem ID: 0x144d
IEEE OUI Identifier: 0x002538
Total NVM Capacity: 1,000,204,886,016 [1.00 TB]

[...]

Namespace 1 Formatted LBA Size: 512

[...]

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0

EDIT: Or also:

#> sudo nvme id-ns -H /dev/nvme0n1

[...]

LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)

(no other supported format shown)

 

[Mussels]

All mine show the same, but i'm still interested in how this turns out

As long as they're aligned, there should be no performance difference.

When i've seen posts about people who fixed performance issues, that's what's always come to mind (older/uncommon OS, not 4K aligning drives that show as 512b)

 

[Solid State Brain]

The above screen shows that your drives are using a 512-bytes logical block size, while their underlying physical structure uses 4096-bytes (4 kB) blocks. This is common for modern storage and it's called "512e mode" or "512 emulation". It does not tell whether the drives also support setting a 4096-bytes logical block size, i.e. "4Kn mode" or "4k native".

In theory, if logical block size equals physical block size, there might potentially be performance benefits, since the firmware will not have to "translate" between the two formats anymore. However, generally they are unlikely to be very large and I certainly didn't notice significant differences with the WD SN850 I previously had. On the other hand, I had with it some inconveniences with partition cloning tools and so on.

This is not related with partition alignment, although a 4096-bytes logical block size will prevent the problem entirely (since partition alignment will then be possible only in multiples of 4 kB blocks).

 

[Deleted member 24505]

Here's mine. Both my SN850's are on 512.

 

[mb194dc]

It's pretty obscure, what we need is review of two drives, one with 4k and one with 512b under otherwise controlled conditions. With a variety of disk work loads to see if it makes any difference.

 

[gahabana]

Unfortunately it is not possible to change the LBA size on the Samsung 980 Pro, which I currently have (I moved the previous WD Black 850 onto a different PC). The usual NVMe utilities show 512-bytes as the only supported format.

Thanks for your report on the KC3000, though. Interesting that you get lower temperatures with 4KB sectors; I didn't pay attention to them last time with the WD850.

#> sudo smartctl -x /dev/nvme0n1
smartctl 7.2 2021-09-14 r5237 [x86_64-linux-5.17.1-1-default] (SUSE RPM)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number: Samsung SSD 980 PRO 1TB
Serial Number: xxxxxxxxxxxxxxxxxxx
Firmware Version: 5B2QGXA7
PCI Vendor/Subsystem ID: 0x144d
IEEE OUI Identifier: 0x002538
Total NVM Capacity: 1,000,204,886,016 [1.00 TB]

[...]

Namespace 1 Formatted LBA Size: 512

[...]

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 0

EDIT: Or also:

#> sudo nvme id-ns -H /dev/nvme0n1

[...]

LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0 Best (in use)

(no other supported format shown)

you're so right. just installed - one with latest 5xxx firmware and one with 3xxx... - 512b is the only option (unlike kingston kc3000)

 

[AsRock]

It's pretty obscure, what we need is review of two drives, one with 4k and one with 512b under otherwise controlled conditions. With a variety of disk work loads to see if it makes any difference.

I noticed no real difference

 

[Mussels]

The above screen shows that your drives are using a 512-bytes logical block size, while their underlying physical structure uses 4096-bytes (4 kB) blocks. This is common for modern storage and it's called "512e mode" or "512 emulation". It does not tell whether the drives also support setting a 4096-bytes logical block size, i.e. "4Kn mode" or "4k native".

In theory, if logical block size equals physical block size, there might potentially be performance benefits, since the firmware will not have to "translate" between the two formats anymore. However, generally they are unlikely to be very large and I certainly didn't notice significant differences with the WD SN850 I previously had. On the other hand, I had with it some inconveniences with partition cloning tools and so on.

This is not related with partition alignment, although a 4096-bytes logical block size will prevent the problem entirely (since partition alignment will then be possible only in multiples of 4 kB blocks).

I know they're 512e, my point is that a modern OS will know that and work with 4K block size, aligned with the drives hardware
older OS's, or just weird esoteric OS's that dont try and compensate can screw that up - and therefore show improvements when forced to 4K


We need someone with a proper test setup like @W1zzard, with drive that can be toggled between the two operating states.


Anything less is just theoretical, because so far the only times this has improved performance for people has been light on details. Look at just a few posts up claiming it can lower temps by @gahabana
We dont know the OS, the file system, the cluster size, or any other important information there (No offense to you @gahabana - you did provide what you thought was relevant)

 

[W1zzard]

with drive that can be toggled between the two operating states

Any suggestions?

 

[Deleted member 24505]

Could you do the tests with a SN850?

 

[Mussels]

Any suggestions?

Could you do the tests with a SN850?

This article has a guide for the SN850
Switching your NVME ssd to 4k - Bjonnh.net

How to switch your NVME SSD to 4KN Advanced Format - Carlos Felicio
(This is a longer guide, that links to that first guide)

Same person did basic testing themselves, found almost no difference
512E vs 4KN NVME Performance - Carlos Felicio


Edit: cant adjust the setting over USB. sigh. This will take more effort than expected.

Final edit: My intel drive is 512 only, my SN730's are 4k out of the box and my other two drives are kinda critical, so i cant format those for testing easily.

 

[Athlonite]

The Adata SX8200 Pro 1TB NVMe SSD use the 512b 4096 block sizing no 4Kn that I could find out about