Hello again fellow HP workstation enthusiasts.
I thought I should post a note about a disconcerting issue I ran into when doing the last upgrade to my z440/z640 "frankenstation".
Background:
For this upgrade, I decided to expand the two raid-5 arrays I had by adding an additional drive to each of them. Expanding my 12Tb array with 3 6tb drives into a 18tb array with an additional 6tb drive, and expanding my 16tb array with 3 8tb drives into a 24tb array with an additional 8tb drive.
As usual (for me), I sourced retired enterprice grade datacenter SAS drives for this that matched the existing drives in the arrays. These are much more robust than any regular desktop SATA drives are, with around 2 million hours mean-time-between-failure (MTBF) specs. So even though they have been spinning for several years in some server farm, these used drives have better longevity than the usual desktop drives or even red "raid" drives do. There are equivalent retired enterprise grade SATA drives out there, but they often go for more money than the SAS drives do, and the SAS drives usually have a bit more throughput than the SATA versions do. Yes, the SAS drives require either a PCI-E hardware RAID controller, or a non-raid SAS HBA (or RAID controller running in HBA mode), but recycled datacenter PCI-e controllers can be found for very little money.
I'll make another post discussing some considerations regarding these controllers....
So here's the issue I ran into:
First off, let me be clear that this is not a hardware issue -- the Zx40 workstations have no issue with large drives. It is a Windows OS issue, specifically, a Windows NTFS volume cluster size issue.
So in my situation, I successfully expanded the raid 5 arrays in the raid controller configuration utility, but when I attempted to expand the Windows NTFS volume to the logical drive's new larger capacity, I got this somewhat cryptic and worrisome error message from the Windows DISKMGMT utility: "The volume cannot be extended because the number of clusters will exceed the maximum number of clusters supported by the file system."
What the heck?!?!
At first I thought the error was saying there was an NTFS limit (which IS, after all, the "file system"), which didn't make sense because I recalled that Microsoft was very verbal about NTFS volumes being able to be defined in Petabyte (1024 Terrabyte) sizes. Some research on the web finally lead me to the real culprit -- the fact that NTFS volumes with a cluster size of 4KB (the Windows default cluster size) can't be expanded beyond 16TB.
When a NTFS volume is formatted that is less than 16TB, the Windows default cluster size of 4KB is used (unless you specify something different, which most people don't do).
This isn't discussed much in many forums because most folks haven't (yet) tried to upgrade/expand existing drives beyond this limit. When you insert a new single drive larger than 16TB, and format a volume on it, Windows will automatically assign a default 8KB cluster size when formatting the volume (again, unless you specify otherwise). So when building a NEW system with or ADDING a new drive and volume >16TB to an existing system, the issue doesn't surface.
The issue only surfaces AFTER you upgrade an existing drive, whether it is a logical drive (like a RAID array) or a physical drive, beyond this 16TB limit, either by expanding a the logical drive of a RAID array or trying to upgrade a smaller single physical drive with a larger single drive (>16TB), which is typically done using a disk cloning utility, often a branded version of Acronis supplied by drive manufacturers, or some other utility like Macrium.
In both of these cases, the drive upgrade will be successful, and the volume will show up in windows as being the same size as before the drive upgrade. This is normal and expected, and requires the additional step to "expand" the size of the volume to incorporate the added storage..
When looked at in the Windows Disk Management Utility (Diskmgmt), it will show the drive as being the new larger size, but the volume will be the original size, with unformatted space following the existing volume. The usual choice for most folks is to use Windows' Diskmgmt to expand the volume, telling it to add that unformatted space on the drive to the existing volume.
That's when you get the above message if the expansion will push the volume past the 4KB cluster size 16TB limit..
Unfortunately, the various disk backup/cloning software utilities (such as Macrium or Acronis) don't deal with this limit either -- they just restore the cloned volumes to the same cluster size the original volume used. I know most of the various disk partition utilities can't do this either. Although some of the paid ones may look like they might, I doubt they can safely do this. I know that the open source Gparted utility can't do this either -- I checked with the developer.
(A word of caution regarding these partition utilities: They make "in-place" modifications to the volumes, and so if they fail, the volume will likely be corrupted. If you haven't done a backup prior to running the utility, you could lose all of your data.)
So how to fix this?
The easiest solution would be to live with this 16TB cluster size limit, and define another volume using the unfromatted space, and live with the data being split into multiple separate volumes -- but that makes things unecessarily complex in the long term.
If you really want to upgrade your existing volume to be >16TB, it seems the only way is to do it manually using the following steps:
1A. If you are working with a logical drive like a RAID array, do a full backup of the existing drive contents using a utility that is known to be trustworthy (not all are), and verify the backup worked. Make sure you can "mount" the backup and access/copy individual files from it. I use the free version of Macrium for this. I have had some issues with Acronis failing to access some backups in the past, so I don't trust that utility any more. An alternative is to not do an array expansion, but instead to define an entirely new array with different drives, keeping the original as the source "backup" (this is a much more expensive option because you have to have and use an entirely different set ot drives).
1B. If you are upgrading a single physical drive, then you can use the original smaller drive as the source "backup", and no separate backup is needed.
2A. For a logical drive, after doing the successful backup and you have verified it and know you can "mount" it in Windows, delete the existing array, and redefine it with the additional drive(s).
2B. For the single drive upgrade, install the new larger than 16TB drive in the system
3. Format a new NTFS volume on the new/expanded drive in Windows. Windows will automatically choose the 8KB default if it sees the volume is going to be >16TB. You can specify it manually as 8KB too.
4. If your original drive was the boot drive with Windows installed on it, you will likely have to reinstall windows and your programs, unless you pay for and use a specialized system copying utility to move the operating system, programs, and windows system registry and user registry files over to the new drive. Such utilities exist, but they are not free and are expensive, and they often don't do everything correctly or the way you want them to. The safest way is to do a "clean" installation of windows, reinstall your programs, and then proceed to step 5 after that. If your drive was NOT the boot drive with windows on it, then you skip this step and go straight to step 5..
5. If using a backup, then mount it in windows so you can access the files in the backup. If you have the original smaller drive, keep it installed on the system, but don't boot from it.
6. Manually copy all directories and files from the mounted backup or original "source" drive. Each file will be copied to the new drive using the larger cluster size. The creation and file modification dates will be the same as on the original drive or backup, but all directories will have the date and time stamp of the time of their creation on the NEW drive. There may be some 3rd party file copying utilities that will preserve the directory dates, but I don't know what they are (if any).
Suggestion: If your drive has a lot of files and directories at the root level, I recommend printing a list of the items in the root directory, and breaking up the copy process into multiple parts -- copying one subset of them at a time, because it will be easier to monitor that the copying is successfully working, and you'll know where you left off or where to restart if something fails (like the system hangs or a power failure occurs). If a failure happens in the middle of copying a directory, then just recopy the whole directory and tell it to "replace" any files that it finds are duplicates. This is safer than trying to figure out where to restart in the middle of a directory.
This copying process will take a long time -- probably twice as long as the regular disk cloning operation would have taken. But I believe it is the safest way to do this. In my situation, it took several days to get it all done.
Note that it is likely that the copy process will identify that there are some duplicate files (usually named "desktop.ini"). You can safely chose to skip these files, or to replace them with the ones from the backup. These desktop.ini files are where Windows stores the viewing preferences for each directory, so I recommend copying them from the backup to retain any custom views you might have made. Note also that it is not unusual for there to be a significant delay (sometimes several minutes) after the last file is copied before the copy windows disappears. Don't worry, just wait until the copy window closes before continuing.
If you are expanding your boot drive (assuming you have done a clean installation of windows, and resinstalled your programs on the new or expanded drive), here are a few things to keep in mind:
1. Many programs store data in the special "appdata" directories -- for instance, your web browser bookmarks, history, extensions and settings are usually stored there. Depending on whether you have the option in the directory "view" settings in windows file explorer set to "Show hidden files and folders" or set to "Do not show hidden files or folders", the "appdata" directories won't be seen or copied.
I don't recommend wholesale copying of them. However, for particular situations (like your web browser settings and bookmarks), you might want to copy a certain particular program's directories over from the original drive.
The pathname for these "appdata" files are usually as follows:
C:\Users\<yourusername>\AppData\Roaming\<programname>
or
C:\Users\<yourusername>\AppData\Roaming\<companyname>\<programname>
Sometimes a program's settings are stored in the AppData\Local directory instead of the Appdata\Roaming directory -- the Google Chrome browser uses the local variant.
For example, for the Firefox browser, it's data is stored in the folder
C:\Users\<yourusername>\AppData\Roaming\Mozilla\Firefox
So, after copying all of your files, if you want to recover your Firefox settings, bookmarks and history, you would copy that directory from the backup/source over to the new volume appdata directory after you complete the rest of the copying. Make sure Firefox is NOT running when you do this copy. Then the next time you go into Firefox, all this stuff will be back to what you remember.
For the Chrome web browser, its settings, bookmarks and extensions can be copied from the following appdata directory:
C:\Users\<yourusername>\AppData\Local\Google\Chrome
Note, if you have the directory view setting "Do not show hidden files or folders" turned on, you won't see the appdata directory. You can either switch that setting to "Show hidden files and folders" so you can see it and then browse it in Windows File Explorer, or you can use a "shortcut" to go directly to the AppData\Roaming directory by typing the following into the file pathname field at the top of any Windows File Explorer window: "%appdata%" (without the quotes) and press <enter>. The shortcut for the AppData\Local directoriy is "%localappdata%".
Suggestion: You might want to make and keep a copy of the old appdata directories from the backup saved somewhere in case you later discover that there's another program whose settings you want to copy over to the new volume.
2. Please do NOT copy the following directories and files from the backup to the new volume:
/Windows and it's subordinate directories
/Program files and its subordinate directories
/Program files (x86) and its subordinate directories
/users/<yourusername>/NTUSER.DAT <===== This file is a part of the Windows Registry
/users/<yourusername>/appdata and its subordinate directories, with the exceptions noted in number 1 above.
3. Make sure you have not unchecked the folder view option "Hide protected operating system files" so you won't accidentally see and copy those files.
I hope this helps others deal with crossing this little known NTFS 4KB cluster 16TB limit.
Philip
