+80TB NAS

[remi]

cache storage, adding a second redundancy drive, sata cards for more then 10 sata ports etc are all things that will be a future upgrade. I want to start with the bare bones.

But please bare in mind, this is not for commercial applications, or networking, it's just for my personal video library. It really shouldn't be that complicated.

 

[Fx]

I hear ya, my first foray into storage also began with a tower because that was all I knew and I wanted it to look good and be quiet. I used a Fractal Design Define R4. It worked great until I quickly ran out room for expansion. This is when I moved onto a Norco 4224, and then eventually the Supermicro that I have now:

https://www.amazon.com/gp/product/B00C8H17LY/?tag=tec06d-20

You eventually get to a point where you have to design strictly for storage requirements and appearance comes second.

 

[newtekie1]

newtekie1 are you referring to Parity resiliency ? i did not know that performance drops if the drives are different size, but i guess i could use all of the hdds in the same size.
Honestly i still dont see a drawback of this software RAID (storage spaces). Although i do agree that i need to do more research.
Any advice is very appreciated.

Yes, I'm talking about parity resiliency.

Storage Spaces is poorly implemented. It is a pain in the ass to set up properly, and a lot of the settings aren't clear.

Supermicro that I have now

I have the 12 bay version of that! I agree, once you get into storage, you realize cases should be about function, not form.

 

[remi]

I hear ya, my first foray into storage also began with a tower because that was all I knew and I wanted it to look good and be quiet. I used a Fractal Design Define R4. It worked great until I quickly ran out room for expansion. This is when I moved onto a Norco 4224, and then eventually the Supermicro that I have now:

https://www.amazon.com/gp/product/B00C8H17LY/?tag=tec06d-20

You eventually get to a point where you have to design strictly for storage requirements and appearance comes second.

I hope one day i'll get to where you are, 150TB ! wow ! what do you store on that ?

It is a pain in the ass to set up properly.

from what i've seen, setting it up is super easy, but i dont know what properly setting it up entails.

 

[thebluebumblebee]

@remi , you want something as simple as an external drive (your comment about connecting via USB), but the capacity that you need pushes you into another strata. If you want to stay with the external storage, and not NAS (if you build a PC like you've talked about, you're building a NAS), then look into something like SANS DIGITAL TowerRAID TR8UM6G JBOD 8 3.5" Drive Bays 8 Bay SATA to eSATA and USB 3.0 JBOD Enclosure . Moving upscale a little, and one that I would recommend, the HighPoint RocketStor 6418TS – 8-Bay Q-SATA Turbo RAID Tower Enclosure would increase speed and I think that a RAID controller is built in. (both of those say that they have a 64 TB capacity, but I think that is just because the description was written before the 10 TB and now 12 TB drives were out) The problem though, is your stated 80 TB capacity. If you use RAID 5 (really, you should, trust me) with 10 TB HDD's, you need to add drives in groups of three, therefore you could only 6 bays and you would be limited to 40 TB. (I saw that Stardom said: "and under the RAID 5 mode, a total of 70TB in volume for storage with a single drive used for redundancy,) (maybe someone can enlighten me on that)

Don't be afraid of RAID. Really, it's not that big of a deal.

 

[newtekie1]

from what i've seen, setting it up is super easy, but i dont know what properly setting it up entails.

They do some really stupid stuff that can put your data as risk.

When you set it up, you pick the storage pool size. Except, you can pick a size bigger than your actual storage allows. So, or example, if you only have 3 10TB drives and set up storage spaces, you can select a pool size of 80TB right off that bat. And it will add the 3 drives to the pool, and a 80TB drive will show up in windows! However, with resiliency you only actually have 20TB of space available. Here comes the problem. You start filling the new Storage Spaces drive with data. You'd think when you hit 20TB it would not allow you to add more data. Wrong. It actually lets you keep adding data, but you pool no longer is resilient. It does warn you that resilience is compromised, but not obviously. It doesn't pop a message up on the screen, it doesn't put something in the system tray warning you. Nope, none of that, it put the warning in the Storage Spaces control panel, which you'll never see unless you go into Storage Spaces...

I used Storage Spaces for a year, it isn't worth using unless you have no other option.

Yes, it has some advantages, being free obviously being one. The fact that if the computer dies, you can move the drives to another Windows computer, and the storage pool should be recognized. I say "should" because there have been people that have tried this and it didn't work.

However, an actual RAID controller has it advantages as well. Speaking for the Highpoint cards here, because that is what I've used for my personal storage setups for years. The first is you can move the array to any computer, running any operating system, and the array will be recognized. If the RAID controller fails, you DO NOT loose the array. The array information is stored on the drives. You can connect the drives to any Highpoint RAID controller and the array will be recognized and usable. Assuming the RAID controller supports the RAID mode the array is using(like you can take a RAID6 array and connect it to a card that doesn't support RAID6). But really RAID6 is the only mode that not every Highpoint card supports. If you use RAID5 or 1 or 0, every RAID card they make will work.

 

[remi]

They do some really stupid stuff that can put your data as risk.

When you set it up, you pick the storage pool size. Except, you can pick a size bigger than your actual storage allows. So, or example, if you only have 3 10TB drives and set up storage spaces, you can select a pool size of 80TB right off that bat. And it will add the 3 drives to the pool, and a 80TB drive will show up in windows! However, with resiliency you only actually have 20TB of space available. Here comes the problem. You start filling the new Storage Spaces drive with data. You'd think when you hit 20TB it would not allow you to add more data. Wrong. It actually lets you keep adding data, but you pool no longer is resilient. It does warn you that resilience is compromised, but not obviously. It doesn't pop a message up on the screen, it doesn't put something in the system tray warning you. Nope, none of that, it put the warning in the Storage Spaces control panel, which you'll never see unless you go into Storage Spaces...

I used Storage Spaces for a year, it isn't worth using unless you have no other option.

That sounds like a huge bug, which i cant believe they haven't fixed by now. So you recommend making the pool size only as big as the drives added ? And when you want to add another drive, then increase the pool size ?
Did you use it in 2016 ?

@remiif you build a PC like you've talked about, you're building a NAS

that's the main plan atm

@remi
Don't be afraid of RAID. Really, it's not that big of a deal.

I'm not afraid, but i really dont like the disadvantages of a hardware RAID, like increasing the size of the array and adding new drives. Which like i said, will be a monthly thing.

 

[newtekie1]

That sounds like a huge bug, which i cant believe they haven't fixed by now.

It isn't a bug, it is actually how they have designed the system to work. Resiliency was a second through when they designed Storage Spaces. And because of that, it is extremely poorly implemented.

So you recommend making the pool size only as big as the drives added ?

That is the problem, if you add 3 drives, and make the pool 30TB, you can still get to the point where you don't have resiliency. When you create the pool you have to account for the size available after resiliency, and they don't even make that size very clear.

And when you want to add another drive, then increase the pool size ?

Yes, and even when changing the pool size, they don't make the size after resiliency that clear.

Did you use it in 2016 ?

Yes, I've been using it in one form or another(Storage Spaces originally started as Drive Extender on Windows Home Server back in 2007). I used Storage Spaces on Windows 10 up until about 2 months ago when I finally ditched the last storage pool for a real RAID setup.

I'm not afraid, but i really dont like the disadvantages of a hardware RAID, like increasing the size of the array and adding new drives. Which like i said, will be a monthly thing.

Like I said before, it isn't that hard. You don't even have to turn the machine off if you have hot-swap bays. And thanks to OCE, you can still access and work with the data while the new drive is being added to the array.

 

[remi]

ok, so for hardware RAID 5, if i make a pool of 30tb, and in 3 months it's full, can i increase the size of the pool and add a new 10tb drive ?
or do i need to save all the ~20TB data ofsite, delete the pool, and then make a new pool of 40tb, and recopy all the ~20tb of data on the new pool ?

 

[Fx]

Remi, this isn't cut n' dry, but it gives you a lot of room for flexibility

Case, your choice

PSU $110
-EVGA SuperNOVA 850 G3

Mobo, your choice. I recommend Socket 1151
-Make sure it has plenty of PCIe connectivity and good ratings for reliability
-Ensure it has an Intel NIC

HBA (HDD controller) $70
- LSI SAS9211-8i (flashed to IT mode). This will present the drives at a low level to the operating system

Memory, your choice

CPU, your choice. I recommend a quad core with hyper-threading (HT)
-Ensure it has high frequency around 3.7-4.0GHz

Stablebit DrivePool Software $30
You can run this on Windows 10, Windows 7, Windows Server 2012 R2, Windows Server 2016

Add the drives. Create two pools. Use software to sync the data from one pool to the other. I use FreeFileSync and create a script which runs via Task Scheduler automatically each night. This is easy to create and documented by the software.

Therefore, no complicated raid information is stored and no parity calculation is performed. If a drive goes bad, you simply remove the drive, replace it and restore the data from the other pool. Data on all other drives remains unaffected.

I would like to add that I would highly recommend you choosing Xeons and ECC memory. This way you can also repurpose the CPU, mobo and memory into a higher capacity case. Again, it isn't necessary, but recommended.

 

[remi]

Remi, this isn't cut n' dry, but it gives you a lot of room for flexibility


Add the drives. Create two pools. Use software to sync the data from one pool to the other. Therefore no complicated raid information stored and no parity calculation is performed. If a drive goes bad, you simply remove the drive, replace it and restore the data from the other pool. Data on all other drives remains unaffected.

so 50% redundancy ? like a raid 1 ? i might as well use raid 0 and use the other 50% of drives as backup. 10-15% redundancy is something i'm ok with, but 50% ?

 

[Fx]

It is 100% redundancy, but not using RAID. It is simply another copy contained on the same system.

You don't have to have your own onsite redundancy, you could pay Backblaze $5/month for unlimited data. It is $4/month if you buy 2 years in advance.

You have to choose how simple, how cheap and how expensive you want it to be. If you want partial redundancy, the software allows you to create duplicate folders for critical stuff.

If you don't want additional risk, more complicated maintenance and working knowledge to manage, expand and recover with RAID then you will go with other alternatives. Someone mentioned unRAID which kind of performs this same kind of setup, but uses its own OS. What I am showing you is a way to do it with Windows which is what you are comfortable with and know. It also doesn't require the cost of having a Server 2012/2016 license.

 

[remi]

sorry, yeah i meant 100% redundancy. Sounds absurd to me, especially since i cant afford it. Otherwise, of course it would be the best idea.

Cloud storage is even more absurd, 5$/TB/month , that means for 70TB, in 10 years i would pay 42,000$, and after that i'm left with absolutely nothing.

 

[newtekie1]

ok, so for hardware RAID 5, if i make a pool of 30tb, and in 3 months it's full, can i increase the size of the pool and add a new 10tb drive ?

Yep. I've done it multiple times with my Highpoint cards. You can do it right inside Windows using the Highpoint RAID Management utility and OCE allows you to continue to work normally while the drive is added to the array.

 

[remi]

Yep. I've done it multiple times with my Highpoint cards. You can do it right inside Windows using the Highpoint RAID Management utility and OCE allows you to continue to work normally while the drive is added to the array.

so you're absolutely sure i dont have to save all the ~20TB data ofsite, delete the pool, and then make a new pool of 40tb, and recopy all the ~20tb of data on the new pool ?

cause more then one people said that this is the way it's done in RAID

 

[Fx]

sorry, yeah i meant 100% redundancy. Sounds absurd to me, especially since i cant afford it. Otherwise, of course it would be the best idea.

Cloud storage is even more absurd, 5$/TB/month , that means for 70TB, in 10 years i would pay 42,000$, and after that i'm left with absolutely nothing.

You misread. $5/month for UNLIMITED data for a single computer or $96 for 2 years.

 

[thebluebumblebee]

You can design, build, configure and maintain your own system:

Or, use something like:

BTW, He started with one of those SansDigital external storage units I suggested above:

 

[newtekie1]

so you're absolutely sure i dont have to save all the ~20TB data ofsite, delete the pool, and then make a new pool of 40tb, and recopy all the ~20tb of data on the new pool ?

cause more then one people said that this is the way it's done in RAID

Yes, I'm 100% sure. Look up OCE(Online Capacity Expansion). I don't know who told you that, but they don't know what they are talking about, so stop listening to anything they say.

 

[Iciclebear]

Hey Remi, here's my 2 cents,

I'm happily using storage spaces at home in a two-way mirroring using a terramaster DAS but that's not that I want to focus on. When it comes to buying that much storage you shouldn't be focusing on the cost of the enclosure but the total cost of ownership because the disks are the primary investment. If you are attempting to build a nas for the total cost of the drives + the hardware you are only going to save 10 - 15% of the total purchase price at best.

All Prices are from amazon.

Ironwolf 8tb drive $260, Ironwolf 10tb drive $360

Let's take 3 high count storage bay NAS devices from Synology, Asustor, and Qnap
Synology DS 2415+ $1400 - 12 Bays 2.4ghz atom quad
Qnap TS-1635 $1276 - 16 bay (4 for ssd caching) 1.7ghz arm processor
Asustor AS6210T $1100 - 10 bays 2.4ghz atom quad core

For the sake of arguement we are gonna set up raid 6 (unless you have a good fast backup I wouldn't raid 5 drives that large, the rebuilds will be terrifying) so 2 drives are "Wasted" in parity.

with 12 bays you can use 8tb drives instead of 10 tb drives at the expense of being able to upgrade further without an extender.

12 8 tb drives = $3120
10 10tb drives = $3600
12 10 tb drives = $4320

80TB Raid 6 Cost
(8TB) Synology 1400 + 3120 = $4520
(8TB) Qnap 1276 + 3120 = 4396
(10TB) Asustor 1100 + 3600 = $4700

You can build a machine that can do most of what these can do for less money, but to be honest when it comes to expanding the raid and waiting to grow your raid constantly like you are saying I would look at the systems that are purpose built for that kind of load and the warranties that come with them.

*Cheapest 8tb drives ive seen are the 8tb wd red's you can get out of the WD mybooks when bestbuy puts them on sale for 160-170. Shucking drivings out of enclosures is a bit of a lottery though. Freedomeclipse's earlier recommendation of the qnap 4 bay + 5 bay expansion would put you needing larger than 10tb disks to make an 80tb raid 6 work but something like an 8 bay + an expansion could probably be done in the same cost range as the asustor if you needed to save some $$ now.

 

[Fx]

One thing that no one has mentioned is that if you ever have to rebuild an RAID 5/6 (especially old) array with high capacity, low RPM HDD drives, that the chance of failure is orders of magnitude higher than when doing it with RAID 10. You could lose the whole array.

What you also might not know is that RAID is not a backup! This is why I am a huge advocate of RAID 10, duplicated pools/arrays, offsite/online backup.

 

[Iciclebear]

One thing that no one has mentioned that if you ever have to rebuild an RAID 5/6 (especially old) array with high capacity, low RPM HDD drives, that the chance of failure is orders of magnitude higher than when doing it with RAID 10. You could lose the whole array.

What you also might not know is that RAID is not a backup! This is why I am a huge advocate of duplicated pools/arrays and RAID 10.

I mentioned the rebuild times a second ago and completely agree. I could run parity in my Storage Space setup but the parity performance is terrible. Two way 2 column is essentially a raid 10 in performance and I have a pair of 8tb mybooks that I use for backups.

Also to be fair you can lose the whole array with raid 10 as well if the other drive on that side of the array fails but the rebuilds are generally a lot faster. My first raid 5 rebuild on consumer hardware was measured in days, not hours.

 

[newtekie1]

What you also might not know is that RAID is not a backup!

So much THIS! I personally have 2 RAID 5 arrays. The main array backs up to the second array every night.

But having two 80TB arrays might be a little harder to do that with...

 

[Fx]

So much THIS! I personally have 2 RAID 5 arrays. The main array backs up to the second array every night.

But having two 80TB arrays might be a little harder to do that with...

Good. At least you are doing due diligence. I have seen more people than not have single RAID 5/6 arrays and feel completely confident in the redundancy via parity.

This is why online services like Backblaze are feasible. $48/year is not a lot of money compared to spending 2-3k more for a duplicate system or additional set of drives. I myself do not use any form of online backup because I am fortunate to make enough money to have multiple servers.

It has its trade-offs though. I have a higher electric bill and much more heat/noise to manage.

I mentioned the rebuild times a second ago and completely agree. I could run parity in my Storage Space setup but the parity performance is terrible. Two way 2 column is essentially a raid 10 in performance and I have a pair of 8tb mybooks that I use for backups.

Also to be fair you can lose the whole array with raid 10 as well if the other drive on that side of the array fails but the rebuilds are generally a lot faster. My first raid 5 rebuild on consumer hardware was measured in days, not hours.

Yes, you can lose the whole array. What makes RAID 10 so awesome besides the performance is the rebuild times being way faster which significantly reduce the amount of risk you are at during recovery.

To reduce multiple disk failure, one should always buy drives from different vendors and at different months if possible. This of course is in addition to properly managing the temperatures and vibration the drives are exposed to. I like to keep my drives around 31-35c.

I should add that I never spin my drives down; they run 24/7. For controllers, you should also have an active fan over them being that many of them do not even ship with one since they are designed for enterprise environments.

 

[Static~Charge]

One item that no one has mentioned yet: an uninterruptible power supply (UPS). You need a good battery backup with a storage server like this, one that's big enough to handle the computer and the drives. A monitored UPS would be best: it plugs into the computer, software monitors the batteries' power level and shuts down the system cleanly when the power level gets too low.

 

[FordGT90Concept]

Yes, you can lose the whole array. What makes RAID 10 so awesome besides the performance is the rebuild times being way faster which significantly reduce the amount of risk you are at during recovery.

Not by much, especially if a lot of small files on an HDD. Build speed is mostly a function of write speed. RAID6 should theoretically be about the same speed as RAID10 (with a good controller) but RAID6 has the advantage of being able to lose any two drives where all the data is lost if both of the RAID1 dives die on one side of the RAID0. RAID10 is also terrible for going beyond four drives. With the capacities he's talking about, beyond four drives is inevitable where RAID10 becomes very inefficient in performance and capacity.