# Setting up a file server



## [Ion] (May 31, 2013)

I've realized that it's impractical and insecure to keep my big files (movies, music, TV shows, pictures) on a single 1TB drive.  Consequently, I'm looking to build a file server.  Currently, I have a pair of 1TB Caviar Green drives.  I'm determined to have either a RAID1 or RAID5 array so I have a degree of redundancy in case something goes south.  I've also already managed to fill up about 2/3 of my 1TB drive, so I'm thinking of adding a third drive for RAID5 so that I can expand a bit more.

My most basic requirement is a system with at least 100 megabit ethernet (ideally gigabit) that can redundantly store at least 1TB of files and be accessible (read & write) from a Windows PC.  RAID5 is not strictly required but is highly desired.

Less pressing features (but still ones I would like) are a web/interface or web server for it, the ability to run a Bittorrent client, the ability to have multiple password-protected accounts so that I can give friends access at school, and a FTP server.  Something that I could configure easily is a bonus.

If I can run a VPN through this system or run WCG on it, those are added benefits but not required.  I can get any version of Windows for free from school--not an issue there 

In terms of hardware, I'm thinking of using a Penryn Pentium Dual Core on probably an Asus G31 board w/ ~2GB+ RAM and an 80GB boot drive.

Suggestions?


----------



## shovenose (May 31, 2013)

I would highly recommend RAID10 with four drives... WD Red 3TB's would do you perfectly well for this purpose.  Pentium Dual Core will be plenty. Would recommend 4GB RAM just for the heck of it and it's cheap.


----------



## MxPhenom 216 (May 31, 2013)

Since you only running two drives. Just run a RAID1. I rebuilt my old dell desktop into a file server to keep all my GoPro videos and edited videos on. Running 2 2TB WD Reds in Raid1. perfect little file server for what I need.


----------



## [Ion] (May 31, 2013)

shovenose said:


> I would highly recommend RAID10 with four drives... WD Red 3TB's would do you perfectly well for this purpose.  Pentium Dual Core will be plenty. Would recommend 4GB RAM just for the heck of it and it's cheap.


The problem is that's expensive---I already have the two 1TB drives and I can get a third for about $30.  And I think that 2TB of space is plenty for the foreseeable future.


MxPhenom 216 said:


> Since you only running two drives. Just run a RAID1. I rebuilt my old dell desktop into a file server to keep all my GoPro videos and edited videos on. Running 2 2TB WD Reds in Raid1. perfect little file server for what I need.



As I said, I'd really like to get a third for RAID5 and double my capacity.  For $30 I think it's worth it.

And I can certainly go 4GB RAM--that's cheap.  I have the RAM to do that sitting on my desk


----------



## shovenose (May 31, 2013)

[Ion] said:


> The problem is that's expensive---I already have the two 1TB drives and I can get a third for about $30.  And I think that 2TB of space is plenty for the foreseeable future.
> 
> 
> As I said, I'd really like to get a third for RAID5 and double my capacity.  For $30 I think it's worth it.
> ...



As an administrator who manages tons of servers on a daily basis (enterprise grade servers in a real datacenter) I absolutely hate RAID5. Then again, that's on multi-tenant servers running shared hosting/VPS... we don't have a single dedicated server customer running RAID5... we set up dedicated servers for customers that order them, with 1 or 10. For a personal file server I am sure RAID5 would be fine, though  I'd be worried about filling up all your space soon it you went with the 1TB drives, since you mentioned that, that's why I suggested 4x 2TB. But understandably you want to save money, and ultimately only you know how much storage you need, so I will stand down


----------



## 1freedude (May 31, 2013)

As far as WebUI, I use HFS. HTTP File Server.  Gets me all my stuff anywhere.  Really handy with Android OS.


----------



## Aquinus (May 31, 2013)

shovenose said:


> As an administrator who manages tons of servers on a daily basis (enterprise grade servers in a real datacenter) I absolutely hate RAID5. *Then again, that's on multi-tenant servers running shared hosting/VPS... we don't have a single dedicated server customer running RAID5...* we set up dedicated servers for customers that order them, with 1 or 10. For a personal file server I am sure RAID5 would be fine, though  I'd be worried about filling up all your space soon it you went with the 1TB drives, since you mentioned that, that's why I suggested 4x 2TB. But understandably you want to save money, and ultimately only you know how much storage you need, so I will stand down



First of all, running multiple VMs on a single box requires potentially significant write speed so that VMs could benefit from RAID-10. Not for this case though, most of what the RAID-5 will be doing is reading and holding on to data. Also, I don't know how you can make such a recommendation being a "system admin who manages tons of servers on a daily basis" because I manage private servers that aren't shared and RAID-5 and RAID-6 are the best RAID modes if you have a lot of mission critical data where losing data is unacceptable and is paramount over all else.

Also a good admin will know how to minimize cost and adapt a server to the needs of the person or institutions so I'm going to call shenanigans.

All in all, no one should never consider RAID-10 unless they absolutely need it, otherwise it's a waste of capacity. We're also talking about storing files, not VM images of entire OSs.

My take is: Get the extra disk and go RAID-5. At least that way you will get the added benefit of data being striped across 3 disks as opposed to only duplicating data on 2, so your read speeds should be improved and it will only require you to get one more drive, so I like that idea. Write speeds should even be better despite having to write parity because now you're stripping data as opposed to duplicating it.

I have RAID-5 and it has saved me from headache multiple times. I shun all nay Sayers of RAID-5 and 6. For strictly a file server, capacity and redundancy is much more important than performance IMHO.


----------



## Mindweaver (May 31, 2013)

I agree with Aquinus and say go with a RAID-5 setup, but only if you are going to use a hardware RAID with a BBU, and not Software (_I think Aquinus and shovenose would agree_). If you plan to use software then do yourself a favor, and setup a RAID1 array. I would never setup a RAID10 array for home use mainly due to cost. Then again I would never setup a RAID-5 with out a backup drive on hand either. Since we are pulling our junk out, and measuring then I've been a CIO/IT Manager for over 16 years.


----------



## [Ion] (May 31, 2013)

Mindweaver said:


> I agree with Aquinus and say go with a RAID-5 setup, but only if you are going to use a hardware RAID with a BBU, and not Software (_I think Aquinus and shovenose would agree_). If you plan to use software then do yourself a favor, and setup a RAID1 array. I would never setup a RAID10 array for home use mainly due to cost. Then again I would never setup a RAID-5 with out a backup drive on hand either. Since we are pulling our junk out, and measuring then I've been a CIO/IT Manager for over 16 years.



I'll have to check and see what sort of board I'll be putting it on, but I'd say that there's a good chance it would have to be software RAID.  I don't _think_ that the G31 board I'm planning on supports HW Raid5 (not sure if it supports hardware RAID at all).  Why would software RAID5 be a problem?  Wouldn't a 2.5GHz+ Penryn CPU be enough to handle that?  I definitely think that RAID10 is more expensive than I need ATM, and would be a poor use of resources.

I suppose I'm mostly wondering what operating system should I use?  A Windows Server variant, some mainstream Linux OS (Ubuntu/etc) or a specialized file server Linux distro?  Would I want to use my 80GB disk for boot and then have a RAID array for data?

I think that a 3x1TB RAID5 array would be plenty--that would have about 3x as much storage space as I'm currently using, so there would be considerable room to grow.


----------



## Mindweaver (May 31, 2013)

[Ion] said:


> I'll have to check and see what sort of board I'll be putting it on, but I'd say that there's a good chance it would have to be software RAID.  I don't _think_ that the G31 board I'm planning on supports HW Raid5 (not sure if it supports hardware RAID at all).  Why would software RAID5 be a problem?  Wouldn't a 2.5GHz+ Penryn CPU be enough to handle that?  I definitely think that RAID10 is more expensive than I need ATM, and would be a poor use of resources.
> 
> I suppose I'm mostly wondering what operating system should I use?  A Windows Server variant, some mainstream Linux OS (Ubuntu/etc) or a specialized file server Linux distro?  Would I want to use my 80GB disk for boot and then have a RAID array for data?
> 
> I think that a 3x1TB RAID5 array would be plenty--that would have about 3x as much storage space as I'm currently using, so there would be considerable room to grow.



With it being just a file server it should be fine, but you'll lose speed with out a BBU. Using a BBU you'll be able to use write-back meaning if power is lost all the write requests are put into cache, and when power is returned it picks up where it powered off using the requests stored in cache. but with out a BBU you'll use write-through which means everything wrote won't go to cache and it will be wrote as soon as it gets the request. So, it's slower compared to using a BBU. 

Now what OS to use.. I'd probably go with CentOS or Debian... But any should do


----------



## [Ion] (May 31, 2013)

Mindweaver said:


> With it being just a file server it should be fine, but you'll lose speed with out a BBU. Using a BBU you'll be able to use write-back meaning if power is lost all the write requests are put into cache, and when power is returned it picks up where it powered off using the requests stored in cache. but with out a BBU you'll use write-through which means everything wrote won't go to cache and it will be wrote as soon as it gets the request. So, it's slower compared to using a BBU.
> 
> Now what OS to use.. I'd probably go with CentOS or Debian... But any should do



My only real requirement in terms of speed is the ability to play a HD movie or TV show from the system over the network.

If I went with Linux I'd probably do Debian as I'm more familiar with it---how challenging is it to set up the software RAID?

I'm also investigating WS.


----------



## Mindweaver (May 31, 2013)

Set it up and play around with it to see how you like it. If it meets your needs then go with it, but I would play around with it to get comfortable. Just remember to well document your drives and Raid Array setup, and label them. Also, remember that it will take awhile to rebuild the software array. Good luck!


----------



## IggSter (May 31, 2013)

A couple of things to be mindful of:

If you use motherboard raid it will be almost impossible to move your hard drives with data intact to a new platform. If will also be highly unliky that you will be able to easily migrate/expand your raid in the future. I would therefore recommend a low cost raid card (Adaptec/LSI/etc) which will allow you to move your data from platform to platform.

It would also be worth checking your WD drives for compatibility - the green drives are terrible in a lot of raid setups (a lot of NAS manufacturers have WD green drives on the do not use list). 

With regard FTP: I use Filezilla server...it's free, secure (sFTP/TLS) and works.


Ninja edit: For a web server I would suggest Uniform Server (Afaik it is the only WAMP package that is in production mode out of the wrapper). Another really big bonus is - portability, you can run it without any install.


----------



## Aquinus (May 31, 2013)

IggSter said:


> f you use motherboard raid it will be almost impossible to move your hard drives with data intact to a new platform. If will also be highly unliky that you will be able to easily migrate/expand your raid in the future. I would therefore recommend a low cost raid card (Adaptec/LSI/etc) which will allow you to move your data from platform to platform.



That depends on how he does it. Intel RST(e) will let you move an array between 6-series and 7-series boards. For example, if I put my RAID-5 in a z77 board, I'm pretty sure once SATA is configured for RAID, it would read the configuration from the drives. This is really only true if you're making a big change like Intel to AMD or a 900-series intel chipset to a 7-series chipset or something like that. The same goes for hardware raid cards as well, like cards will still detect the RAID. I also happen to know that RSTe handles RAID fairly well and that expanding RAID-5 is fairly easy and you can even do it while the system is up iirc.



IggSter said:


> It would also be worth checking your WD drives for compatibility - the green drives are terrible in a lot of raid setups (a lot of NAS manufacturers have WD green drives on the do not use list).



Because of lack of TLER, but that's an issue with newer WD blacks too, but I've run into practically no issues with them ever since I disabled patrol read.



IggSter said:


> Ninja edit: For a web server I would suggest Uniform Server (Afaik it is the only WAMP package that is in production mode out of the wrapper). Another really big bonus is - portability, you can run it without any install.



I don't know, I recommend nginx because it is event based, very responsive, uses very little CPU, and has an incredibly small memory footprint. Nginx also excels at static content which is what the OP is looking at serving up. That's just me though.



Mindweaver said:


> Set it up and play around with it to see how you like it. If it meets your needs then go with it, but I would play around with it to get comfortable. Just remember to well document your drives and Raid Array setup, and label them. Also, remember that it will take awhile to rebuild the software array. Good luck!


Speak for yourself, my 3x1TB drives rebuild in just a few hours or so using X79. Back when I was using an nForce4 for RAID, it would take twice as long so it really depends on how you do implement said RAID.

If you can go for it, a dedicated RAID card with a BBU is ideal, but fake raid isn't too bad depending on the board. You can enable write-caching with RST/RSTe but you'll really want the server to be connected to a UPS. Also RAID-5 performance isn't all that bad, more often than not it can saturate 1Gbps which is all you really need, even using software or some form of FakeRAID.



Mindweaver said:


> I agree with Aquinus and say go with a RAID-5 setup, but only if you are going to use a hardware RAID with a BBU, and not Software (I think Aquinus and shovenose would agree).



I wouldn't say a BBU is required for RAID-5. If you don't enable write caching then there is no point of having it in the first place. This sounds like it will be mostly storage so write speed doesn't have to be perfect, just good enough. Reliability seems to be paramount and a BBU won't offer that unless write caching is enabled. With all of this said, I would opt for a BBU if it's feasible though but I wouldn't bend over backwards for it.


----------



## Mindweaver (May 31, 2013)

I'm glad you agree with me Aquinus and just to add I never said you had to use a BBU.. I even explained the difference. Also, remember that to some a few hours is awhile.  I should have stated that a Software RAID rebuild will be slower than a Hardware Raid rebuild.


----------



## Aquinus (Jun 1, 2013)

Mindweaver said:


> I'm glad you agree with me Aquinus and just to add I never said you had to use a BBU.. I even explained the difference.



I know, I'm just making that clear. BBUs are overrated IMHO, but that's my own personal opinion. It is better to use one if you can.



Mindweaver said:


> Also, remember that to some a few hours is awhile.



It really depends on the controller. I have an LSI controller on a server at work that took several hours to rebuild a 500Gb drive on RAID-6 and those are Constellation ES drives, by no means slow or unreliable. I think the simple fact that you can use the RAID array while it's degraded makes the time to rebuild a much smaller factor. For my RAID-5 I have mostly music, video, pictures, files, and a couple games on it as well as a couple VM images. Generally speaking, even when it's degraded and rebuilding, the performance will still let me do everything I normally do without any degradation of my experience and this is on the same machine as the RAID and that experience will remain the same between RAID 1 and RAID 5. So from a user perspective, the real consideration is capacity, which is why I would opt for RAID 5.

Ion: Does the board have any options for fake raid? (Setting SATA to RAID mode.) I really would discourage software RAID if at all possible. I think that will be a mutual consensus.


----------



## newtekie1 (Jun 1, 2013)

If you have a UPS, and you should, a BBU is pointless.  You can still safely enable Write-Back cache with a UPS.

I'd avoid software RAID as well.  I'd buy a dedicated RAID card, an inexpensive one though.  Something like this: HighPoint RocketRAID 640L  Controller Card - Neweg...

Before anyone says it, I know this isn't a true hardware RAID card, it is actually firmware RAID but it is a hell of a lot better than software RAID.  The CPU still does the bulk of the work, the RAID card just issues the commands and houses the RAID configuration settings, but it is still way better than doing software RAID in the OS.

Now, speaking of OS.  I see no reason that any of what you want to do can't be done with Windows 7 Pro, so that is what I'd go with.  Remote admin is provided by remote desktop, Apache gives you a webserver, filezilla gives you a FTP server.  That is really all you need.


----------



## Aquinus (Jun 1, 2013)

newtekie1 said:


> I see no reason that any of what you want to do can't be done with Windows 7 Pro, so that is what I'd go with.



I just would hate recommending using up a Windows 7 key for just this. Debian is more than enough for this and you don't need to burn up a 7 key to do it. Worse comes to worse, if Debian doesn't feel like the right fit he can fall back on Windows. I would strongly recommend giving a *nix distro a try first though.


----------



## newtekie1 (Jun 1, 2013)

Aquinus said:


> I just would hate recommending using up a Windows 7 key for just this. Debian is more than enough for this and you don't need to burn up a 7 key to do it. Worse comes to worse, if Debian doesn't feel like the right fit he can fall back on Windows. I would strongly recommend giving a *nix distro a try first though.



Why not? He says he gets them for free from his school, so why not use it?


----------



## [Ion] (Jun 1, 2013)

Aquinus said:


> I know, I'm just making that clear. BBUs are overrated IMHO, but that's my own personal opinion. It is better to use one if you can.
> 
> 
> 
> ...


If it takes a few hours to build that's fine by me.  Realistically, none of the stuff that is going to be stored on here is mission-critical anyways--all of my important school and work files are continually backed up.  This is all going to be stuff that is just for fun and could always be re-downloaded/re-ripped if it absolutely came to it.

I _expect_ to be using a Gigabyte P35-UD3R, which has the Intel ICH9R South Bridge that supports RAID 0/1/5/10.  However, I don't actually _have_ it yet, so I can't guarantee that's what I'll be using.


newtekie1 said:


> If you have a UPS, and you should, a BBU is pointless.  You can still safely enable Write-Back cache with a UPS.
> 
> I'd avoid software RAID as well.  I'd buy a dedicated RAID card, an inexpensive one though.  Something like this: HighPoint RocketRAID 640L  Controller Card - Neweg...
> 
> ...


I would love to be able to stream .mkv files through a web browser---with a sufficiently fast internet connection and VLC on the target system, would Apache let me do that?

Would that RAID card that you linked be any better than using the ICH9R to do RAID5?  And how key is a UPS?  The power at school has proven quite stable so far...


Aquinus said:


> I just would hate recommending using up a Windows 7 key for just this. Debian is more than enough for this and you don't need to burn up a 7 key to do it. Worse comes to worse, if Debian doesn't feel like the right fit he can fall back on Windows. I would strongly recommend giving a *nix distro a try first though.



I do get Windows licenses for free through the engineering department here.  I am open to trying Debian though.


----------



## newtekie1 (Jun 1, 2013)

[Ion] said:


> I would love to be able to stream .mkv files through a web browser---with a sufficiently fast internet connection and VLC on the target system, would Apache let me do that?
> 
> Would that RAID card that you linked be any better than using the ICH9R to do RAID5?  And how key is a UPS?  The power at school has proven quite stable so far...



Apache wouldn't allow you to do that directly.  Apache is just a web server, you have to build the website yourself.  Though if that is all you want the webserver for forget about apache.  Use something like Twonky Media Server, not free but very cheap.  I use  that to stream media through a web browser, no need for VLC at all.  Though you'll need a pretty beefy internet connection to stream HD mkv files.  For those it would probably just be better to download the file via FTP before playing.  If they are SD it shouldn't be as much of an issue, but you'll still need a decent connection.  Obviously being on a school connection you probably have a decent connection, but you'll need 20Mb/s+ upload to stream HD content, and that is just one stream.

Other than the SATA 6Gb/s ports there really wouldn't be much advantage over the ICH9R, they are both "firmware" RAID, though I prefer an actual card over using the onboard simply because it is much easier to move to another machine if the motherboard has an issue.

A UPS isn't necessary, especially if you don't use Write-Back cache, but I'd still recommend one.  And with basic ones starting at $40 it only makes sense to have one.  Though if you have to skip it initially due to financial constraints it isn't going to kill you, but I'd pick one up eventually.


----------



## 1freedude (Jun 1, 2013)

http://www.rejetto.com/hfs/

So easy...I use it to stream to my phone, my wife's laptop, my tablet (1080p).  Open browser, copy link ( it is set up like a FTP), paste into VLC, or whatever, enjoy.

To get to it from outside my network, No-IP handles the routing.  All free.  I just checked, HFS will run on WINE.


----------



## Aquinus (Jun 1, 2013)

[Ion] said:


> I would love to be able to stream .mkv files through a web browser---with a sufficiently fast internet connection and VLC on the target system, would Apache let me do that?



On your local network? iSCSI is always an option and would behave much more like a drive that was physically connected to your computer. If you're outside on the internet you could do some weird SSH forwarding magic to get iSCSI over the internet. I don't think it would be very fast though. I don't have a whole lot of experience with streaming content over the web. I usually only work on a local network when it comes to stuff like that.


----------



## [Ion] (Jun 3, 2013)

newtekie1 said:


> Apache wouldn't allow you to do that directly.  Apache is just a web server, you have to build the website yourself.  Though if that is all you want the webserver for forget about apache.  Use something like Twonky Media Server, not free but very cheap.  I use  that to stream media through a web browser, no need for VLC at all.  Though you'll need a pretty beefy internet connection to stream HD mkv files.  For those it would probably just be better to download the file via FTP before playing.  If they are SD it shouldn't be as much of an issue, but you'll still need a decent connection.  Obviously being on a school connection you probably have a decent connection, but you'll need 20Mb/s+ upload to stream HD content, and that is just one stream.
> 
> Other than the SATA 6Gb/s ports there really wouldn't be much advantage over the ICH9R, they are both "firmware" RAID, though I prefer an actual card over using the onboard simply because it is much easier to move to another machine if the motherboard has an issue.
> 
> A UPS isn't necessary, especially if you don't use Write-Back cache, but I'd still recommend one.  And with basic ones starting at $40 it only makes sense to have one.  Though if you have to skip it initially due to financial constraints it isn't going to kill you, but I'd pick one up eventually.


Oh, I'm aware that Apache wouldn't let me do that directly--I can probably find instructions or a pre-made thing on how to get the streaming working.

The HD TV shows I have are about 1.4 gigabytes per 40 minute episode, which comes out to about .58 megabytes/second.  I can consistently get twice that uploading from the dorm, so I assume that this would be feasible?

I'll try to get a UPS at some point--at $40 that seems very reasonable.


1freedude said:


> http://www.rejetto.com/hfs/
> 
> So easy...I use it to stream to my phone, my wife's laptop, my tablet (1080p).  Open browser, copy link ( it is set up like a FTP), paste into VLC, or whatever, enjoy.
> 
> To get to it from outside my network, No-IP handles the routing.  All free.  I just checked, HFS will run on WINE.


We have a static IP at school, so that's easy.  I can just memorize it easily.  I'll check out HFS--I think that I've seen it before 


Aquinus said:


> On your local network? iSCSI is always an option and would behave much more like a drive that was physically connected to your computer. If you're outside on the internet you could do some weird SSH forwarding magic to get iSCSI over the internet. I don't think it would be very fast though. I don't have a whole lot of experience with streaming content over the web. I usually only work on a local network when it comes to stuff like that.



I am entirely unfamiliar with iSCSI--I'll check it out though!  The goal would be to stream to another system on the same network or another hall on campus (so across a consistently very fast network).

Thanks for the help everyone!


----------



## Aquinus (Jun 3, 2013)

[Ion] said:


> I am entirely unfamiliar with iSCSI--I'll check it out though!  The goal would be to stream to another system on the same network or another hall on campus (so across a consistently very fast network).



Oh man! You have no idea what you're missing then. iSCSI sends SCSI commands over an IP network, so you're basically accessing files directly on the iSCSI target over a network. Think of it this way. Streaming will work because you're accessing the file over the network directly as if it were a file locally on the machine you're using. It's like a mapped network drive but better. It's not incredibly easy to configure though. This would be best if *your machines* are connecting to *your server only*.

If other people will be connecting to it you will want to use something like Samba. Just about every machine likes Samba and since this is a "local" network it should work well. So in the name of simplicity and that mapping network drives works well (on OS X and Windows,) I would say, go with Samba. 

( All of this is doable in Debian.  )


----------



## newtekie1 (Jun 3, 2013)

A basic UPS like this should do: CyberPower CP350SLG  UPS - Newegg.com

It won't shut the computer down automatically, it is basically only good for extremely breif power outages.

Though for a few bucks more you can get this one: CyberPower Standby Series CP425SLG  UPS - Newegg.c...

It would be better, last longer on battery(though probably only like a minute or so more), and connects to the computer so the computer knows when the power goes out and it is running on battery and can shut itself down before the battery dies.


----------

