# This is going to take a while...



## newtekie1 (Sep 12, 2009)

<-- My moods as I watched the Analyze process from start to finish.


----------



## temp02 (Sep 12, 2009)

OMG 1.5TB of data completely fragmented!
Please do a Start->Space (on both HDs), please, please, please!
Also post the time that was required to sort it out if you can.


----------



## n-ster (Sep 12, 2009)

use the ultimatedefrag2008 or something like that.... awesome defragmenter AND you get to put the thing you use the most on the edges so it loads up faster


----------



## newtekie1 (Sep 12, 2009)

n-ster said:


> use the ultimatedefrag2008 or something like that.... awesome defragmenter AND you get to put the thing you use the most on the edges so it loads up faster



Nah, O&O is better, IMO.  Automatically does that, based on last modified, name, or most accessed, if I wish.  It will also put everything as close to the edge as possible.



temp02 said:


> OMG 1.5TB of data completely fragmented!
> Please do a Start->Space (on both HDs), please, please, please!
> Also post the time that was required to sort it out if you can.



That is what I'm doing.  I started it right after taking the screenshot, and I'm letting it run overnight.  I'll let you know if it is still going in the morning.


----------



## btarunr (Sep 12, 2009)

How did it fragment so bad? 

I find O&O the best defrag tool.


----------



## newtekie1 (Sep 12, 2009)

btarunr said:


> How did it fragment so bad?
> 
> I find O&O the best defrag tool.



Here is what happened.

I bought two Seagate LP 5900RPM 1.5TB drives.
I put them in RAID0 to give me a 3TB volume.(Speed wasn't an issue, nor redundancy, as this is purely a backup volume for my 3TB RAID5 array.)
I then started copying everything over to the volume using Robocopy, which took about 16 hours over my gigabit network.
Appearently, because of the way Robocopy works, it fragmented pretty much everything as it coppied.
That is why everything is neatly packed at the top of the drive, but yet still horribly fragmented.

Edit: Just checked on it.  31% finished, 3 Days 6 Hours remaining...


----------



## Mussels (Sep 12, 2009)

>.< lawl


you should try teracopy, it doesnt have these corruption issues and seems similar.


ultimate defrag is better than O&O - O&O always had weird issues (the services are always running, it would wake my PC out of sleep mode sometimes for no reason, and so on)


----------



## newtekie1 (Sep 12, 2009)

Mussels said:


> >.< lawl
> 
> 
> you should try teracopy, it doesnt have these corruption issues and seems similar.
> ...



I prefer RichCopy actually.  The only reason I used Robocopy is because I can automate it easily with a bat file, so I could easily schedule it as a task to run nightly to back up my data, and it comes built into Windows.

And the services running witn O&O allow it to run the automated jobs, if you don't use that feature, just disable them in services.msc.  Never had the issue with it waking from sleep, though I rarely put any of my machines to sleep anymore, they are all on 24/7 folding, or completely shut down.

I don't really see anything that Ultimate Defrag offers over O&O, and I don't get Ultimate Defrag for free...


----------



## Mussels (Sep 12, 2009)

if you disable O&O's services, the entire program stops working. the client ties into the service, and cant run without it.

ultimate defrag lets you choose high and low priority files - EG, windows to the fast part of the drive, your porn folder to the slow part.


----------



## DRDNA (Sep 12, 2009)

Mussels said:


> ultimate defrag is better



+1
so Far Ultimate defrag is the best in mho


----------



## newtekie1 (Sep 12, 2009)

Mussels said:


> if you disable O&O's services, the entire program stops working. the client ties into the service, and cant run without it.
> 
> ultimate defrag lets you choose high and low priority files - EG, windows to the fast part of the drive, your porn folder to the slow part.



It isn't that hard to go in and start the service when you want, not that the service takes up much resources anyway, I'm sure I can manage without the 5MB of RAM the O&O serice uses...(and yes, I've checked, 5MB is all it uses)

And O&O does the same thing, it just does it all automatically for you.  Most accessed files to the fast part of the drive, least access files to the slow.  It is one of the defrag options.

Ultimate Defrag doesn't offer anything over O&O, it doesn't defrag any better, and it isn't free.  So there is really no reason to continue the discussion.


----------



## Tau (Sep 12, 2009)

Out of curiosity why is your backup machine running windows?


----------



## newtekie1 (Sep 12, 2009)

Tau said:


> Out of curiosity why is your backup machine running windows?



Because no other operating system would allow me the functionality I need.  It isn't _just_ a backup machine.  Single purpose machines are stupid unless you are rich, and I ain't rich.


----------



## Tau (Sep 12, 2009)

newtekie1 said:


> Because no other operating system would allow me the functionality I need.  It isn't _just_ a backup machine.  Single purpose machines are stupid unless you are rich, and I ain't rich.



What else does it do?


----------



## AphexDreamer (Sep 12, 2009)

Is it done yet?


----------



## newtekie1 (Sep 12, 2009)

Tau said:


> What else does it do?



FTP Server
Web Server
Backup Server
Media Server
File Server
Print Server
GPU Folding
CPU Folding
Gaming
Normal Workstation Stuff(Internet, Email, Office, etc.)

As soon as Linux allows me to do all that, and is as easy to setup and configure each thing, I'll switch in a heartbeat.



AphexDreamer said:


> Is it done yet?



Nope.


----------



## Tau (Sep 12, 2009)

newtekie1 said:


> FTP Server
> Web Server
> Backup Server
> Media Server
> ...




Ah, yeah some of that could be a bit rough to do under a *nix OS.  Ever thought of running the fileserver part in a VM?  If the machine has enough power (file serving in your house takes no power at all) that would be a nice way to set it up, and administor the data a bit better.  Again just some suggestions 

And IMO windows defraging on huge volumes is a MEGA PITA....  Place i did some work for (actually setting them up a new fileserver setup lol)  had their data stored on a windows based machine with ~8TB's of live data.....  that was a mess took me the better part of a week to get it all straightened away and sorted out.

On a side note my home fileserver runs a *nix build on it, and i also run a couple VMs on it as well for other things.... mind you it has more power than it knows what to do with.... quadcore @ 3Ghz, 8GB ram, 3x gigE bonded NICs in it... lol


----------



## newtekie1 (Sep 12, 2009)

The FTP, Media, and File server all share the same data basically.  No real reason to run it in a virtual machine from what I've seen.


----------



## Tau (Sep 12, 2009)

newtekie1 said:


> The FTP, Media, and File server all share the same data basically.  No real reason to run it in a virtual machine from what I've seen.



yeah if it doesent suite your needs then i guess your stuck with yucky windows for serving


----------



## newtekie1 (Sep 12, 2009)

Tau said:


> yeah if it doesent suite your needs then i guess your stuck with yucky windows for serving



Nothing wrong with Windows, it actually works really well in the roles I've put it in.  Just don't use IIS, and your fine.


----------



## Tau (Sep 12, 2009)

newtekie1 said:


> Nothing wrong with Windows, it actually works really well in the roles I've put it in.  Just don't use IIS, and your fine.



*eye twitch*  IIS is an abomination.


I agree windows does some serving things nicely... all depends what you are running though too.


----------



## Mussels (Sep 13, 2009)

so... is it done yet?


----------



## newtekie1 (Sep 13, 2009)

Nope.  36% Finished, 2 Days 16 Hours remaining...


----------



## temp02 (Sep 13, 2009)

Mussels said:


> ...
> O&O always had weird issues (the services are always running, it would wake my PC out of sleep mode sometimes for no reason, and so on)



Disable Monitoring for all drives... (O&O automatically defragments HDs when PC is idle for a while).
As for the service, it's required to detect new HDs (hot-swap and flash drives for the monitoring stuff) and to "lock" the Windows defragment service, so that if the power is cut during defragment your hard drive don't became corrupted (try doing this with other defragers than O&O and JkDefrag and you'll see what happens).

Also *newtekie1* disable everything else that is running before defragging (like AV, Media players, web browser, screen-saver, really everything and let only the defragger run).


----------



## Mussels (Sep 13, 2009)

temp02 said:


> Disable Monitoring for all drives... (O&O automatically defragments HDs when PC is idle for a while).
> As for the service, it's required to detect new HDs (hot-swap and flash drives for the monitoring stuff) and to "lock" the Windows defragment service, so that if the power is cut during defragment your hard drive don't became corrupted (try doing this with other defragers than O&O and JkDefrag and you'll see what happens).



my point was that while its required for O&O, its not for ultimatedefrag.

You have two similar (and competing) utilities, one needs services running 24/7, hijacks your screensaver and tries to defrag when you're idle, and another that doesnt.


----------



## temp02 (Sep 13, 2009)

Hijacks you screensaver because you wanted it to when installing.


----------



## Mussels (Sep 13, 2009)

temp02 said:


> Hijacks you screensaver because you wanted it to when installing.



actually, even if you tell it not to defrag, i've noticed it changes my  screensaver behaviour (which is off) - sometimes it sets other screensavers on instead of leaving it alone.


----------



## temp02 (Sep 13, 2009)

Either you have a "buggy" build/version of O&O with bad OS integration code or this behavior its not related to O&O, because, at least on XP, mine was off when installing and is stays off while defragging.

Either way, with still more than 60% to go and 2 days have passed, at this rate, only next week we'll see the final results.
Also, *newtekie1* can you tell us how many fragments does the most fragmented file have (bottom-left, File Status, hit Fragments if need to sort the list)?


----------



## newtekie1 (Sep 13, 2009)

Mussels said:


> actually, even if you tell it not to defrag, i've noticed it changes my  screensaver behaviour (which is off) - sometimes it sets other screensavers on instead of leaving it alone.



It does not hijack your screensaver.  It gives you the option when installing to install the screensaver or not, tell it not to and it does nothing to the screensaver.

And I've already told you the service is not required to run 24/7, it only needs to run when O&O runs.

It also only defrags when idle if you install the screensaver, that is the defragging when idle function.

Like I've said already though, the argument is kind of pointless.  I've used both, and O&O is simply better, IMO.  You can argue until your fingers fall off, it won't change this.  And the fact that Ultimate Defrag is not available for free to me, means I'll never use it as a full time defragger.



temp02 said:


> Either you have a "buggy" build/version of O&O with bad OS integration code or this behavior its not related to O&O, because, at least on XP, mine was off when installing and is stays off while defragging.
> 
> Either way, with still more than 60% to go and 2 days have passed, at this rate, only next week we'll see the final results.
> Also, *newtekie1* can you tell us how many fragments does the most fragmented file have (bottom-left, File Status, hit Fragments if need to sort the list)?



It doesn't really bother me how long it takes, and I need the machine's functions to stay running.  Since it is just a data volume, the slowed down transfer rates caused by the defrag running aren't really affecting anything.

And the most fragmented file has 41,804 fragments.


----------



## temp02 (Sep 13, 2009)

newtekie1 said:


> ...
> And the most fragmented file has 41,804 fragments.


You've beaten up my record by 35,000 fragments (+/-)


----------



## Wile E (Sep 13, 2009)

I agree that O&O is better than Ultimate Defrag. It's the only defragger I will use now.


----------



## Timothy B. Schmit (Sep 14, 2009)

temp02 said:


> You've beaten up my record by 35,000 fragments (+/-)



Yes this is completely insane!


----------



## Wile E (Sep 14, 2009)

Any updates? Did it finish yet? You have my record of 1 day and a few hours beat. lol.


----------



## newtekie1 (Sep 14, 2009)

I meant to check it this morning before leaving for work, but forgot.  So I'm not sure, I could remote into the machine and check, but that would kill my F@H clients.  I'll give an update when I get home tonight.


----------



## newtekie1 (Sep 15, 2009)

Wile E said:


> Any updates? Did it finish yet? You have my record of 1 day and a few hours beat. lol.



Update: Still running.  Still says 36% finished, but only 22 Hours left, and down to 37% fragmented.

And just to correct myself, I looked at the analysis report that was generated right before I started the defrag, and the most fragmented file had 54,867 fragments!


----------



## Wile E (Sep 15, 2009)

So, ummm, yeah. I don't think I'll ever used Robocopy for a job that big. I bet you'll think twice as well, eh newtekie? lol.


----------



## newtekie1 (Sep 15, 2009)

Wile E said:


> So, ummm, yeah. I don't think I'll ever used Robocopy for a job that big. I bet you'll think twice as well, eh newtekie? lol.



Depends on the situation, if I had it to do over again, I would use RichCopy to do the initial copy, then Robocopy to do the automated nightly backups.

Though I think even that would cause a large number of fragments, since RichCopy copies 3 or more files at once to maximize throughput.  And it would be writing all 3 files to the drive at once, I'm sure that would almost guarantee all the files would be fragmented to hell also...

Edit:  I was looking at Teracopy, and it seems like it is just a frontend GUI for Robocopy, just like Richcopy is.


----------



## Wile E (Sep 15, 2009)

I probably would've just used Acronis True Image. Not free tho.


----------



## newtekie1 (Sep 15, 2009)

Wile E said:


> I probably would've just used Acronis True Image. Not free tho.



Problem is it is coping from multiple drives/sources into a single volume.  Otherwise I would have, I have a copy of True Image Home 2009 I got through work.


----------



## Mussels (Sep 15, 2009)

newtekie1 said:


> was looking at Teracopy, and it seems like it is just a frontend GUI for Robocopy, just like Richcopy is.



interesting.


about the only comment i have there is that my teracopy transfers dont fragment into the millions


----------



## Stearic (Sep 15, 2009)

Reg the defrag analysis in the original post: I've never seen any volume fragmented that much. 
Well, I use Teracopy too, and frequently move a few tens of GBs between different volumes, but I've never noticed it to fragment the files all that much.  It queues up all the files for transfer and copies them one after the other. Admittedly, I have not copied from multiple source volumes to a single destination volume like the OP has, but any intelligent file transfer utility should queue up the requests to prevent fragmentation and/or thrashing the destination drive's read/write head unnecessarily.

Also, try Diskeeper 2009 pro; it's a very fast defragger and works really well  in auto background defrag mode (better than O&O IMHO) with very low resource consumption. While you use the system, it will defrag/optimize the volumes using only idle resources; I've never had it interfere with other apps on my system.


----------



## newtekie1 (Sep 15, 2009)

Robocopy only copies a single file at a time, Richcopy is the only program I've come across that does multiple file transfers at the same time to maximize throughput.  It still fragmented the files horribly.

Edit: I've copies some other larger files around on different drives using robocopy and none of them came out fragmented like this.  So I did a little messing around, and notices that somehow "compress drive to save space" was enabled.  I wonder if that caused the fragmentation when I was copying files...

Also, still says 36% but only 30 Minutes left.


----------



## Mussels (Sep 16, 2009)

newtekie1 said:


> Robocopy only copies a single file at a time, Richcopy is the only program I've come across that does multiple file transfers at the same time to *maximize throughput*.



are you sure it does? if you're doing one drive to another, a single transfer is almost always faster - mechanical drives can only do one read/write at a time.


you can pretty easily test it by starting up a copy and checking bandwidth (task manager, whatever - anything not windows minutes) and then starting another - the throughput always drops, never goes up. (unless your source is far slower than the destination)

and to be honest, if it fragments your files that badly its not worth it, any speed boost you supposedly got is offset by the slow load times due to fragmentation, and the downtime as you defrag it


----------



## kuroikenshi (Sep 16, 2009)

You can always try SyncToy 2.0 from Microsoft for your copying need's.

SyncToy 2.0


----------



## Hayder_Master (Sep 16, 2009)

is this really completely fragmented , im go download O&O now , very nice


----------



## newtekie1 (Sep 16, 2009)

Mussels said:


> are you sure it does? if you're doing one drive to another, a single transfer is almost always faster - mechanical drives can only do one read/write at a time.
> 
> 
> you can pretty easily test it by starting up a copy and checking bandwidth (task manager, whatever - anything not windows minutes) and then starting another - the throughput always drops, never goes up. (unless your source is far slower than the destination)
> ...



Yes, I'm sure it does.  I've watched it when transfering over the network.  Running a single Robocopy or teracopy usually pushes my gigabit to ~50% usage.  Copying with Richcopy can send it to 75% or more.

Richcopy is going to fragment files, simply because you are working with multiple files, and as you said the drive can only handle one at a time.  So it is going to write pieces of different files right next to eachother.

The fragmentation doesn't really hurt load times, it is just a data volume, so the slight speed loss due to fragmentation isn't noticeable.  And there is no downtime when I defrag.


----------



## newtekie1 (Sep 20, 2009)

The fragmentation was definitely caused by having drive compression turned on.  I formatted the volume, and disabled compression, then recopied everything using Robocopy.  It came through with no fragmentation.  So now we all know the cause, and not to enable drive compression.


----------



## Mussels (Sep 20, 2009)

newtekie1 said:


> The fragmentation was definitely caused by having drive compression turned on.  I formatted the volume, and disabled compression, then recopied everything using Robocopy.  It came through with no fragmentation.  So now we all know the cause, and not to enable drive compression.



good info, thanks for posting the cause.


drive compression is  - it has to unpack into memory to decompress, so it can really hurt performance (and chew a lot of ram)


----------

