Backup media, strategies and pitfalls

[qxp]

What do you use for backups ? How safe are they ? Let's discuss !

Here is what I use:

• USB stick for relatively small files
• SSD drives for moderately large files - at the moment 2 or 4TB SSDs are cheaper per GB than SD cards. One caveat is that as of now (2024) larger capacity QLC drives need to be powered on once a year or so to be sure that data does not decay.
• Spinning magnetic media - good for large data, stores data for longer periods, but susceptible to mechanical damage and magnetic fields. Any opinions on CMR vs Shingled ?
• Optical media - I use M-disk because it promises long term storage. Unfortunately, the largest I got is 100GB. So for relatively small files.
• Print on paper - for tiny text files Inkjet can bleach over time or be washed out by moisture. Laser printing on plastic (like transparencies) probably lasts longer - is anyone doing that ?
• Upload to Internet - for public code and data
• Upload to cloud
• I don't use tape anymore - the capacity is not there, expensive, and can lose data due to tape stretching.

A good strategy is to use multiple media, ideally in separate physical locations. RAID is nice for very large storage, but might be problematic if there is a need to retrieve data from really old volume, especially if using hardware controller.

Anything I missed ? Any new hardware to consider ?

 

[Harlow]

Any opinions on CMR vs Shingled

I avoid SMR drives like the plague in desktop PCs and especially in my four TrueNAS Core servers. Resilvering times can take days instead of hours with SMR. No doubt my external USB drives are all SMR, but they're for archive use only.

RAID is nice for very large storage, but might be problematic if there is a need to retrieve data from really old volume, especially if using hardware controller.

Which is why I use software RAID. I pulled eight SAS drives running RAID-Z2 out of one of my HP Servers and installed them in a desktop PC with an LSI SAS controller. A quick configuration file update and the array is visible again in new hardware.

Optical media - I use M-disk because it promises long term storage.

But will optical drives still be available in 100 years time that can read M-disc? I used to back up to 25GB Blu-ray, but not any more. I do have four Blu-ray burners in various machines though. Best option is to perform a rolling backup to new media every 5 years, in case the old media goes bad or becomes obsolete.

I don't use tape anymore - the capacity is not there, expensive, and can lose data due to tape stretching.

Back in 2018 I started using LTO-4. "Only" 800GB native per tape, but drives and media are dirt cheap now. I can fit 600GB of RAW & JPG files from a 4-week vacation on to a single LTO-4 tape and make several backups on more tapes, in case of "stretching". I'd need boxes of expensive 100GB M-discs to achieve the same number of backups.

Inkjet can bleach over time or be washed out by moisture.

I laminate inkjet photos and distribute 1500 prints each year to people I've photographed. The prints stay reasonably colour fast if not subjected to direct sunlight.

 

[qxp]

Which is why I use software RAID. I pulled eight SAS drives running RAID-Z2 out of one of my HP Servers and installed them in a desktop PC with an LSI SAS controller. A quick configuration file update and the array is visible again in new hardware.

Software raid definitely has advantages and I have used it in the past. For backup, I am concerned that if one or more drives would go bad it would be very difficult to retrieve data from unaffected drives. On the other hand, if a drive was just formatted with ext4 there is hope of going through the image manually.

But will optical drives still be available in 100 years time that can read M-disc? I used to back up to 25GB Blu-ray, but not any more. I do have four Blu-ray burners in various machines though. Best option is to perform a rolling backup to new media every 5 years, in case the old media goes bad or becomes obsolete.

If that issue arises one could make a reader using available parts, such as a microscope: https://hackaday.com/2023/01/22/reading-data-from-a-cd-with-a-microscope/
More short term I like M-Disks because they are immune to magnetic fields, water and drops (while in adequate packaging). Should be fairly resistant to heat and sunlight as well.

Back in 2018 I started using LTO-4. "Only" 800GB native per tape, but drives and media are dirt cheap now. I can fit 600GB of RAW & JPG files from a 4-week vacation on to a single LTO-4 tape and make several backups on more tapes, in case of "stretching". I'd need boxes of expensive 100GB M-discs to achieve the same number of backups.

Interesting ! What would be nice if there was an optical tape like M-disk, but with higher capacity.

I laminate inkjet photos and distribute 1500 prints each year to people I've photographed. The prints stay reasonably colour fast if not subjected to direct sunlight.

Any special ink ?

 

[Frick]

Backups are really simple in theory, but it's always a race between function and simplicity. If it isn't easy enough to do it will not be done. So my answer to all of this is basically that the only things I backup are some media on an external HDD (rarely done), and the "important" stuff to Onedrive.

 

[Harlow]

and the "important" stuff to Onedrive.

As a matter of interest, do you pay for additional storage capacity on OneDrive? I'm too mean (thrifty) to shell out for Terabytes of space in the Cloud., or 1 Gigabit internet access to speed up data retrieval. I make do with 5GB of free space on OneDrive for file transfers to family and friends.

 

[GerKNG]

i have every personal data i own in one folder and a second one with an installer of every program i use. i upload one encrypted copy to my google drive, one copy on my second internal SSD and one third copy on my external SSD which i'll copy to my laptop afterwards.

if my PC burns to a crisp in a house fire i can restory my PC with a Windows USB stick and access to the internet within 30 minutes from anywhere in the world.

 

[Frick]

As a matter of interest, do you pay for additional storage capacity on OneDrive? I'm too mean (thrifty) to shell out for Terabytes of space in the Cloud., or 1 Gigabit internet access to speed up data retrieval. I make do with 5GB of free space on OneDrive for file transfers to family and friends.

100GB version, so I have a total of 115GB (Onedrive originally was 15GB, not 5GB). I also use the camera upload function on Android. Abour €16/year for 100GB is good, and the Windows integration is a huge plus.

 

[DirtyDingusMcgee]

But will optical drives still be available in 100 years time that can read M-disc

I wonder who is gonna care about my mp3 collection in 100 years?

1 hot backup, 1 cold, 1 offsite is plenty for all but the most extreme cases.......missle launch codes or kfc chicken recipe.

 

[Frick]

I wonder who is gonna care about my mp3 collection in 100 years?

Given how crazy we are about archiving in general I'm going to say a lot of people. Maybe not your personal playlists, but some sweet Jaked Off Shorts and Loaded Heads original MP3s? Abolutely.

 

[unwind-protect]

A server with a zraid ZFS filesystem, snapshotted. Hardware has ECC memory.

A trick here is to use zfs receive as non-root, so that snapshots cannot be destroyed by the user doing the backup. So randomware, no matter how clever, can only destroy the newest iteration but not snapshots.

Also plain harddrives, SAS in a drive cage one by one, that I store off-site.

I used tapes until this summer, but something is up with the drive. Haven't investigated.

Small files always synced to a different continent.

 

[W1zzard]

I stopped worrying about backups the moment I used cloud storage. Everything encrypted of course. After backup, replicated to two sites, both on different continents at different hosting companies.

Google: rclone, restic, resticprofile. Stopped using borg
For my personal work machine: Duplicati

All free and open source, cloud storage is so cheap nowadays

Our primary backup is with iDrive E2, secondaries at Hetzner and Microtronix.

 

[Vayra86]

I keep everything locally but I have yet to make some good backup system, the gist now is I just upgrade fast enough for things not to die on me, move my data, and go on. Last upgrade I ditched my last harddrive and am now 100% on SSDs. Might grab an 8TB HDD and hook that up periodically.

I wonder who is gonna care about my mp3 collection in 100 years?

1 hot backup, 1 cold, 1 offsite is plenty for all but the most extreme cases.......missle launch codes or kfc chicken recipe.

I just figured out how fantastic it really is to still have all my mp3's from the earliest days of collecting to today. Not a single ondemand service has everything, and the functionalities keep changing, but your autoplay-lists stay largely the same. At some point letting the algorithm decide for you is becoming boring / annoying AF and I've already reached that point after a few months, it seems. I've gone back to a good old usb stick in the car; spent a few hours to fill it up with the awesomest sauce I could find and its done, now I've got exactly what I want when I want it. Need surprise? Play list at random. Done.

 

[Solaris17]

Clustered storage
Backup to San
SAN backup to backblaze B2

At home:
User folders sym linked to onedrive
Otherwise backup to nas
Nas backup to backblaze b2

 

[W1zzard]

sym linked to onedrive

You trust MS with unencrypted data?

 

[Solaris17]

You trust MS with unencrypted data?

Not trying to start that argument; more importantly I trust myself LESS when it comes to housing wedding photos then I do hosting files for nerds.

When something like owncloud comes along that isn’t a bloated mess; and has enough market penetration for apps and implementations not to suck I will reconsider.

 

[Shrek]

Straight backups are not enough... let's say a file gets corrupted, but you don't know; back it up and you have lost everything.

One needs something like Apple's Time Machine

 

[Frick]

I stopped worrying about backups the moment I used cloud storage.

Yeahhh.... Them corps are better than me at storage for sure.

You trust MS with unencrypted data?

Ooohhh and here is where it gets interesting. Trust. For me I'd say ...... if it was genuinely sensitive, like state secrets, or something criminal, I'd say no. But that is always the equation, isn't it? For a long time I had a journal of sorts in the form of a .docx on my Onedrive, and it was highly personal, but it's also stuff I'd largely be willing to share (because it's all shit about self-hate and wordy ruminations about existing as a human being, and probably a bunch of suicide stuff), so do I really care if ... what would happen BTW? AI training? Ads targeting? What is the worst case scenario here? Do I trust MS, as I would a close friend? Gods no. But I do trust them to not care about me or what I do, beyond if it can make them money somehow. They don't care if I want to die, they don't care what I think of myself or the world, and they shouldn't care about that. It would bother me immensively if they sent me a message or something about maybe seeking out professional help, but I'd be far more bothered by them pretending to care than them having "data" on me.

 

[unwind-protect]

Here is cloud backup for the paranoid:

Tarsnap - Online backups for the truly paranoid

Tarsnap is a secure online backup system for UNIX

www.tarsnap.com

Full encryption, and encryption happens on the user's machine, using a program with open source so that you can review it.

 

[DirtyDingusMcgee]

Our primary backup is with iDrive

I use iDrive for most of my offsite back-up/storage needs. Great company to work with. My go to if i get to choose vendor.

 

[W1zzard]

Here is cloud backup for the paranoid:

Tarsnap - Online backups for the truly paranoid

Tarsnap is a secure online backup system for UNIX

www.tarsnap.com

Full encryption, and encryption happens on the user's machine, using a program with open source so that you can review it.

Last updated 2022, vendor lock-in. Better use restic, rclone, duplicati. These work with ANY storage, local, NAS, any cloud, S3, free cloud, SFTP, WebDAV

beyond if it can make them money somehow

Exactly. They'll just index it with AI to sell you something or to sell the data to someone else who wants to sell you something

Straight backups are not enough... let's say a file gets corrupted, but you don't know; back it up and you have lost everything.

Always use incremental backups

When something like owncloud comes along that isn’t a bloated mess; and has enough market penetration for apps and implementations not to suck I will reconsider.

Check my software suggestions

 

[Calenhad]

I don't really care anymore if my stuff is scanned by vendor X, Y, or Z (or all of the above). It is just a bunch of old documents and cheesy pictures/videos anyway (no nuke launch codes or KFC recipes that I am aware of). Luckily for me I am in Europe, so I have some protection from the most abusive scanning methods. I put what I care most about on cloud. Other, nice to have, stuff is stored locally on an external hdd in case the storage in my computer faceplants. I also stopped caring about storing media content locally years ago. If the world is on fire so Netflix is no longer available, I will have other worries than re-watching *insert content here* one more time. That being said, I also have shelves of blu-ray movies. So I guess I have loads of offline content technically speaking.

Last updated 2022, vendor lock-in. Better use restic, rclone, duplicati. These work with ANY storage, local, NAS, any cloud, S3, free cloud, SFTP, WebDAV

Is it vendor lock-in with fully open sourced software though? I am genuinely curious, not trying to play Devil's Advocate. Otherwise I agree that your recommendations seem the better choice. And for some reason, the name alone makes me want to check out Duplicati.

 

[W1zzard]

Is it vendor lock-in with fully open sourced software though?

If I read correctly, you can only use their storage service, which is backed by Amazon S3 and not free

 

[Dr. Dro]

For long-term storage, I purchase cheap HDDs, and keep them stored in my drawer alongside some dry packs and silica gel to keep the humidity out. None of my data is particularly sensitive, though some of the things I've gathered over the years have become quite difficult to find on the internet. On the cloud, I've had a OneDrive with 1TB of storage thanks to MS365, but no plans to renew that.

 

[Waldorf]

anytime there isnt a relative/friend interested in a slightly used drive, when i "upgrade", i keep it as trash (dl/temp/shadow files) or backup,
and this way can replace "older" tech every couple years anyway.

as the Admin saying goes, if it wasn't backed up, it wasn't important.


@W1zzard
how do you recover os with duplicati?
the documentation/guide section for that is empty, which to me isnt a good sign (they dont care to spend the time/money to give guidance?).

@Shrek
just dont copy the same files to all drives at same time?

as long as you stagger your file updates, its not an issue.
i have one small drive to clone important stuff from my internal backup drive every couple of days/weeks depending what it is, another external that is getting all files updated every couple month,
and another drive that gets "updated" once per year (if), so i can even go back to older versions of tools, e.g. if someone decides to suddenly charge for feature that were free before.
water/fire safe storage that fits a few drives, isnt that expensive, and a good idea, if you dont have "offsite" storage for at least one of the drives.

 

[Ferrum Master]

My diy NAS using OMV. I quit using online solutions due to price and speed. I usually backup my camera RAWs, and those ain't small sized. And the heck I am glad I did backup my oldest RAWs as with current AI denoise and algos you can get much better pictures, I have to revisit my older trips, like Chernobil lol, for fun to compare with Stalker 2.

I have six bay 2.5 inch hotswap with 3x SATA SSDs RAID5, and I have pair of CMR RAID0 3.5 inchers. I have managed to fit all of that and more in a Sharkoon C10 case, flipped it and put some small wheels. Z590i Vision D with i9 11900T.

I dump my workstation data to SSDs then it snapshots to rust spinners, so I have backup of a backup.

The only thing I would need to do is to build another one, to put in some other location, my close friend has an interest, as he also has gathered more than TB with varios trip pics and videos, we could join our servers so it would become more redundant.

While it is more expensive on start, later it pays off. Speed and large storage + you get a server and docker. To be a hostage for online services? No thanks.