[TechSpot] Japanese university loses 77TB of research data following a buggy software update

[Space Lynx]

Japanese university loses 77TB of research data following a buggy software update

The culprit for this huge data loss was a faulty script originally meant to delete old, unnecessary log files from Kyoto university's Cray/HPE supercomputer as part of...

www.techspot.com

34 million files got deleted from Kyoto university's supercomputer due to a backup error


Just read this article... thought the storage community would be interested. Is there seriously no way to recover something like this? Why would there not be a redundancy backup not connected to anything? Like every 5-10 tb's the offline backup gets backed up, why isn't this standard procedure for every critical storage situation? I was a history major in college and even I know this... so why didn't they know this? What am I not getting?

 

[micropage7]

when updates give you more headache than before

 

[Chomiq]

The university’s supercomputer supplier, Hewlett Packard Japan (HPE), admitted 100 percent responsibility for the incident and issued a letter of apology later published by the university

Great, that fixes it.

 

[Ferrum Master]

it seems good ol tape drives will back in fashion there.

 

[Harlow]

Agreed. LTO-9 tapes have a native capacity of 26TB and LTO-10 is 48TB. A few tapes and 77TB would be safe. I use LTO-4 at home because 2nd hand drives and tapes are cheap. Of course it does take time, effort and money to back up data.

 

[VulkanBros]

or roughly 53.472.222 3.5" floppy diskettes

 

[Bomby569]

if you can afford a supercomputer and you can't afford 77TB of backup storage at that point you're a moron, sorry

 

[Wirko]

or roughly 53.472.222 3.5" floppy diskettes

The two guys whose job is changing floppies for backups couldn't keep up with two hundred researchers who are generating 10,000 floppies worth of data daily.

Japan's government has been using floppy disks this whole time

Sony stopped making the product in 2011, but Tokyo city is only just starting to phase them out.

www.inputmag.com

 

[Solaris17]

Here is some good conversation when this happened several days ago.

https://www.reddit.com/r/sysadmin/comments/rtnmn8
One user explains it well. As someone who also deals with hundreds of TB of data daily, you have to take this with a grain of salt. Valuable research is valuable research but they are not dead in the water. Especially if you know how big that research is. I do something like 40+tb of deltas a night. Losing a night would not be the end. Just annoying.

 

[bug]

A lot seems to have gone wrong there. A script that's supposed to clean up logs shouldn't have the rights to delete actual data, regardless of the environment variables being set or not. Losing backup data shouldn't be a problem, original data should still be available somewhere. Not testing the upgrade on some sort of a pre-prod environment. And probably more, that's all I got at a glimpse.

 

[neatfeatguy]

Without reading through the article and just reading the headline, I figured they ran a Windows 10 update.

 

[Bomby569]

Without reading through the article and just reading the headline, I figured they ran a Windows 10 update.

a supercomputer running w10 is not a thing , they all run Linux

 

[neatfeatguy]

a supercomputer running w10 is not a thing , they all run Linux

It was a joke. Shot at MS for how they managed their Windows 10 updates and one of them causing deletion of data.

 

[bug]

a supercomputer running w10 is not a thing , they all run Linux

I wouldn't be surprised if every supercomputer ran Win10 in a VM so that people will be able to use Office and other crap

 

[Bomby569]

I wouldn't be surprised if every supercomputer ran Win10 in a VM so that people will be able to use Office and other crap

then proceeds to open link in the mail "how to enlarge penis"

 

[skizzo]

if you can afford a supercomputer and you can't afford 77TB of backup storage at that point you're a moron, sorry

you didn't read the article did you.....they have a backup system....as any supercomputer system would....backup system got an updated script which ends up deleting tons of stuff it shouldn't have.....all thanks to HPJ implementing said borked script.

 

[Bomby569]

you didn't read the article did you.....they have a backup system....as any supercomputer system would....backup system got an updated script which ends up deleting tons of stuff it shouldn't have.....all thanks to HPJ implementing said borked script.

as i read it they deleted the "backup" but that was not a backup, that was storage, as there were no other source for the files, that's not how backup works

 

[R0H1T]

It's a good thing it was "just" 77TB & not 77PB or 77EB

issued a letter of apology later published by the university

At least they delivered the letter? No wait the letter got deleted as well, those a***holes

 

[xtreemchaos]

so what about the old "nothings really deleted unless it been overwrote" im mean it should be recoverable surely.

 

[pygosceles]

Agreed. LTO-9 tapes have a native capacity of 26TB and LTO-10 is 48TB. A few tapes and 77TB would be safe. I use LTO-4 at home because 2nd hand drives and tapes are cheap. Of course it does take time, effort and money to back up data.

You use LTO-4? I've been looking into this for backup and found some drives etc. for cheap, but as usual I can't seem to find enough documentation about what will/won't work for sure, that something's nearly guaranteed to be missing. What's your setup? Is a fiber channel card on a host PC running Ubuntu or Red Hat sufficient?
I concur that LTO3-LTO4 was probably the sweet spot for price/capacity as of about a year ago. Given inflation and gouging, probably still is.

 

[Bomby569]

so what about the old "nothings really deleted unless it been overwrote" im mean it should be recoverable surely.

this is the "backup" so suposedly they are backing up to it and this happened for 2 days before they found out, so i assume it wrote over those 77TB.
What i don't get is you don't mess with a backup, you don't delete stuff on a backup, you do with in storage, so if it goes wrong you have a BACKUP. If you start deleting stuff on a backup that's not a backup anymore, that's storage. Really confusing.

 

[Steevo]

then proceeds to open link in the mail "how to enlarge penis"

I am nagerian bank officl who has 10 bajillion ruppees for you, merely clikc link and provide bank detail for deposit!!


Very legit!

 

[Dr. Dro]

Pardon my French, but くそ, that's gotta hurt

I hope that data scientists and HPE can recover at least a good chunk out of it... this is very much akin to a great library fire