Man Accidentally 'Deletes His Company' With One Line of Bad Code

[Monroe]

Man Accidentally 'Deletes His Entire Company' With One Line of Bad Code
‘I feel sorry to say that your company is now essentially dead,’ one person on a coding forum advised Marco Marsala.

Thursday 14 April 2016

http://www.independent.co.uk/life-style/gadgets-and-tech/news/man-accidentally-deletes-his-entire-company-with-one-line-of-bad-code-a6984256.html

A man appears to have deleted his entire company with one mistaken piece of code.

By accidentally telling his computer to delete everything in his servers, hosting provider Marco Marsala has seemingly removed all trace of his company and the websites that he looks after for his customers.

Mr Marsala wrote on a forum for server experts called Server Fault that he was now stuck after having accidentally run destructive code on his own computers. But far from advising them how to fix it, most experts informed him that he had just accidentally deleted the data of his company and its clients, and in so doing had probably destroyed his entire company with just one line of code.

The problem command was "rm -rf": a basic piece of code that will delete everything it is told to. The “rm” tells the computer to remove; the r deletes everything within a given directory; and the f stands for “force”, telling the computer to ignore the usual warnings that come when deleting files.

...

Edited April 14, 2016 by monroe

[xper]

Hehe, sounds familiar to me 

What about backup sites and system files?

[submix8c]

Very familiar, although in my case the bad line of code only allowed payroll tax deduction routine to give a man $5 from the company coffers. Fortunately, Payroll caught it. Assembly Language was so much fun!

And yes, we were smart enough to have triple-backups of the whole company database on-site and a weekly off-site vault backup. IT Manager was very paranoid.

I wonder if they know about Testdisk Photorec?

[xper]

I do on-site backup every day and off-site backup 3 times a week.

[Monroe]

I almost didn't post this story ... my thinking was that the story can't be true ... who doesn't have some sort of backup to rely on.

So I guess maybe this guy didn't ... I have no servers to fool with ... just a simple XP computer setup but I would hate to face a complete reinstall of the OS with all my settings and software added through the years. I do have that backup ready to go.

However, the story had some information in it that might be useful to someone working with servers ... the code that he used and that forum for "server experts" called Server Fault. Maybe that forum was already known by people here ... maybe not.

Still hard to believe there was no backup.

...

[Tripredacus]

Did Server Fault delete the thread? The link from the article goes to a 404.

[xper]

11 minutes ago, Tripredacus said:

Did Server Fault delete the thread? The link from the article goes to a 404.

Seems like they deleted thread 

[xper]

Quote

“I won't even begin enumerating how many errors are simultaneously required in order to be able to completely erase all your servers and all your backups in a single strike.

Good point. I can't imagine how he did it. All servers, backups, off-site backups etc. Something is definitely fishy here.

[Monroe]

I found some other articles with more information about this ... this is from Popular Mechanics and it may contain some of what was first posted at Server Fault before it disappeared ... the link in this article is dead also. It mentions that the guy has around 1,535 customers.

Man Deletes His Whole Company With One Bad Line of Code

Thanks to just one mistake, he instructed the computer to delete everything it could find.

http://www.popularmechanics.com/technology/a20405/man-deletes-company-and-backups-accidentally/

By Eric Limer
Apr 14, 2016

The next time you accidentally close a file without saving, just keep in mind that it could be worse. You could be Marco Marsala, a hosting provider who accidentally and irrevocably deleted his entire business with a faulty line of code.

As Marsala wrote on Server Fault, a forum where he was asking for help with the bind he'd gotten himself into, "I run a small hosting provider with more or less 1535 customers ...All servers got deleted and the offsite backups too."

A dire situation for certain, but one that Marsala was apparently hopeful he could rectify. "How I can recover from a rm -rf / now in a timely manner?" his plea for assistance ends.

That "rm -rf/" is the troublesome line that got Marsala into trouble. It is, essentially, a command that will forcibly delete data without asking for confirmation. As The Independent explains:

The "rm" tells the computer to remove; the r deletes everything within a given directory; and the f stands for "force", telling the computer to ignore the usual warnings that come when deleting files.

... [Because] of an error in the way it was written, the code didn't actually specify anywhere – and so removed everything on the computer.

Why was Marsala running this command at all? It was actually part of his backup procedure, presumably intended to delete old backups. But due to the lack of specificity, it just deleted everything it could get its hands on—including customer websites. Because Marsala didn't have a backup somewhere that was completely isolated from what this particular command could touch, it all went away. It's probably gone forever, according to the experts on Server Fault.

The lessons to be learned here? A few: Always double check instructions to delete anything. Make sure you back up your important data offline. And last but not least, a computer will always do exactly what you tell it to do. If it doesn't do what you wanted, you are the one who messed up. 

Source: Server Fault via The Independant

Also this link ...

This is from developers.slashdot.org and has more information that could also have been in the original Server Fault post ... between the two links ... a person can have a general idea maybe what was originally posted. It also mentions he posted at the Centos help forum.

https://developers.slashdot.org/story/16/04/14/1542246/man-deletes-his-entire-company-with-one-line-of-bad-code

Reader JustAnotherOldGuy writes: Marco Marsala appears to have deleted his entire company with one mistaken piece of code. By accidentally telling his computer to delete everything in his servers, the hosting provider has seemingly removed all trace of his company and the websites that he looks after for his customers. Marsala wrote on a Centos help forum, "I run a small hosting provider with more or less 1535 customers and I use Ansible to automate some operations to be run on all servers. Last night I accidentally ran, on all servers, a Bash script with a rm -rf {foo}/{bar} with those variables undefined due to a bug in the code above this line. All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script)." The terse "rm -rf" is so famously destructive that it has become a joke within some computing circles, but not to this guy. Can this example finally serve as a textbook example of why you need to make offsite backups that are physically removed from the systems you're archiving?"Rm -rf" would mark the block as empty, and if the programmer hasn't written anything new, he should be able to recover nearly all of the data. Something about the story feels weird.

...

Edited April 15, 2016 by monroe

[xper]

he must be running older OS than. 

rm -rf must be run with --no-preserve-root

in order to delete everything.

[submix8c]

4 hours ago, monroe said:

"Rm -rf" would mark the block as empty, and if the programmer hasn't written anything new, he should be able to recover nearly all of the data. Something about the story feels weird.

Yup! All you have to do is "chase" the Indices (not that it's that easy). The info on how this works is all over the internet if you can figure out how to look for. Ever hear of DOS PC-Tools or (Norton?) UnDelete? Same concept.

Maybe he could pay jaclaz to write an ASM program to do it for him. Pretty sure I wouln'd have the patience and my memory is rather fuzzy on Mainframe ASM and PC-ASM is a (kind of) world difference.

Bottom line - Don't indiscriminately run a program on a live system until full testing in test environment is done. (Must have had a hot-hot-hottie date.)

Side not - I can see why said programmer had that thread deleted. Really embarrassing. Wonder if he still has a job.

[jaclaz]

Just in case:

Spoiler

The ServerFault page is cached on Wayback Machine alright:
https://web.archive.org/web/20160415001447/http://serverfault.com/questions/769357/recovering-from-a-rm-rf

Anyway it was a kind of joke/hoax - one in a very bad taste - and seemingly he is going around insisting on how clever he was.

http://www.storypick.com/wrong-code-company-delete/

http://www.pcworld.com/article/3057235/data-center-cloud/that-man-who-deleted-his-entire-company-with-a-line-of-code-it-was-a-hoax.html 
It is a a rare case of extreme stupidity, not curable unfortunately.

He managed to have his name ( Marco Marsala ) and his photos published on a number of online newspaers, so that you can easily recognize him and never trust him with *anything*:

http://www.repubblica.it/tecnologia/2016/04/15/news/cancella_l_azienda_per_sbaglio_la_disavventura_tecnologica_di_marco_marsala-137693154/
and optionally tell him how stupid he is when you happen to meet him.

An interesting kind of publicity, becoming known all over the country and also internationally as a mindless jerk.

jaclaz

 

Edited April 16, 2016 by jaclaz

[Monroe]

jaclaz ... thanks for all the new additional information ... and that the whole thing was just a hoax. Not too funny but it explains why the Server Fault page link no longer worked.

I was expecting someone older ... after checking the photo out ... well, maybe he'll get a few new customers but the stunt may not be all that funny to many people.

...

[submix8c]

HAHAHAHAHAHAHAH!!!! <throat punch...>