A Backup Horror Story
My second job (starting 1992) was for a company named Falcon Information Services, and my task was to rewrite a DOS based technical analysis program written in TP – for TPW for Windows. Incidentally, my first development task ever for Windows.
It was a small company, only 15 people, but with a significant market share in the financial market, and it was an awesome place to work.
One day, my boss told me the story of the backup system they bought a few years before I started. At the time, their backbone data delivery was all done on Unix (Solaris), and the operations machine was the development machine as well. So, one day, they have a discussion: “Perhaps we should get a backup system? We are a bit vulnerable”. So, they bought a high end backup system, installed it, and made a full disk backup.
“We should try a restore too, so that we know that it works”.
They did, and it didn’t. It was Friday evening, and the disk partitions were all blank.
My boss ended up on the front page of Norway’s biggest business magazine “Kapital” with the headline “Lost Everything”. Inside was the story of how he and the three Unix developers rewrote the distribution system from scratch and memory in a single weekend.
On a side note – In Falcon, I got my first “partner in crime” Delphi developer – Hallvard Vassbotn – and I learned a lot!
Two years after I joined, Reuters bought Falcon, and the the core parts of that Unix based system was still in use when I left Reuters in 2007, and remained there until it was replaced by a Reuters system in 2011 – so at least they did a good job!
Morale?
Hire good programmers that can rewrite software in a weekend?
Make sure you can survive that your backup restore fails!
Keep making backups. You just never know.
The best “backup” is a duplicate copy…:-) I do backups, but I never trust them. I always have a lifeboat copy of critical files.
Lol – great story. Protecting assets is overlooked in many companies. I once met a guy who made software that could test whether backups worked or not, by restoring them and accessing the data. All plug’n’play and really well done, the dream of everybody who care about data assets. But how do you sell this to an IT department, making them admit that they have no clue about the state of their backup systems?
About 16 years ago in my small software house, we had 6 computers. To be safe, we kept copies of all the source code on all computers. It worked until burglars broke in one night and stole all our computers… I had a week to rewrite all the code of our main program staring from an old version source I copied to a couple of diskettes.
Ouch, Enrico Colautti!
I have done a overnight recreation of a missing file once, but it was a pretty basic unit for some string massaging.
These days, the VCS is on a SAN with nightly backup, but I still snapshot VCS tags into encrypted zip archives and store them in three different locations now and then.
That incident was a classic, Lars Fosdal 🙂 It was some time before I joined the company. I remember the job interview session where I was grilled by you and two other developers / owners. It was a great time, and we learned a lot 🙂
Good and memorable times, Hallvard Vassbotn 🙂
Lars Fosdal “string massaging” I can’t stop laughing….
You know, Dalija Prasnikar: the trims, the replaces, the formatters, etc – from before there was a strutils 🙂
Lars Fosdal
Um, sure, we knew that’s what you meant.
Nowadays (2014): I know a company with very special division: the Backup division. They thoroughly make backup plans and actual backups.
One day we had to restore an SQL DB – “no problemo! We are we are insured, we have backup!”.
Well, mostly insured 🙂 The restoration process failed! Grrrr…
Then we asked the Backup division – why that happened?
They answered: “We only do backups, dude! No one asked us to restore from it!”
Moral? Be careful when doing havoc (;