A Backup Horror Story

A Backup Horror Story

My second job (starting 1992) was for a company named Falcon Information Services, and my task was to rewrite a DOS based technical analysis program written in TP – for TPW for Windows.  Incidentally, my first development task ever for Windows.

It was a small company, only 15 people, but with a significant market share in the financial market, and it was an awesome place to work.

One day, my boss told me the story of the backup system they bought a few years before I started.  At the time, their backbone data delivery was all done on Unix (Solaris), and the operations machine was the development machine as well.  So, one day, they have a discussion: “Perhaps we should get a backup system? We are a bit vulnerable”.  So, they bought a high end backup system, installed it, and made a full disk backup.

“We should try a restore too, so that we know that it works”.

They did, and it didn’t.  It was Friday evening, and the disk partitions were all blank.

My boss ended up on the front page of Norway’s biggest business magazine “Kapital” with the headline “Lost Everything”.  Inside was the story of how he and the three Unix developers rewrote the distribution system from scratch and memory in a single weekend.

On a side note – In Falcon, I got my first “partner in crime” Delphi developer – Hallvard Vassbotn – and I learned a lot!

Two years after I joined, Reuters bought Falcon, and the the core parts of that Unix based system was still in use when I left Reuters in 2007, and remained there until it was replaced by a Reuters system in 2011 – so at least they did a good job!

Morale?  

Hire good programmers that can rewrite software in a weekend?

Make sure you can survive that your backup restore fails!

Keep making backups. You just never know.

#backup   #failure  

11 thoughts on “A Backup Horror Story


  1. Lol – great story. Protecting assets is overlooked in many companies. I once met a guy who made software that could test whether backups worked or not, by restoring them and accessing the data. All plug’n’play and really well done, the dream of everybody who care about data assets. But how do you sell this to an IT department, making them admit that they have no clue about the state of their backup systems?

    Like


  2. About 16 years ago in my small software house, we had 6 computers. To be safe, we kept copies of all the source code on all computers. It worked until burglars broke in one night and stole all our computers… I had a week to rewrite all the code of our main program staring from an old version source I copied to a couple of diskettes.

    Like


  3. That incident was a classic, Lars Fosdal 🙂 It was some time before I joined the company. I remember the job interview session where I was grilled by you and two other developers / owners. It was a great time, and we learned a lot 🙂

    Like


  4. Nowadays (2014): I know a company with very special division: the Backup division. They thoroughly make backup plans and actual backups. 


    One day we had to restore an SQL DB – “no problemo! We are we are insured, we have backup!”.


    Well, mostly insured 🙂 The restoration process failed! Grrrr…


    Then we asked the Backup division – why that happened?


    They answered: “We only do backups, dude! No one asked us to restore from it!”


    Moral? Be careful when doing havoc (;

    Like

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.