anomalous territory

My primary motivation for doing this was security. I’ve been reading some forensic logs lately and a great way to shoot yourself is if your critical servers perform “push” backups to elsewhere.

A typical example of this would be if I have a database server, and on this happy database server I have a crontab entry that creates a nightly snapshot of all the databases and then runs a little script to scp them all over to a backup server somewhere.

On first impressions, this sounds fine. You’ve got a production database server. You’ve got a remote backup server. You’ve got nightly jobs that run. Awesome! Well protected!

*bzzzzzz* You’re protected from innocent hardware failure, sure, but what happens when someone manages to compromise your database server and login? Regardless of their motivation, if they can read your nightly backup script, you are unbelievably screwed. They’re now not only pwning your production database server, but they have access to your remote backup server too and any historical data might as well be kissed good bye.

I wanted to be defensive and consider this scenario a “when” and not an “if”. My solution is nothing new, but it’s something I don’t see suggested much: Make the backup server PULL from the production database server using private/public ssh keys . For example: backup server has crontab entries that tell it to login to your production systems and copy the same databases you were previously telling the production servers to worry about. Also you can limit any commands the backup user’s login is allowed to issue on the production side. The backup server should also be behind a firewall, on a private network, isolated away from the outside world except for outbound connections in the direction of the production server.

Now not only do you have a happy database, you have an isolated backup server with absolutely no references to where or how the backups are stored.

An added benefit is that if you need to deploy any new production servers, all of their backup requirements can be centralized instead of adding multiple crontab entries to multiple new server at each deployment state. (It’s also a pain if your backup strategy changes greatly as you’d have to maintain many distributed references to your backup infrastructure.)

Obviously this is primarily for smaller scale systems. If you’re running Oracle 10g and have the infrastructure to support distributed RAID arrays full of redundant backups with offline tape libraries, well you’re probably here by accident then. However, if you’re running PostgreSQL or MySQL on a VPS somewhere, I hope this has been thought provoking at the least.

(Later I’ll post some actual How To code/commands to implement this.)