Thursday, July 13, 2017

MongoDB Queryable Backups - Time Travel in the Database

MongoDB maintains a statement  based transaction log of all write operations called the OpLog.

This  is used to keep High Availability Replicas in sync with the Master copy.

The Backup Agent ships this, in encrypted slices to the backup server every minute.

The Backup Server stores these slices in a database, called the Oplog Store.

The Backup server then replays them into a copy of the database on the Backup Server  called the Head Database.

Every few hours, it stops replaying them, looks at what has changed in the binary files of the Head Database and saves those changed file blocks  to another database called the Blockstore, deduplicating as it goes. This is called a snapshot.

When you restore it can pull back a snapshot, and if it still has the transactions since then roll it forward to when you want. Typically it keeps the transactions for 48 hours and the snapshots to a schedule for years.

That's all nice but not the amazing part:

With the Queryable Backups - MongoDB can point the database server at the Blockstore and the Oplog store  and bring up a working, read only copy of the database at some past point in time - without recreating the actual database.

If you thought Oracles flashback was neat - this is mind-blowing.