Sunday, July 7, 2013

Data Protection Manager 2012 SP1 and Azure - Long Term Data Retention Options

I recently had an engagement with a customer to provide a solution to facilitate long term retention (10+ years) of backups to the "cloud". The customer wanted to use Microsoft System Center 2012 Data Protection Manager (DPM) with Azure Storage to retain long term backups.

The requirements were as followed:
  • The solution must leverage Microsoft DPM 2012 SP1.
  • Short term data retention (1 week) should be kept on premise on a local DPM Storage Pool.
  • Long term data retention (10 + years) should be kept in Windows Azure Storage.
  • No tapes are to be used for Long term data retention.
What a great idea! Keep short term data on premise for quick DR recovery (short RTO) and use windows Azure to recover ad-hoc files, mailboxes etc (Longer RTO).

Many of you may already know that DPM (as of 2012 SP1) has an inbuilt Windows Azure connector to retain backups in the cloud. See TechNet: http://technet.microsoft.com/en-us/library/jj728752.aspx
This at first seems like a great option (A single solution and single interface for long term data retention in Azure).

During discovery and testing the following roadblocks/issues for meeting the long term retention requirements were encounted with the inbuilt DPM to Azure solution:
  • The maximum retention range for Windows Azure backups of DPM is 120 days. The retention range depends on the synchronization settings. If you set to two backups per day then the retention range will be 60 days.
  • You can synchronize to Windows Azure Backup at a maximum of twice per day.
  • A maximum of 448 days is available for Disk Based Recovery (http://technet.microsoft.com/en-us/library/jj628009.aspx)
At this stage Microsoft's stance on long term retention of backup data (over 448 days Disk Based and 120 days Azure based) is for data to go to tape. With the recent acquisition of StorSimple (http://www.storsimple.com/) Appliances by Microsoft this may potentially change in the future.

In the mean time there is still a solution or workaround to retain long term data in Azure and that is to "trick" DPM into thinking it is writing to a tape by using a Virtual Tape Library (VTL) Product. Another cloud appliance (such as StorSimple or the Riverbed WhiteWater) is then needed to copy the backup data to the cloud for long term retention.

At a high level the solution would look like this:
  • Short term data retention for protection groups (under 448 days) to on premise DPM Disk (Storage Pool).
  • Install a Virtual Tape Library (VTL) on the DPM Server such as FireStreamer (https://www.cristalink.com/fs/).
  • Configure the Virtual Tape Library (VTL) to write it's files to a cloud appliance such as Riverbed WhiteWater (http://www.riverbed.com/products-solutions/products/cloud-storage-whitewater/) or StorSimple (http://www.storsimple.com/).
  • Configure the cloud appliance to connect to Windows Azure and setup the disk and azure retention policies i.e. when data is over a week old copy the data up to Azure storage.
  • Configure your Protection Groups to retain long term backups using tape and point it to the Virtual Tape Library.

The solution is not a fully integrated solution (I have faith Microsoft will introduce something in the future). However once configured the long term retention of data is treated as a traditional Tape method, the good thing is there is no requirement for physically managing the tapes off site (this is all handled by the cloud appliance.

I hope this is useful for anyone thinking of using DPM 2012 SP1 for long term backup (over 120 days) retention in Azure.