Restoring Data from Hardware based clones/snapshots

This procedure is based on a taking hardware level SAN snapshots and clones on a NetApp filer of a Red Hat Enterprise Linux 5.1 system and using these to restore data to a running production system. This should work with other setups, but your mileage may vary. Please add/update if you find you need to adapt this procedure for your storage device.

Scenario: You have an issue on a production system where you need to restore data from last week. The server is running a live production service, with data stored on a LUN exported over fibre channel (you could also apply this to iSCSI and other transport protocols), this LUN has hardware snapshots taken 3 times daily on the SAN. You want to copy these files back to the production LUN one at a time, the clone you create from the snapshot has the same Physical Volume (PV) and Volume Group (VG) UUID's.

For the purpose of this example:

/dev/mapper/mpath0 - Boot LUN (This system is also booting from SAN disk)

/dev/mapper/mpath1 - Production Application LUN

/dev/mapper/mpath2 - Snapshot Clone of Application LUN from 3 days ago

1. Prepare your SAN Storage

2. Check servers LVM filter and make sure it can only see the application LUN's and boot LUN's, you can do this through the /etc/lvm/lvm.conf filter e.g.:

 filter = [ "a|/dev/mapper/mpath0|","a|/dev/mapper/mpath1|","r/.*/" ] 

3. Re-scan SCSI bus to find the new LUN: (or whatever you need to do for your transport method)

 echo "- - -" > /sys/class/scsi_host/host[0-3]/scan

4. At this point you want to make sure no-one else is running LVM commands on the system!

5. Run

 vgimportclone.sh -n vgAPP-clone /dev/mapper/mpath2 

Which will change the volume group name to vgAPP-clone and change all the associated uuids. If you are using an older (pre 2.02.28) version of LVM and vgimportclone.sh is not installed on your system you can either pull it from CVS (it is known to work back to 2.02.21) or follow the "Changing uuids on a clone by hand" procedure below to change the uuids by hand (vgimportclone.sh is an implementation of this procedure with error checking)

6. You should now be able to mount your /dev/vgAPP-clone volume groups, file systems and recover data as required

Changing uuids on a clone by hand

1. Create a temporary LVM system directory, copy lvm.conf to this and set LVM_SYSTEM_DIR to point to the directory:

 mkdir /lvmtemp 

 cp /etc/lvm/lvm.conf /lvmtemp 

 export LVM_SYSTEM_DIR=/lvmtemp/ 

NB: The purpose of this temp LVM system directory is so that if your box is fenced/rebooted mid-way through this process, your production services should not be affected by any changes you have made

2. Change filter in your temporary /lvmtemp/lvm.conf file to only see the cloned LUN and your boot LUN:

 filter = [ "a|/dev/mapper/mpath0|","a|/dev/mapper/mpath2|","r/.*/" ] 

3. Run pvscan: Output should be similar to the following and only your Boot and Cloned application LUN should be visible:

  PV /dev/mapper/mpath2  VG  vgAPP      lvm2 [6.00 GB / 0 free]  

  PV /dev/mapper/mpath0p2 VG vgRoot     lvm2 [32.00 GB / 0 free]  

4. Change the UUID's of the cloned PV & VG and rename to avoid duplicates and avoid clashes with the application LUN:

  pvchange --uuid /dev/mapper/mpath2 --config 'global{activation=0}'  

  vgchange --uuid vgAPP --config 'global{activation=0}'  

  vgrename vgAPP vgAPP-clone --config 'global{activation=0}'  

NB: The key here is the switch --config 'global{activation=0}' this deactivates device-mapper interaction and thus the commands only change the LVM metadata and not the devices on the system (otherwise the system will think you are changing the VG used for the production application)

5. Remove temporary LVM system directory and lvm.conf:

 unset LVM_SYSTEM_DIR 

 rm -rf /lvmtemp 

6. Add cloned LUN to your system LVM filter in /etc/lvm/lvm.conf

 filter = [ "a|/dev/mapper/mpath0|","a|/dev/mapper/mpath1|","a|/dev/mapper/mpath2|","r/.*/" ] 

7. Run  pvscan ,  vgscan  &  lvscan  and see both physical volumes, volume groups and logical volumes now appear without any error messages about duplicate PV/VG UUID's

8. Activate the clone's VG:

 vgchange -ay vgAPP-clone 

9. Recreate the LVM /dev/ nodes:

There is a minor bug in the vgchange command, even though we specified  --config 'global{activation=0}'  there was still minor interaction in /dev/vgAPP, renaming it to /dev/vgAPP-clone.

Without affecting any production application/volume group you can recreate these device node entries in /dev/:

 vgmknodes 

10. You should now be able to mount your /dev/vgAPP-clone volume groups, file systems and recover data as required

None: HardwareSnapshotRestore (last edited 2009-12-21 09:02:38 by ChrisProcter)