Restoring Data from Hardware based clones/snapshots
This procedure is based on a taking hardware level SAN snapshots and clones on a NetApp filer of a Red Hat Enterprise Linux 5.1 system and using these to restore data to a running production system. This should work with other setups, but your mileage may vary. Please add/update if you find you need to adapt this procedure for your storage device.
Scenario: You have an issue on a production system where you need to restore data from last week. The server is running a live production service, with data stored on a LUN exported over fibre channel (you could also apply this to iSCSI and other transport protocols), this LUN has hardware snapshots taken 3 times daily on the SAN. You want to copy these files back to the production LUN one at a time, the clone you create from the snapshot has the same Physical Volume (PV) and Volume Group (VG) UUID's.
For the purpose of this example:
/dev/mapper/mpath0 - Boot LUN (This system is also booting from SAN disk)
/dev/mapper/mpath1 - Production Application LUN
/dev/mapper/mpath2 - Snapshot Clone of Application LUN from 3 days ago
1. Prepare your SAN Storage
- Create a clone from snapshot
- Map clone to running server
- Make sure the LUN is 'Online'
2. Check servers LVM filter and make sure it can only see the application LUN's and boot LUN's, you can do this through the /etc/lvm/lvm.conf filter e.g.:
filter = [ "a|/dev/mapper/mpath0|","a|/dev/mapper/mpath1|","r/.*/" ]
3. Re-scan SCSI bus to find the new LUN: (or whatever you need to do for your transport method)
- Issue:
echo "- - -" > /sys/class/scsi_host/host[0-3]/scan
Monitor tail -f /var/log/messages and multipath -d -l and all paths to the new LUN should be visible
4. At this point you want to make sure no-one else is running LVM commands on the system!
5. Run
vgimportclone.sh -n vgAPP-clone /dev/mapper/mpath2
Which will change the volume group name to vgAPP-clone and change all the associated uuids. If you are using an older (pre 2.02.28) version of LVM and vgimportclone.sh is not installed on your system you can either pull it from CVS (it is known to work back to 2.02.21) or follow the "Changing uuids on a clone by hand" procedure below to change the uuids by hand (vgimportclone.sh is an implementation of this procedure with error checking)
6. You should now be able to mount your /dev/vgAPP-clone volume groups, file systems and recover data as required
Changing uuids on a clone by hand
1. Create a temporary LVM system directory, copy lvm.conf to this and set LVM_SYSTEM_DIR to point to the directory:
mkdir /lvmtemp
cp /etc/lvm/lvm.conf /lvmtemp
export LVM_SYSTEM_DIR=/lvmtemp/
NB: The purpose of this temp LVM system directory is so that if your box is fenced/rebooted mid-way through this process, your production services should not be affected by any changes you have made
2. Change filter in your temporary /lvmtemp/lvm.conf file to only see the cloned LUN and your boot LUN:
filter = [ "a|/dev/mapper/mpath0|","a|/dev/mapper/mpath2|","r/.*/" ]
3. Run pvscan: Output should be similar to the following and only your Boot and Cloned application LUN should be visible:
PV /dev/mapper/mpath2 VG vgAPP lvm2 [6.00 GB / 0 free]
PV /dev/mapper/mpath0p2 VG vgRoot lvm2 [32.00 GB / 0 free]
4. Change the UUID's of the cloned PV & VG and rename to avoid duplicates and avoid clashes with the application LUN:
pvchange --uuid /dev/mapper/mpath2 --config 'global{activation=0}'
vgchange --uuid vgAPP --config 'global{activation=0}'
vgrename vgAPP vgAPP-clone --config 'global{activation=0}'
NB: The key here is the switch --config 'global{activation=0}' this deactivates device-mapper interaction and thus the commands only change the LVM metadata and not the devices on the system (otherwise the system will think you are changing the VG used for the production application)
5. Remove temporary LVM system directory and lvm.conf:
unset LVM_SYSTEM_DIR
rm -rf /lvmtemp
6. Add cloned LUN to your system LVM filter in /etc/lvm/lvm.conf
filter = [ "a|/dev/mapper/mpath0|","a|/dev/mapper/mpath1|","a|/dev/mapper/mpath2|","r/.*/" ]
7. Run pvscan , vgscan & lvscan and see both physical volumes, volume groups and logical volumes now appear without any error messages about duplicate PV/VG UUID's
8. Activate the clone's VG:
vgchange -ay vgAPP-clone
9. Recreate the LVM /dev/ nodes:
There is a minor bug in the vgchange command, even though we specified --config 'global{activation=0}' there was still minor interaction in /dev/vgAPP, renaming it to /dev/vgAPP-clone.
Without affecting any production application/volume group you can recreate these device node entries in /dev/:
vgmknodes
10. You should now be able to mount your /dev/vgAPP-clone volume groups, file systems and recover data as required