Cluster LVM

Patrick Caulfield

Red Hat Confidential

Orientation Meeting 25/1/2004

Red Hat Confidential

Orientation Meeting 25/1/2004

Cluster LVM

Cover name for several related sub-projects

Cluster LVM (clvm)

Base clvm is command-line tools, shared LVs

Working now

Cluster snapshots “csnaps”

Cluster mirroring

Anything else related

Red Hat Confidential

Orientation Meeting 25/1/2004

Cluster LVM

Central control of LVM resources

From any cluster node

Same commands as single-host

Extra functionality for clusters as add-ons to command-line or config file

Aims

Red Hat Confidential

Orientation Meeting 25/1/2004

Cluster LVM

Same LVM tools (same binary) as single-host

Plugin external “locking” module that talks to…

clvmd daemon

Runs on all nodes

Executes remote suspend/reload/resume

Use external lock manager for synchronisation

Architecture

Red Hat Confidential

Orientation Meeting 25/1/2004

Cluster LVM

Distributes LVM operations

Runs on all nodes

Knows if a node is not running one

Talks to lvm binary over a pipe

Holds state of active LVs using locks

Multi-threaded to allow for blocking operations

clvmd

Red Hat Confidential

Orientation Meeting 25/1/2004

Cluster LVM

clvmd has dependencies on the cluster manager

Lock manager

Node services (up/down/names)

Communication services (optional)

These are separated from the main code but not as cleanly as I might like

Cluster interactions

Red Hat Confidential

Orientation Meeting 25/1/2004

Cluster LVM

clvmd was originally written for cman

dlm for lock manager

cman for node management

cman for communications (multicast commands)

clvmd and cman

Red Hat Confidential

Orientation Meeting 25/1/2004

Cluster LVM

Later “ported” to gulm clusters

gulm for lock manager

Limitations on lock modes hack

gulm for node management

Own tcp/ip code for communications

Unicast commands.

clvmd and gulm

Cluster LVM is a blanket term to cover the clustering of the LVM components. The current state is that all basic LVM components can be used in cluster LVM. If you are sharing a block device you need a shared FS - eg GFS!

Single-host (existing) snapshots are supported but can only be used on a single node - enforces by locking.

Other components are in progress.

Basic aim is management of all LVM resources from one (any) node in the cluster. Whatever you can do on a single-host, you can also do cluster-wide.

Resources are cluster-wide unless specified in config file (except single-host snapshots as mentioned).

The external “locking” module was so-called to obscure it’s clustering function when clvm was to be a proprietary add-on. We may be able to come clean on that !

Things like cluster snapshots and cluster mirroring will have their own kernel components (which are dm targets) but the existing architecture should accommodate those without changes because it supports the tools rather than the targets.

Apart from the LVM2 tools, clvmd is the largest chunk of code. The “locking” module for LVM2 is simply code that stuffs the commands down a pipe to the module.

If it spots that a cluster node is not running a clvmd it will error the command as that could affect data integrity.

Clvmd was originally written for cman with no consideration of cluster manager portability (my omission maybe) so the abstraction of cluster manager functions is not that nice. But it is usable as we’ll see.

The current codebase builds two binaries, one for cman and one for gulm. It may be possible to use a shared library plugin for this as I think all the #define conflicts I was having have been sorted out.

The gulm “port” of cman is quite recent but works.

Because it likes the VMS lock modes PW is simulated on gulm using two EX locks.

Also gulm has no communications services so tcp/ip is used. Meaning commands are unicasted rather than multicasted. This has scalability implications and could be fixed.

PowerPoint Presentation

Patrick Caulfield

2004-01-21T10:15:53