Integrating GFS, Samba, and iSCSI:
Challenges and Opportunities
Matthew O'Keefe, Red Hat
Chris Hertel, Samba Team/UofMN
Storage Geeks
okeefe@redhat.com
crh@samba.org

Quick Overview

Short History of Storage

Shared Storage Clustering Infrastructure in Open Source

Issues in Shared Storage Clustering for Samba

Questions and Open Discussion

Networked Storage: A Short History

Paper tapes, cassette drives, magnetic tapes...

First PC had its own floppy drive, Winchester/hard drive was a step up from that: first drives were 5.25 in., double height

5 Megabyte capacity

Next came LANs and opportunity for network file storage

NFS, Samba, Appleshare, Novell...

Networks grew larger, client/server became standard architecture

Local storage shared from across network

Then, RAID came along so that multiple drives were logically combined and virtualized, and separated from the server

Network Storage History

Multi-ported RAID allowed multiple servers to connect to the same storage device (originally for HA and more performance)

Parallel SCSI ran out of gas

Fibre Channel storage networks built to replace parallel SCSI

Goal: scale storage connectivity (serial interfaces, switched fabric, high speed)

PC technology had 1-to-1 mapping between computer and the logical storage

Storage networks used for performance, availability, scalability, and adaptability

Shared Storage Clustering

VAXcluster deja vu...

Any computer can connect to and mount any logical block device

Shared access to block devices via clustering

More resilience to system crash or shutdown

Fast Forward: Rise of iSCSI

Speed, ubiquity, and familiarity of IP networks drives interest in using existing, paid-for IP infrastructure for storage

Many apps don't need performance of Fibre Channel everywhere, instead

Fibre Channel in core data center

IP-based storage network across the enterprise campus

MAN, even WAN

iSCSI: SAN connectivity over cheap networks

Now for the train wreck: when iSCSI meets clustering

To date, clusters have assumed tightly-coupled, high performance storage networks

Work needed to insure iSCSI integrated with standard clusters

System Software for Clusters

High Availability Fail-Over Software is common

Red Hat Cluster Manager, Wolfpack, VCS, etc.

Typically do not require shared file name space, only shared blocks

Cluster File System

GFS, Lustre, etc...

Cluster Logical Volume Manager

CLVM, Petal

The Cluster File System should look and feel like a local file system

Ditto for Cluster Logical Volume Manager

Physical SAN versus Shared Strg Cluster

1-to-1 server to block device

Duplication

SAN

With Clustering Software

GFS, CLVM, CM

SAN

Storage Pool

Increased throughput & performance

High availability and load balancing

Simplified management

SAN + Data Sharing = Independence of Computing and Storage

The Global File System

Symmetric Cluster File System

GPL'ed in June 2004, being submitted to mainline

Each server in the cluster can read and write to the same block device on the SAN

Lock manager layer tracks read and writes across different servers in the cluster to insure coherency

64-bit Files and File System: Scalable Data Structures

Journal Per Server (for scalability)

High Reliability and Performance

GFS can support the loss of any single component in the cluster (server, lock server, storage, network, SAN)

Flexibly adapts to most hardware

Currently requires SAN infrastructure

Red Hat Cluster Manager

Provides for application migration to another node in the event of node failure or as required for routine maintenance

Heartbeat mechanism between nodes, state information also maintained on quorum disk (which services running, node statys info, etc.)

Administrator defines failover “domains”

Defines target hosts for failed application and all associated resources

Disk, IP Addresses, Files, Databases

stop/start/status scripts

Cluster Logical Volume Manager (CLVM)

CLVM builds upon LVM 2.0 and the kernel device mapper component included in 2.5 and 2.6 Linux distributions

Essentially a cluster-aware version of LVM 2.x

Commands, features, functions all work just fine in a cluster, any Linux server may mount any volume

Provides

Cluster-wide snapshots

Cluster-wide mirroring

Dynamic Volume resizing

Dynamic Multi-Pathing

Volume Replication: synchronous and asynchronous

CLVM to be released with RHEL 4

Scalable File Services

Dynamically add servers to scale

NFS or Samba

(i.e. support more clients)

Dynamically add storage to scale file system capacity and performance

GFS, CLVM, CM

SAN

Storage Pool

NFS/Samba

IP

NFS/Samba Clients

Server Farm acting as NAS head

Scalable Filer

Block Based

File Based

Database Clustering

Data

Oracle9i RAC

Data

Oracle9i RAC

GFS, CLVM, CM

105,000 Oracle Home files replicated on private direct-attach storage

Additional storage and management overhead

Oracle Home files shared

Data sharing for DB and other apps

Embedded Volume Management

Database management and capacity planning simplified

Home

Without GFS

With GFS

Web Services

Internet

Web Server

Shared Storage for Database

Application Farm

Program files and session data shared through Red Hat GFS

GFS, CLVM, CM

DB

Why aren't cluster file systems popular yet?

I.e., why do I have Samba-envy?

SANs have been niche hardware

Its a hard technical problem

Requires tight hardware/software integration

Implementations were often

not fully POSIX-compliant

not integrated with applications

slow on some file workloads (e.g., small files)

not scalable (<10 servers)

not modular, too monolithic

not open source

GFS Features

Redundant high-performance lock manager (RLM)

Flexible new configuration system (CCS))

Oracle and full Direct IO Support (deferred locking)

Full shared mmap() support

Flock/Fcntl lock support

Fuzzy atime modification

Lock LVBs (global operations speedup)

Multi-pathing in Pool (but no auto-failback yet)

Read-Only FS support: support for simultaneous read/write and read-only mounts

Useful for distributed data sharing with GFS

File system Quiescing support (for HW snapshots)

Quotas (including fuzzy updates for performance)

Single Node Performance Near Equivalent to Local File Systems

Context Dependent Path Names

GFS Features

Dynamic inodes

Flat/64-bit metadata structure

Platform independent metadata

Extendible Hashing Directories

Full use of the buffer and page cache (full read and write-back caching)

Online Growable

Works on IP SANs (iSCSI)

GFS Summary

A high performance local file system with inter-machine locking

Optimized from the start to be a cluster file system

Distributed metadata

Does not assume spatial locality is always a good thing

Goal: solve hard problems around building a truly symmetric CFS

When the locks are removed, GFS makes a good local file system

Goals for Clustered Samba:
Jeremy Allison

“A clustered file server ideally has the following properties :

All clients can connect to any server.

A server can fail and clients are transparently reconnected to another server.

All servers can serve out the same set of files.

All file changes are immediately seen on all servers.

Distributed filesystem.

Ability to scale by adding more servers/disk backend”

Recovery Semantics

What are current “normal” semantics for CIFS clients?

If the server goes away, its up to clients to recover

Most do this badly: they punt on the application or ask you to save the file locally

If the app saves the file locally, how is it re-synced to the server?

Client reaction to clustered server failure is the same, but the client can now reconnect to another member of the cluster

We argue that heroic efforts should not be made to save the original TCP connection because

(1) it would shock the natives

(2) its hard to implement true TCP/IP connection fail-over

We suggest a focus on data availability instead

Samba State

A hard part of clustering Samba is keeping the shared state coherent across the cluster

There are two kinds of state:

Protocol-driven state (semantics)

Oplocks

Share modes (read-only, read/write)

Open/Close State

Notify

Samba-implementation state

tdb state is mmap'ed for performance

tdb state includes ?

Issues in Clustering Samba

Oplock breaks must be propagated

All protocol state must be shared and kept reasonably consistent

All Samba-implementation state must be shared and kept reasonably consistent

Assumption: Samba-3 is probably not the right starting point for clustered Samba

Best to design it into Samba-4

Need a consistent programming model for parallelism that works across different kinds of architectures

Tradeoffs

Making Samba cluster-capable is hard work

Extra code, some overhead

More complexity and end cases to handle

Need to identify and segragate the code that affects shared state

One approach: assume no fast, low-latency hardware interconnect is available, but be able to exploit one if it is there

Should run on standard, commodity hardware

Operations that affect global state should be localized as much as possible

Changes that might need to be propagated across the cluster should be minimized and bunched up as much as possible

Summary

Network storage has evolved to support network file systems and shared block storage: iSCSI (edge) and FC (core)

Base clustering substrate now in open source: should be pushed upstream into the mainline Linux kernel

Trying to reach agreement within the community of open source cluster infrastructure software developers

Designing Samba-4 to allow shared storage clustering has significant benefits: scalability, performance, availability, ease of management

Design tradeoffs must be weighed carefully

Go or no-go decision based on tradeoff between additional code complexity and performance impact, versus perceived need for shared storage clustering by Samba team and user community

Some Inspiration...

MICOM global ocean modeling code taken from single Cray vector processor to massively parallel Cray T3D (100s of processors) in 4 months

The Miami team developing the code went from grids that were 50x50x11 grid points to 1500x1500x15 (1000-fold improvement in resolution)

First ocean model calculation to properly resolve Gulf Stream current flow in North Atlantic

Required sophisticated programming techniques which revealed underlying code structure, which was in fact extremely regular

Questions?

crh@samba.org okeefe@redhat.com

Open Source Cluster Infrastructure

http://sources.redhat.com/cluster (includes Cluster Summit talks)

http://openssi.org

http://www.linux-ha.org/

http://clustering.foundries.sourceforge.net/

http://sources.redhat.com/cluster/events/summit2004/

Simplified space and configuration management. Saves money in capacity and management complexity

Eliminates need to replicate 105,000 Oracle HOME files across each node in the cluster

Key GFS Features for this environment include:

Direct I/O: Provides 512-byte block direct I/O to disk for optimal performance. This is available regardless of kernel support of direct I/O capability (new for 5.2)

Context Dependent Path Names (CDPN) allow for node specific data to reside in oracle home partition without complexity of setting up symbolic links (what polyserve and GPFS has to do).

Deferred Locking: New locking state available with 5.2. This allows for intelligent, distributed applications such as oracle 9iRAC to perform multiple direct I/O writes simultaneously to the same file in parallel.

Advantages over OCFS: OCFS provides shared file system capability for Oracle data and index files *only*. The reason is that

OCFS does not support standard Linux file system interface preventing non-db applications read and write to OCFS. Oracle home was promised by Oracle but is not available now. Additionally, standard backup utilities from Legato and Veritas cannot be used to backup data on OCFS file systems for this reason. Oracle provides their own proprietary tools for this.

CIFS Conference 2004

2004-03-09T12:26:10