NAME¶

sanlock - shared storage lock manager

SYNOPSIS¶

sanlock [COMMAND] [ACTION] ...

DESCRIPTION¶

sanlock is a lock manager built on shared storage. Hosts with access to the storage can perform locking. An application running on the hosts is given a small amount of space on the shared block device or file, and uses sanlock for its own application-specific synchronization. Internally, the sanlock daemon manages locks using two disk-based lease algorithms: delta leases and paxos leases.

•

delta leases are slow to acquire and demand regular i/o to shared storage. sanlock only uses them internally to hold a lease on its "host_id" (an integer host identifier from 1-2000). They prevent two hosts from using the same host identifier. The delta lease renewals also indicate if a host is alive. ("Light-Weight Leases for Storage-Centric Coordination", Chockler and Malkhi.)

•

paxos leases are fast to acquire and sanlock makes them available to applications as general purpose resource leases. The disk paxos algorithm uses host_id's internally to represent different hosts, and the owner of a paxos lease. delta leases provide unique host_id's for implementing paxos leases, and delta lease renewals serve as a proxy for paxos lease renewal. ("Disk Paxos", Eli Gafni and Leslie Lamport.)

Externally, the sanlock daemon exposes a locking interface through libsanlock in terms of "lockspaces" and "resources". A lockspace is a locking context that an application creates for itself on shared storage. When the application on each host is started, it "joins" the lockspace. It can then create "resources" on the shared storage. Each resource represents an application-specific entity. The application can acquire and release leases on resources.

To use sanlock from an application:

•

Allocate shared storage for an application, e.g. a shared LUN or LV from a SAN, or files from NFS.

•

Provide the storage to the application.

•

The application uses this storage with libsanlock to create a lockspace and resources for itself.

•

The application joins the lockspace when it starts.

•

The application acquires and releases leases on resources.

How lockspaces and resources translate to delta leases and paxos leases within sanlock:

Lockspaces

•

A lockspace is based on delta leases held by each host using the lockspace.

•

A lockspace is a series of 2000 delta leases on disk, and requires 1MB of storage. (See Storage below for size variations.)

•

A lockspace can support up to 2000 concurrent hosts using it, each using a different delta lease.

•

Applications can i) create, ii) join and iii) leave a lockspace, which corresponds to i) initializing the set of delta leases on disk, ii) acquiring one of the delta leases and iii) releasing the delta lease.

•

When a lockspace is created, a unique lockspace name and disk location is provided by the application.

•

When a lockspace is created/initialized, sanlock formats the sequence of 2000 on-disk delta lease structures on the file or disk, e.g. /mnt/leasefile (NFS) or /dev/vg/lv (SAN).

•

The 2000 individual delta leases in a lockspace are identified by number: 1,2,3,...,2000.

•

Each delta lease is a 512 byte sector in the 1MB lockspace, offset by its number, e.g. delta lease 1 is offset 0, delta lease 2 is offset 512, delta lease 2000 is offset 1023488. (See Storage below for size variations.)

•

When an application joins a lockspace, it must specify the lockspace name, the lockspace location on shared disk/file, and the local host's host_id. sanlock then acquires the delta lease corresponding to the host_id, e.g. joining the lockspace with host_id 1 acquires delta lease 1.

•

The terms delta lease, lockspace lease, and host_id lease are used interchangeably.

•

sanlock acquires a delta lease by writing the host's unique name to the delta lease disk sector, reading it back after a delay, and verifying it is the same.

•

If a unique host name is not specified, sanlock uses the product_uuid if one is available, otherwise generates a uuid to use as the host's name. The delta lease algorithm depends on hosts using unique names.

•

The application on each host should be configured with a unique host_id, where the host_id is an integer 1-2000.

•

If hosts are misconfigured and have the same host_id, the delta lease algorithm is designed to detect this conflict, and only one host will be able to acquire the delta lease for that host_id.

•

A delta lease ensures that a lockspace host_id is being used by a single host with the unique name specified in the delta lease.

•

Resolving delta lease conflicts is slow, because the algorithm is based on waiting and watching for some time for other hosts to write to the same delta lease sector. If multiple hosts try to use the same delta lease, the delay is increased substantially. So, it is best to configure applications to use unique host_id's that will not conflict.

•

After sanlock acquires a delta lease, the lease must be renewed until the application leaves the lockspace (which corresponds to releasing the delta lease on the host_id.)

•

sanlock renews delta leases every 20 seconds (by default) by writing a new timestamp into the delta lease sector.

•

When a host acquires a delta lease in a lockspace, it can be referred to as "joining" the lockspace. Once it has joined the lockspace, it can use resources associated with the lockspace.

Resources

•

A lockspace is a context for resources that can be locked and unlocked by an application.

•

sanlock uses paxos leases to implement leases on resources. The terms paxos lease and resource lease are used interchangeably.

•

A paxos lease exists on shared storage and requires 1MB of space. It contains a unique resource name and the name of the lockspace.

•

An application assigns its own meaning to a sanlock resource and the leases on it. A sanlock resource could represent some shared object like a file, or some unique role among the hosts.

•

Resource leases are associated with a specific lockspace and can only be used by hosts that have joined that lockspace (they are holding a delta lease on a host_id in that lockspace.)

•

An application must keep track of the disk locations of its lockspaces and resources. sanlock does not maintain any persistent index or directory of lockspaces or resources that have been created by applications, so applications need to remember where they have placed their own leases (which files or disks and offsets).

•

sanlock does not renew paxos leases directly (although it could). Instead, the renewal of a host's delta lease represents the renewal of all that host's paxos leases in the associated lockspace. In effect, many paxos lease renewals are factored out into one delta lease renewal. This reduces i/o when many paxos leases are used.

•

The disk paxos algorithm allows multiple hosts to all attempt to acquire the same paxos lease at once, and will produce a single winner/owner of the resource lease. (Shared resource leases are also possible in addition to the default exclusive leases.)

•

The disk paxos algorithm involves a specific sequence of reading and writing the sectors of the paxos lease disk area. Each host has a dedicated 512 byte sector in the paxos lease disk area where it writes its own "ballot", and each host reads the entire disk area to see the ballots of other hosts. The first sector of the disk area is the "leader record" that holds the result of the last paxos ballot. The winner of the paxos ballot writes the result of the ballot to the leader record (the winner of the ballot may have selected another contending host as the owner of the paxos lease.)

•

After a paxos lease is acquired, no further i/o is done in the paxos lease disk area.

•

Releasing the paxos lease involves writing a single sector to clear the current owner in the leader record.

•

If a host holding a paxos lease fails, the disk area of the paxos lease still indicates that the paxos lease is owned by the failed host. If another host attempts to acquire the paxos lease, and finds the lease is held by another host_id, it will check the delta lease of that host_id. If the delta lease of the host_id is being renewed, then the paxos lease is owned and cannot be acquired. If the delta lease of the owner's host_id has expired, then the paxos lease is expired and can be taken (by going through the paxos lease algorithm.)

•

The "interaction" or "awareness" between hosts of each other is limited to the case where they attempt to acquire the same paxos lease, and need to check if the referenced delta lease has expired or not.

•

When hosts do not attempt to lock the same resources concurrently, there is no host interaction or awareness. The state or actions of one host have no effect on others.

•

To speed up checking delta lease expiration (in the case of a paxos lease conflict), sanlock keeps track of past renewals of other delta leases in the lockspace.

Resource Index

The resource index (rindex) is an optional sanlock feature that applications can use to keep track of resource lease offsets. Without the rindex, an application must keep track of where its resource leases exist on disk and find available locations when creating new leases.

The sanlock rindex uses two align-size areas on disk following the lockspace. The first area holds rindex entries; each entry records a resource lease name and location. The second area holds a private paxos lease, used by sanlock internally to protect rindex updates.

The application creates the rindex on disk with the "format" function. Format is a disk-only operation and does not interact with the live lockspace, so it can be called without first calling add_lockspace. The application needs to follow the convention of writing the lockspace at the start of the device (offset 0) and formatting the rindex immediately following the lockspace area. When formatting, the application must set flags for sector size and align size to match those for the lockspace.

To use the rindex, the application:

•

Uses the "create" function to create a new resource lease on disk. This takes the place of the write_resource function. The create function requires the location of the rindex and the name of the new resource lease. sanlock finds a free lease area, writes the new resource lease at that location, updates the rindex with the name:offset, and returns the offset to the caller. The caller uses this offset when acquiring the resource lease.

•

Uses the "delete" function to remove a resource disk on disk (also corresponding to the write_resource function.) sanlock clears the resource lease and the rindex entry for it. A subsequent call to create may use this same disk location for a different resource lease.

•

Uses the "lookup" function to discover the offset of a resource lease given the resource lease name. The caller would typically call this prior to acquiring the resource lease.

•

Uses the "rebuild" function to recreate the rindex if it is damaged or becomes inconsistent. This function scans the disk for resource leases and creates new rindex entries to match the leases it finds.

•

The "update" function manipulates rindex entries directly and should not normally be used by the application. In normal usage, the create and delete functions manipulate rindex entries. Update is mainly useful for testing or repairs.

Expiration

•

If a host fails to renew its delta lease, e.g. it looses access to the storage, its delta lease will eventually expire and another host will be able to take over any resource leases held by the host. sanlock must ensure that the application on two different hosts is not holding and using the same lease concurrently.

•

When sanlock has failed to renew a delta lease for a period of time, it will begin taking measures to stop local processes (applications) from using any resource leases associated with the expiring lockspace delta lease. sanlock enters this "recovery mode" well ahead of the time when another host could take over the locally owned leases. sanlock must have sufficient time to stop all local processes that are using the expiring leases.

•

sanlock uses three methods to stop local processes that are using expiring leases:

1. Graceful shutdown. sanlock will execute a "graceful shutdown" program that the application previously specified for this case. The shutdown program tells the application to shut down because its leases are expiring. The application must respond by stopping its activities and releasing its leases (or exit). If an application does not specify a graceful shutdown program, sanlock sends SIGTERM to the process instead. The process must release its leases or exit in a prescribed amount of time (see -g), or sanlock proceeds to the next method of stopping.

2. Forced shutdown. sanlock will send SIGKILL to processes using the expiring leases. The processes have a fixed amount of time to exit after receiving SIGKILL. If any do not exit in this time, sanlock will proceed to the next method.

3. Host reset. sanlock will trigger the host's watchdog device to forcibly reset it. sanlock carefully manages the timing of the watchdog device so that it fires shortly before any other host could take over the resource leases held by local processes.

Failures

If a process holding resource leases fails or exits without releasing its leases, sanlock will release the leases for it automatically (unless persistent resource leases were used.)

If the sanlock daemon cannot renew a lockspace delta lease for a specific period of time (see Expiration), sanlock will enter "recovery mode" where it attempts to stop and/or kill any processes holding resource leases in the expiring lockspace. If the processes do not exit in time, sanlock will force the host to be reset using the local watchdog device.

If the sanlock daemon crashes or hangs, it will not renew the expiry time of the per-lockspace connections it had to the wdmd daemon. This will lead to the expiration of the local watchdog device, and the host will be reset.

Watchdog

sanlock uses the wdmd(8) daemon to access /dev/watchdog. wdmd multiplexes multiple timeouts onto the single watchdog timer. This is required because delta leases for each lockspace are renewed and expire independently.

sanlock maintains a wdmd connection for each lockspace delta lease being renewed. Each connection has an expiry time for some seconds in the future. After each successful delta lease renewal, the expiry time is renewed for the associated wdmd connection. If wdmd finds any connection expired, it will not renew the /dev/watchdog timer. Given enough successive failed renewals, the watchdog device will fire and reset the host. (Given the multiplexing nature of wdmd, shorter overlapping renewal failures from multiple lockspaces could cause spurious watchdog firing.)

The direct link between delta lease renewals and watchdog renewals provides a predictable watchdog firing time based on delta lease renewal timestamps that are visible from other hosts. sanlock knows the time the watchdog on another host has fired based on the delta lease time. Furthermore, if the watchdog device on another host fails to fire when it should, the continuation of delta lease renewals from the other host will make this evident and prevent leases from being taken from the failed host.

If sanlock is able to stop/kill all processing using an expiring lockspace, the associated wdmd connection for that lockspace is removed. The expired wdmd connection will no longer block /dev/watchdog renewals, and the host should avoid being reset.

Storage

The sector size and the align size should be specified when creating lockspaces and resources (and rindex). The "align size" is the size on disk of a lockspace or a resource, i.e. the amount of disk space it uses. Lockspaces and resources should use matching sector and align sizes, and must use offsets in multiples of the align size. The max number of hosts that can use a lockspace or resource depends on the combination of sector size and align size, shown below. The host_id of hosts using the lockspace can be no larger than the max_hosts value for the lockspace.

Accepted combinations of sector size and align size, and the corresponding max_hosts (and max host_id) are:

sector_size 512, align_size 1M, max_hosts 2000
sector_size 4096, align_size 1M, max_hosts 250
sector_size 4096, align_size 2M, max_hosts 500
sector_size 4096, align_size 4M, max_hosts 1000
sector_size 4096, align_size 8M, max_hosts 2000

When sector_size and align_size are not specified, the behavior matches the behavior before these sizes could be configured: on devices which report sector size 512, 512/1M/2000 is used, on devices which report sector size 4096, 4096/8M/2000 is used, and on files, 512/1M/2000 is always used. (Other combinations are not compatible with sanlock version 3.6 or earlier.)

Using sanlock on shared block devices that do host based mirroring or replication is not likely to work correctly. When using sanlock on shared files, all sanlock io should go to one file server.

Example

This is an example of creating and using lockspaces and resources from the command line. (Most applications would use sanlock through libsanlock rather than through the command line.)

1.

Allocate shared storage for sanlock leases.

This example assumes 512 byte sectors on the device, in which case the lockspace needs 1MB and each resource needs 1MB.

The example shared block device accessible to all hosts is /dev/leases.

2.

Start sanlock on all hosts.

The -w 0 disables use of the watchdog for testing.

# sanlock daemon -w 0

3.

Start a dummy application on all hosts.

This sanlock command registers with sanlock, then execs the sleep command which inherits the registered fd. The sleep process acts as the dummy application. Because the sleep process is registered with sanlock, leases can be acquired for it.

# sanlock client command -c /bin/sleep 600 &

4.

Create a lockspace for the application (from one host).

The lockspace is named "test".

# sanlock client init -s test:0:/dev/leases:0

5.

Join the lockspace for the application.

Use a unique host_id on each host.

host1:
# sanlock client add_lockspace -s test:1:/dev/leases:0
host2:
# sanlock client add_lockspace -s test:2:/dev/leases:0

6.

Create two resources for the application (from one host).

The resources are named "RA" and "RB". Offsets are used on the same device as the lockspace. Different LVs or files could also be used.

# sanlock client init -r test:RA:/dev/leases:1048576
# sanlock client init -r test:RB:/dev/leases:2097152

7.

Acquire resource leases for the application on host1.

Acquire an exclusive lease (the default) on the first resource, and a shared lease (SH) on the second resource.

# export P=`pidof sleep`
# sanlock client acquire -r test:RA:/dev/leases:1048576 -p $P
# sanlock client acquire -r test:RB:/dev/leases:2097152:SH -p $P

8.

Acquire resource leases for the application on host2.

Acquiring the exclusive lease on the first resource will fail because it is held by host1. Acquiring the shared lease on the second resource will succeed.

# export P=`pidof sleep`
# sanlock client acquire -r test:RA:/dev/leases:1048576 -p $P
# sanlock client acquire -r test:RB:/dev/leases:2097152:SH -p $P

9.

Release resource leases for the application on both hosts.

The sleep pid could also be killed, which will result in the sanlock daemon releasing its leases when it exits.

# sanlock client release -r test:RA:/dev/leases:1048576 -p $P
# sanlock client release -r test:RB:/dev/leases:2097152 -p $P

10.

Leave the lockspace for the application.

host1:
# sanlock client rem_lockspace -s test:1:/dev/leases:0
host2:
# sanlock client rem_lockspace -s test:2:/dev/leases:0

11.

Stop sanlock on all hosts.

# sanlock shutdown

OPTIONS¶

COMMAND can be one of three primary top level choices

sanlock daemon start daemon
sanlock client send request to daemon (default command if none given)
sanlock direct access storage directly (no coordination with daemon)

OTHER¶

timestamp=5332 read_ms=482 write_ms=5525 next_timeouts=0 next_errors=0
timestamp=5353 read_ms=99 write_ms=3161 next_timeouts=0 next_errors=0

INTERNALS¶

0x000 0         leader_record         (sector 0, for host_id 1)


                magic: 0x12212010 


                space_name: foo


                resource_name: host uuid/name


                ...


                host_id bitmap        (leader_record + 256)

0x200 512       leader_record         (sector 1, for host_id 2)


                magic: 0x12212010


                space_name: foo


                resource_name: host uuid/name


                ...


                host_id bitmap        (leader_record + 256)

0x400 1024      leader_record         (sector 2, for host_id 3)


                magic: 0x12212010


                space_name: foo


                resource_name: host uuid/name


                ...


                host_id bitmap        (leader_record + 256)

0xF9E00         leader_record         (sector 1999, for host_id 2000)


                magic: 0x12212010


                space_name: foo


                resource_name: host uuid/name


                ...


                host_id bitmap        (leader_record + 256)

0x000 0         leader_record         (sector 0)


                magic: 0x06152010


                space_name: foo


                resource_name: example1
0x200 512       request_record        (sector 1)


                magic: 0x08292011
0x400 1024      paxos_dblock          (sector 2, for host_id 1)
0x480 1152      mode_block            (paxos_dblock + 128)
0x600 1536      paxos_dblock          (sector 3, for host_id 2)
0x680 1664      mode_block            (paxos_dblock + 128)
0x800 2048      paxos_dblock          (sector 4, for host_id 3)
0x880 2176      mode_block            (paxos_dblock + 128)
0xFA200         paxos_dblock          (sector 2001, for host_id 2000)
0xFA280         mode_block            (paxos_dblock + 128)

0x000 0         leader_record         (sector 0)


                magic: 0x06152010


                space_name: foo


                resource_name: example2
0x200 512       request_record        (sector 1)


                magic: 0x08292011
0x400 1024      paxos_dblock          (sector 2, for host_id 1)
0x480 1152      mode_block            (paxos_dblock + 128)
0x600 1536      paxos_dblock          (sector 3, for host_id 2)
0x680 1664      mode_block            (paxos_dblock + 128)
0x800 2048      paxos_dblock          (sector 4, for host_id 3)
0x880 2176      mode_block            (paxos_dblock + 128)
0xFA200         paxos_dblock          (sector 2001, for host_id 2000)
0xFA280         mode_block            (paxos_dblock + 128)

FILES¶

/etc/sanlock/sanlock.conf

The current settings in use by the sanlock daemon can be seen in the output of 'sanlock status -D'.

•

quiet_fail = 1
See -Q

•

debug_renew = 0
See -R

•

logfile_priority = 4
See -L

•

logfile_use_utc = 0
Use UTC instead of local time in log messages.

•

syslog_priority = 3
See -S

•

names_log_priority = 6
Log resource names at this priority level (uses syslog priority numbers). If this number less than or equal to logfile_priority, each requested resource name and location is recorded in sanlock.log.

•

use_watchdog = 1
See -w

•

high_priority = 1
See -h

•

mlock_level = 1
See -l

•

sh_retries = 8
The number of times to try acquiring a paxos lease when acquiring a shared lease when the paxos lease is held by another host acquiring a shared lease.

•

uname = sanlock
See -U

•

gname = sanlock
See -G

•

our_host_name = <str>
A unique name that a host uses to ensure exclusive ownership of a lockspace host_id (delta lease owner.) The maximum length is 48 characters. If no value is provided in sanlock.conf or on the command line (-e), sanlock attempts to set our_host_name from /sys/devices/virtual/dmi/id/product_uuid. If that is not available, sanlock generates a random uuid to use as our_host_name. Using a fixed our_host_name value will reduce delays when using a lockspace. Using product_uuid will reduce delays further.

•

renewal_read_extend_sec = <seconds>
If a renewal read i/o times out, wait this many additional seconds for that read to complete at the start of the subsequent renewal attempt. When not configured, sanlock waits for an additional io_timeout seconds for a previous timed out read to complete.

•

renewal_history_size = 180
See -H

•

paxos_debug_all = 0
Include all details in the paxos debug logging.

•

debug_io = <str>
Add debug logging for each i/o. "submit" (no quotes) produces debug output at submission time, "complete" produces debug output at completion time, and "submit,complete" (no space) produces both.

•

max_sectors_kb = <str>|<num>
Set to "ignore" (no quotes) to prevent sanlock from checking or changing max_sectors_kb for the lockspace disk when starting a lockspace. Set to "align" (no quotes) to set max_sectors_kb for the lockspace disk to the align size of the lockspace. Set to a number to set a specific number of KB for all lockspace disks. A larger existing max_sectors_kb value will not be reduced by this setting.

•

debug_clients = 0
Enable or disable debug logging for all client connections to the sanlock daemon.

•

debug_cmd = +|-<name>
Enable (+name) or disable (-name) debug logging at the command processing level for specifically named commands, e.g. "debug_cmd = +acquire", or "debug_cmd = -inq_lockspace". Repeat this line for each command name. Use a plus prefix before the name to enable and a minus prefix to disable. By default sanlock disables some command level debugging for commands that are often repetitive and fill the in memory debug buffer. This only affects debug logging, not errors or warnings, and disabling command level debugging for a command does not disable lower level debugging for that command. Special values +all and -all can be used to enable or disable all commands, and can be used before or after other debug_cmd lines.

•

debug_hosts = 1
Log information about other host_id lease renewals. When set to 1 (the default), messages are logged when a host_id lease is observed reaching the failed and dead states. When set to 2, messages are logged when any update (e.g. renewal) is observed for another host_id lease. When set to 0, neither are logged.

•

write_init_io_timeout = <seconds>
The io timeout to use when initializing ondisk lease structures for a lockspace or resource. This timeout is not used as a part of either lease algorithm (as the standard io_timeout is.)

•

max_worker_threads = <num>
See -t

•

io_timeout = <seconds>
The io timeout for disk operations, most notably delta lease renewals. This value is basis for calculating most other timeout values. (Some special cases may use a different io timeout.) Tune this value with caution, it can substantially alter the overall sanlock behavior.

•

watchdog_fire_timeout = <seconds>
The watchdog device timeout. The watchdog device must support the specified value. It is critical that all hosts use the same value. Not doing so will invalidate the lease protection provided by sanlock. The io_timeout should usually be tuned along with this value, e.g. watchdog_fire_timeout = 30 with io_timeout = 5.

•

use_hugepages = <str>
Set to "all" to use transparent hugepages (2MB via MADV_HUGEPAGE.) This should minimize, or prevent, splitting read io's on lease areas. 2MB is allocated for 1MB lease areas, causing some extra memory usage. Set to "none" to disable.

SEE ALSO¶

wdmd(8)