Problem

Many processes need to update the same file simultaneously.

Solution

Have all processes honor advisory locking by using flock:

open(FH, "+< $path")                or die "can't open $path: $!";
flock(FH, 2)                        or die "can't flock $path: $!";
# update file, then...
close(FH)                           or die "can't close $path: $!";

Discussion

Operating systems vary greatly in the type and reliability of locking techniques available. Perl tries hard to give you something that works, even if your operating system uses its own underlying technique. The flock function takes two arguments: a filehandle and a number representing what to do with the lock on that filehandle. The numbers are normally represented by names like LOCK_EX, which you can get from the Fcntl or IO::File modules.

The LOCK_SH, LOCK_EX, LOCK_UN, and LOCK_NB symbolic values were not available in the Fcntl module before the 5.004 release, and even now they are available only if you ask for them specifically using the :flock tag. Their values are 1, 2, 4, and 8 respectively, which you may supply yourself instead of using the symbolic constants. You'll therefore often see people write:

sub LOCK_SH()  { 1 }     #  Shared lock (for reading)
sub LOCK_EX()  { 2 }     #  Exclusive lock (for writing)
sub LOCK_NB()  { 4 }     #  Non-blocking request (don't stall)
sub LOCK_UN()  { 8 }     #  Free the lock (careful!)

Locks come in two varieties: shared and exclusive. Despite what you might infer by "exclusive," processes aren't required to obey locks on files. Another way of saying this is that flock implements advisory locking. It allows processes to let the operating system suspend would-be writers of a file until any readers are finished with it.

Flocking files is like putting up a stoplight at an intersection. It works only if people pay attention to whether the light is red or green  - or yellow in the case of a shared lock. The red light doesn't stop traffic; it merely signals that traffic should stop. A desperate, ignorant, or rude person will still go flying through the intersection no matter what the light says. Likewise, flock only blocks out other flockers  - not processes trying to do I/O. Unless everyone is polite, accidents can (and will) happen.

The polite process customarily indicates its intent to read from the file by requesting a LOCK_SH. Many processes can have simultaneous shared locks on the file because they (presumably) won't be changing the data. If a process intends to write to the file, it should request an exclusive lock via LOCK_EX. The operating system then suspends the requesting process until all other processes have released their locks, at which point it grants the lock to the suspended process and unblocks it. You are guaranteed that no other process will be able to run flock(FH, LOCK_EX) on the same file while you hold the lock. This is almost but not quite like saying that there can be only one exclusive lock on the file. Forked children inherit not only their parents' open files, but, on some systems, also any locks held. That means if you hold an exclusive lock and fork without execing, your child may also have that same exclusive lock on the file.

The flock function is therefore by default a blocking operation. You can also acquire a lock without wedging your process by using the LOCK_NB flag when you request a lock. This lets you warn the user that there may be a wait until other processes with locks are done:

unless (flock(FH, LOCK_EX|LOCK_NB)) {
    warn "can't immediately write-lock the file ($!), blocking ...";
    unless (flock(FH, LOCK_EX)) {
        die "can't get write-lock on numfile: $!";
    }
}

If you use LOCK_NB and are refused a LOCK_SH, then you know that someone else has a LOCK_EX and is updating the file. If you are refused a LOCK_EX, then someone holds either a LOCK_SH or a LOCK_EX, so you shouldn't try to update the file.

Locks dissolve when the file is closed, which may not be until your process exits. Manually unlocking the file without closing can be perilous due to buffering. The buffer might not have been flushed, leading to a time between unlocking and closing when another process could read data that the contents of your buffer were supposed to replace. A safer way to deal with this is:

if ($] < 5.004) {                   # test Perl version number
     my $old_fh = select(FH);
     local $| = 1;                  # enable command buffering
     local $\ = '';                 # clear output record separator
     print "";                      # trigger output flush
     select($old_fh);               # restore previous filehandle
}
flock(FH, LOCK_UN);

Before version 5.004 of Perl, you had to force a flush. Because folks would often forget to do that, 5.004 changed LOCK_UN so that any pending unwritten buffers are automatically flushed right before the lock is released.

Here's how you increment a number in a file, using flock:

use Fcntl qw(:DEFAULT :flock);

sysopen(FH, "numfile", O_RDWR|O_CREAT)
                                    or die "can't open numfile: $!";
flock(FH, LOCK_EX)                  or die "can't write-lock numfile: $!";
# Now we have acquired the lock, it's safe for I/O
$num = <FH> || 0;                   # DO NOT USE "or" THERE!!
seek(FH, 0, 0)                      or die "can't rewind numfile : $!";
truncate(FH, 0)                     or die "can't truncate numfile: $!";
print FH $num+1, "\n"               or die "can't write numfile: $!";
close(FH)                           or die "can't close numfile: $!";

Closing the filehandle flushes the buffers and unlocks the file. The truncate function is discussed in Chapter 8.

File locking is not as easy as you might think  - or wish. Because locks are advisory, if one process uses locking and another doesn't, all bets are off. Never use the existence of a file as a locking indication because there exists a race condition between the test for the existence of the file and its creation. Furthermore, because file locking is an intrinsically stateful concept, it doesn't get along well with the stateless model embraced by network filesystems such as NFS. Although some vendors claim that fcntl addresses such matters, practical experience suggests otherwise.

NFS locking intimately involves both server and client. Consequently, we know of no general mechanism for guaranteed reliable locking over NFS. You can do it if you know certain operations are atomic in the server or client implementation. You can do it if you know that both server and client support flock or fcntl ; most don't. But in general you can't hope to write code that works everywhere.

Don't confuse Perl's flock with the SysV function lockf. Unlike lockf, flock locks entire files at once. Perl doesn't support lockf directly. The only way to lock a portion of a file is to use the fnctl function, as demonstrated in the lockarea program at the end of this chapter.

See Also

The flock and fcntl functions in perlfunc (1) and in Chapter 3 of Programming Perl; the documentation for the standard Fcntl and DB_File modules (also in Chapter 7 of Programming Perl); Recipe 7.21; Recipe 7.22