6.5 Use Multiple Queues

Operations that cause disk heads to move, such as file creation and removal, are called IOPs. Under Unix, inodes contain information about files, such as permissions, size, and lists of data blocks. A file itself (its data) is rarely, if ever, located on the same track as its inode. To write and sync a file to disk causes data blocks to be allocated and written to, the file's inode to be written, the directory containing the file's name to be written, and the inode for the directory to be written. Deleting the file causes just as many updates to the disk.

Hard disks are typically limited to about 120 IOPs per second. When sendmail successfully delivers a message, it can consume from 10 to 13 IOPs per message. One way to increase the IOPs per second is to install a RAM disk (no head to move). Such solid-state disks often support upward of 14,000 IOPs per second.

Another way to increase the IOPs per second is to spread the queue over many disks and many disk controllers. This is done by using sendmail's built-in support for multiple-queue directories (Section 11.3).

Consider, for example, a major mail-sending machine that has six disks available for use as sendmail's queues:

/queues      not mounted, part of the root filesystem
/queues/q.1  not mounted, part of the root filesystem
/queues/q.1/qf   first disk
/queues/q.1/df   fourth disk
/queues/q.1/xf   mounted as a tmpfs or mfs filesystem
/queues/q.2  not mounted, part of the root filesystem
/queues/q.2/qf   second disk
/queues/q.2/df   fourth disk
/queues/q.2/xf   mounted as a tmpfs or mfs filesystem
/queues/q.3  not mounted, part of the root filesystem
/queues/q.3/qf   third disk
/queues/q.3/df   fourth disk
/queues/q.3/xf   mounted as a tmpfs or mfs filesystem

Note two special characteristics about this arrangement. First, the actual queue directories q.1, q.2, and q.3 need not be mount points for disks. That is because, beginning with V8.10 sendmail, the separate parts that compose an email message can be placed in separate subdirectories corresponding to those parts (specifically, the qf, df, and xf subdirectories).

Second, note that qf files are rewritten each time delivery is tried, whereas df files are written once and only read again during actual delivery. Consequently all the df files could be located on a single disk, with the qf files spread over multiple disks. If you choose this arrangement, you should ensure that the df disk is large enough to hold many very large files.

Third, note that the xf subdirectories are mounted as memory-based filesystems. Under Solaris, for example, they can be mounted as tmpfs filesystems. Under Free BSD they can be mounted as mfs filesystems. In either event, the xf subdirectories are volatile and any files in them will vanish if the machine reboots. This does not matter, however, because the xf files are re-created with each delivery attempt.

Finally, note that symbolic links are legal in the queue directory, so a common area for mounts is not really necessary. You could, for example, change all the df mount points onto symbolic links to a single disk mounted elsewhere (such as /disks/df). Or, at sites that mount all disks generically, all the directories could be symbolic links to that farm of generic disks.

With this multidisk arrangement, we have increased the IOP capacity of the queueing disk system by a factor of six. Even with fast/wide SCSI, six disks are probably the maximum you would want to place on a single controller.

The configuration mc file line that supports this setup looks like this:

define(`QUEUE_DIR',`/queues/q.*')

There is no need to define the qf, df, and xf subdirectories in the configuration file. The sendmail program automatically detects and uses them if they exist.

By using many disks for the queue, you reduce the likelihood that any given disk will ever get too full of messages. Because any queue will remain small, small disks can be used. Many small disks means many cheap disks. Although multiple queues over multiple disks can increase the overall number of IOPs available, there will come a point where the high expense of solid-state disks buys more improvement than the equivalent number of cheap disks and controllers.