|
|
21.1 Unix Log File UtilitiesBecause Unix was designed for use in a time-sharing environment, Unix systems have always maintained log files that recorded who logged into the system and who logged out. Over time, the amount of information in the Unix log files has increased significantly. Today, Unix provides for dramatically expanded logging facilities that record such information as files that are transferred over the network, attempts by users to become the superuser, summary information about all electronic mail messages sent and received, every web page that is downloaded, and much more. In fact, practically any program that engages in periodic or repeating activity, or that runs without user intervention, can record in some log file the fact that it ran. There are two primary ways that Unix log events can be recorded into a log file:
Logs can be recorded in multiple locations:
Figure 21-1. Secure logging host A remote log server can significantly increase the security of an installation. That's because one of the first things that successful attackers do is erase their tracks. They do this by erasing the log files that showed how they became superuser. Such erasing is relatively easy to do if the logs are stored on the computer that was compromised. It is much harder to erase logs that are stored on a remote system because the remote system must also be compromised. In some cases, this is simply not possible! So a remote log server won't prevent people from breaking into your systems, but it might prevent them from hiding their traces. A centralized, remote logging system may also be an ideal place to run intrusion detection software on the collected logs. 21.1.1 Essential Log FilesMost log files are text files that are written line by line by system programs. For example, each time a user on your system tries to become the superuser by using the su command, the su program might append a single line to the log file sulog, which records whether or not the su attempt was successful. Over the years, different versions of Unix have stored their log files in different directories. Early versions of Unix used the directory /usr/adm; this was changed to /var/adm when diskless workstations were introduced. Today, most versions of Unix store their log files in /var/log. Of course, as any program running as root can create files practically anywhere on the system, many programs still store log files in nonstandard directories. Within the log file directory you will typically find several dozen files. Some of these files store the logs for a particular program. Other log files store log events from many programs. And in some cases, a single program may log to more than one file. Table 21-1 lists some of the more common Unix log files.
The following sections describe some of these files and how to use the Unix syslog facility.
21.1.2 Unix syslogUnix provides a general-purpose logging facility called syslog,[3] which consists of:
Individual programs that need to have information logged send the information to syslog. The messages can then be logged to various files, devices, or computers, depending on the sender of the message and its severity. Syslog messages can also be generated from within the Unix kernel. 21.1.2.1 The syslog messageAny program can generate a syslog log message. Each message consists of several parts:
For example, consider this message: Aug 14 08:02:12 <mail.info> r2 postfix/local[81859]: 80AD8E44308: to=<jhalonen@ex.com>, relay=local, delay=1, status=bounced (unknown user: "jhalonen") This message is a log message generated by the postfix program. It means that a message with the ID 80AD8E44308 was received for the user jhalonen@ex.com. The message was bounced because there is no user jhalonen@ex.com. The messages's facility is mail; the priority is info. Here are a few more messages: 7 Jan 18:01:44 ntpd[60085]: offset -0.0039 sec freq 76.340 ppm error 0.053344 poll 10 Aug 18 10:11:52 <daemon.notice> r2 named[85]: denied update from [194.90.12.197].2188 for "ex.com" IN Mar 22 15:01:32 <local0.err> r2 ./capture[498]: capture: ***pcap open fxp1: BIOCSETIF: fxp1: Device not configured The syslog facilities are summarized in Table 21-2. Not all facilities are present on all versions of Unix. The syslog priorities are summarized in Table 21-3.
21.1.2.2 The syslog.conf configuration fileWhat syslog does with a log message is determined by the syslog configuration file, usually /etc/syslog.conf. This file specifies which messages are processed and which are ignored. The /etc/syslog.conf file also controls where messages are logged. A typical syslog.conf file might look like this: *.err;kern.debug;auth.notice /dev/console daemon,auth.notice /var/log/messages lpr.* /var/log/lpd-errs auth.* root,nosmis auth.* @prep.ai.mit.edu *.emerg * *.alert |dectalker mark.* /dev/console
Each line of the file contains two parts:
Message selectors have two parts: a facility and a priority. kern.debug, for example, selects all debug messages (the priority) generated by the kernel (the facility). It also selects all priorities that are greater than debug. An asterisk in place of either the facility or the priority indicates "all." (That is, *.debug means all debug messages, while kern.* means all messages generated by the kernel.) You can also use commas to specify multiple facilities. Two or more selectors can be grouped together by using a semicolon. (See the earlier examples.) The action field specifies one of five actions:[4]
With the following explanation, understanding the typical syslog.conf configuration file shown earlier becomes easy:
Some versions of the syslog daemon use additional characters on the lefthand side to specify additional filters or functionality. Consult your documentation to see all of the control that you have over your syslogd through the syslog.conf file! 21.1.2.3 Using syslog in a networked environmentOne of the tremendously powerful aspects of syslog is that log messages can be sent over a network connection. Using the syslog.conf file, you can specify that some or all of the log messages be sent to another machine. For example, in the previous example, all auth.* messages were sent to the machine prep.ai.mit.edu. One of the problems with the syslog system is that there is no obvious way within it to restrict incoming log messages.[5] In the example, prep.ai.mit.edu will receive the syslog messages whether prep wants them or not. The only control that most versions of syslog have is the -r flag. Specifying the -r flag causes syslogd to reject all remote messages on some systems (or accept them, on most Linux systems!).
syslog's willingness to accept remote messages can result in a denial of service attack when the port is flooded with messages faster than the syslog daemon can process them. Individuals can also log fraudulent messages. For this reason, you must properly screen your network against outside syslog log messages. Another approach is to use a host-based firewall and only accept messages on UDP port 514 from hosts that are deemed to be safe. (Even so, it is possible for an attacker to mount a denial-of-service attack against your log server if one of the acceptable IP addresses is outside your network because there would be no way for your log server to tell the difference between a legitimate log event and a message from an attacker.) You can configure a machine so that all log messages are sent to a remote loghost. To do this, add this line to your syslogd.conf file: *.* @loghost If you are concerned about the possibility of an attacker eavesdropping on syslog packets, you can use a program such as netcat to transmit the logs between the systems using TCP instead of UDP, and direct the TCP traffic through an SSH or SSL tunnel to provide encryption and integrity protection. 21.1.2.4 Incorporating syslog into your own programsThe syslog network protocol has become a de facto[6] standard for logging program and server information over the Internet. Many routers, switches, and remote access devices will transmit syslog messages, and there are syslog servers available for all kinds of computers, even those running Windows.
You may want to insert syslog calls into your own programs to record information of importance. You can do this with the openlog( ) and syslog( ) functions. For example, this program will log "Hi Mom!" to the local0 facility with the info priority: #include <syslog.h>
#include <stdarg.h>
int main(int argc,char **argv)
{
openlog(argv[0],LOG_PID,LOG_LOCAL0);
syslog(LOG_INFO,"Hi Mom!");
return(0);
}
Now let's give it a spin: [simsong@r2 ~] 303 % cc -o mom mom.c [simsong@r2 ~] 304 % ./mom [simsong@r2 ~] 305 % tail -1 /var/log/local0.log Aug 18 23:44:46 <local0.info> r2 ./mom[6581]: Hi Mom! [simsong@r2 ~] 306 % If you are writing shell scripts, you can also log to syslog. Usually, systems with syslog come with the logger command. To log a warning message about a user trying to execute a shell file with invalid parameters, you might include: logger -t ThisProg -p user.notice "Called without required # of parameters" 21.1.2.5 Beware false syslog log entriesThe Unix syslog facility allows any user to create log entries. This capability opens up the possibility for false data to be entered into your logs. An interesting story of such logging was given to us by Alec Muffet:
21.1.3 Rotating Logs with newsyslogLog files grow with time. In fact, unless you make some provisions for pruning your system's log files, your log files will grow and grow until they fill up the partition on which they reside. Early Unix systems relied on their human operators to manually prune the log files. Most sites found this an onerous task; many sites developed software that would automatically roll over the log files as needed. This task was complicated because some programs keep an open file handle pointing to their log files and need to be sent a signal (typically a kill -1) when the log file is renamed. The newsyslog program provides a unified system for rotating log files.[7] Designed to run on an hourly basis, the program reads a configuration file that specifies the names of log files and rules that determine when the files should be rotated and how that rotation should be done. newsyslog has many features, including:
The newsyslog program is typically run hourly from cron. When the program runs, it examines its configuration file and determines which of the log files need to be rotated. It then rotates the necessary files and exits. The format of each line in the file is shown in Table 21-4. A sample configuration file is shown in Example 21-1.
Example 21-1. A sample /etc/newsyslog.conf configuration file# logfilename [owner:group] mode count size when [ZJB] [pidfile] [sig_num] /var/log/cron 600 3 100 * Z /var/log/amd.log 644 7 100 * Z /var/log/kerberos.log 644 7 100 * Z /var/log/lpd-errs 644 7 100 * Z /var/log/maillog 644 7 * 168 Z /var/log/messages 644 5 100 * Z /var/log/all.log 600 7 * @T00 Z /var/log/slip.log 600 3 100 * Z /var/log/ppp.log 600 3 100 * Z /var/log/security 600 10 100 * Z /var/log/wtmp 644 3 1000 * B /var/log/daily.log 640 7 * @T00 Z /var/log/weekly.log 640 5 1 $W6D0 Z /var/log/monthly.log 640 12 * $M1D0 Z /var/log/console.log 640 5 100 * Z newsyslog is widely used on Unix systems. However, the default configuration file is very conservative. Many sites may wish to modify their /etc/newsyslog.conf configuration file so that logs are kept for longer periods of time. Log rotation should be coordinated with other backup procedures so that you can access a continuous log history. Another good idea is to generate MD5 or SHA-1 cryptographic checksums of logs when they are rotated so that you can verify their integrity in the future. (This is considerably easier with rotation software that allows you to run arbitrary commands after rotation, like logrotate.) 21.1.4 Swatch: A Log File Analysis ToolSwatch is the System Watchdog. Developed by E. Todd Atkins at Stanford's EE Computer Facility, Swatch is a simple Perl program that monitors log files and alerts you if a particular pattern is noticed. Swatch allows a great deal of flexibility. Although Swatch is not currently included as standard software with any Unix distribution, it is available at http://www.oit.ucsb.edu/~eta/swatch/. Swatch seems well-suited to organizations operating between 1 and 20 servers. Organizations with a larger number of servers tend to create their own log file analysis tools. If you are at such an organization, you may wish to learn about Swatch to see which features would be appropriate to put into your own system. Or you might want to try to use Swatch, because it's pretty good.[8]
21.1.4.1 Running SwatchSwatch has two modes of operation. It can be run in batch, scanning a log file according to a preset configuration. Alternatively, Swatch can monitor your log files in real time, looking at lines as they are added. Swatch is run from the command line: % swatch options input-source The following are the options that you are most likely to use when running Swatch:
The input source is specified with the following arguments:
21.1.4.2 The Swatch configuration fileSwatch's operation is controlled by a configuration file. Each line of the file consists of four tab-delimited fields, and has the form: /pattern/[,/pattern/,...] action[,action,...] [[[HH:]MM:]SS] [start:length] The first field specifies a pattern that is scanned for on each line of the log file. The pattern is in the form of a Perl regular expression, which is similar to regular expressions used by egrep. If more than one pattern is specified, then a match on any pattern will signify a match. The second field specifies an action to be taken each time a pattern in the first field is matched. Swatch supports the following actions:
The third and fourth fields are optional. They give you a technique for controlling identical lines which are sent to the log file. If you specify a time, then Swatch will not alert you for identical lines that are sent to the log file within the specified period of time. Instead, Swatch will merely notify you when the first line is triggered, and then after the specified period of time has passed. The fourth field specifies the location within the log file where the timestamp takes place. For example, on one system, you may have a process that generates the following message repeatedly in the log file: Apr 3 01:01:00 next routed[9055]: bind: Bad file number Apr 3 02:01:00 next routed[9135]: bind: Bad file number Apr 3 03:01:00 next routed[9198]: bind: Bad file number Apr 3 04:01:00 next routed[9273]: bind: Bad file number You can catch the log file message with the following Swatch configuration line: /routed.*bind/ echo 24:00:00 0:16 This line should cause Swatch to report the routed message only once a day, with the following message: *** The following was seen 20 times in the last 24 hours(s): ==> next routed[9273]: bind: Bad file number Be sure that you use the tab character to separate the fields in your configuration file. If you use spaces, you may get an error message like this: parse error in file /tmp/..swatch..2097 at line 24, next 2 tokens "/routed.*bind / echo" parse error in file /tmp/..swatch..2097 at line 27, next token "}" Execution of /tmp/..swatch..2097 aborted due to compilation errors. 21.1.5 lastlog FileUnix records the last time that each user logged into the system in the lastlog log file. This time is displayed each time you log in: login: ti
password: books2sell
Last login: Tue Jul 12 07:49:59 on tty01
This time is also reported when the finger command is used: % finger tim Login name: tim In real life: Tim Hack Directory: /Users/tim Shell: /bin/csh Last login Tue Jul 12 07:49:59 on tty01 No unread mail No Plan. % Some versions derived from System V Unix display both the last successful login and the last unsuccessful login when a user logs into the system: login: tim
password: books2sell
Last successful login for tim : Tue Jul 12 07:49:59 on tty01
Last unsuccessful login for tim : Tue Jul 06 09:22:10 on tty01
Try to teach your users to check the last login time each time they log in. If the displayed time doesn't correspond to the last time a user used the system, somebody else might have been using his account. If this happens, the user should immediately notify the system administrator. Unfortunately, the design of the lastlog mechanism is such that the previous contents of the file are overwritten at each login. As a result, if a user is inattentive for even a moment, or if the login message clears the screen, the user may not notice a suspicious time. Furthermore, even if a suspicious time is noted, it is no longer available for the system administrator to examine. One way to compensate for this design flaw is to have a cron-spawned task periodically make an on-disk copy of the file that can be examined at a later time. For instance, you could have a shell file run every six hours to do the following: mv /var/log/lastlog.3 /var/log/lastlog.4 mv /var/log/lastlog.2 /var/log/lastlog.3 mv /var/log/lastlog.1 /var/log/lastlog.2 cp /var/log/lastlog /var/log/lastlog.1 This will preserve the contents of the file in six-hour periods. If backups are done every day, then the file will also be preserved in the backups for later examination. If you have saved copies of the lastlog file, you will need a way to read the contents. Unfortunately, there is no utility under standard versions of Unix that allows you to read one of these files and print all the information. Therefore, you need to write your own. The Perl script shown in Example 21-2 will work on Linux systems, and you can modify it to work on others.[9]
Example 21-2. Script that reads lastlog file#!/usr/local/bin/perl
$fname = (shift || "/var/log/lastlog");
setpwent;
while (($name, $junk, $uid) = getpwent) {
$names{$uid} = $name;
}
endpwent;
# Size of the "line" and "host" fields, in bytes.
# These values are for Linux. On Solaris, use 8 and 16, respectively.
$linesize = 32;
$hostsize = 256;
$recordsize = $linesize + $hostsize + 4; # 4 bytes for the time value
$unpacktemplate = "l A$linesize A$hostsize";
open(LASTL, $fname);
for ($uid = 0; read(LASTL, $record, $recordsize); $uid++) {
($time, $line, $host) = unpack($unpacktemplate, $record);
next unless $time;
$host = "($host)" if $host;
($sec, $min, $hour, $mday, $mon, $year) = localtime($time);
$year += 1900;
printf "%-16s %-12s %10s %s\n",
$names{$uid}, $line, "$mday/$mon/$year", $host;
}
close LASTL;
This program starts by checking for a command-line argument (the "shift"); if none is present, it uses the default. Next, it builds an associative array of UIDs to login names. After this initialization, the program reads a record at a time from the lastlog file. Each binary record is then unpacked and decoded. The stored time is decoded into something more understandable, and then the output is printed. While the lastlog file is designed to provide quick access to the last time that a person logged into the system, it does not provide a detailed history recording the use of each account. For that, Unix uses the wtmp log file. 21.1.6 utmp and wtmp FilesUnix keeps track of who is currently logged into the system with a special file called utmp. This is a binary file that contains a record for every active tty line, and generally does not grow to be more than a few kilobytes in length (at the most). It is usually found in /etc, /var/adm, or /var/run. A second file, wtmp, keeps a record of both logins and logouts. This file grows every time a user logs in or logs out, and can grow to be many megabytes in length unless it is pruned. It is usually found in /var/adm or /var/log. In Berkeley-derived versions of Unix, the entries in the utmp and wtmp files contain:
In System V Unix derivatives, the wtmp file is placed in /etc/wtmp and is also used for accounting. The AT&T System V.3.2 utmp and wtmp entries contain:
The extended wtmpx file used by Solaris, IRIX, and other SVR4 Unix operating systems includes the following:[10]
21.1.6.1 Examining the utmp and wtmp filesUnix programs that report the users that are currently logged into the system (who, whodo, w, users , and finger) do so by scanning the /etc/utmp file. The write command checks this file to see if a user is currently logged in, and determines which terminal he is logged in at. The last program, which prints a detailed report of the times of the most recent user logins, does so by scanning the wtmp file. The ps command gives you a more accurate account of who is currently using your system than the who, whodo, users, and finger commands because under some circumstances, users can have processes running without having their usernames appear in the utmp or wtmp files. (For example, a user may have left a program running and then logged out, or used the rsh command instead of rlogin.) However, the commands who, users, and finger have several advantages over ps:
21.1.6.2 The su command and the utmp and wtmp filesWhen you use the su command (see Section 5.3 in Chapter 5), it creates a new process with both the process's real UID and effective UID altered. This gives you the ability to access another user's files, and run programs as the other user. Because su does not change your entry in the utmp or the wtmp files, the finger command will continue to display the account to which you logged in, not the one that you sued to. Many other programs as well may not work properly when used from within a su subshell, as they determine your username from the utmp entry and not from the real or effective UID. Note that different versions of the su command have different options available that allow you to reset your environment, run a different command shell, or otherwise modify the default behavior. One common argument is a simple dash, as in su - user. This form will cause the shell for user to start up as if it were a login shell. Thus, the su command should be used with caution. While it is useful for quick tests, because it does not properly update the utmp and wtmp files, it can cause substantial confusion to other users and to some system utilities. 21.1.6.3 last programEvery time a user logs in or logs out, Unix makes a record in the wtmp file. The last program displays the contents of this file in an understandable form.[11] If you run last with no arguments, the command displays all logins and logouts on every device. last will display the entire file; you can abort the display by pressing the interrupt character (usually Ctrl-C).
% last dpryor ttyp3 std.com Sat Mar 11 12:21 - 12:24 (00:02) simsong ttyp2 204.17.195.43 Sat Mar 11 11:56 - 11:57 (00:00) simsong ttyp1 204.17.195.43 Sat Mar 11 11:37 still logged in dpryor console Wed Mar 8 10:47 - 17:41 (2+06:53) devon console Wed Mar 8 10:43 - 10:47 (00:03) simsong ttyp3 pleasant.cambrid Mon Mar 6 16:27 - 16:28 (00:01) dpryor ftp mac4 Fri Mar 3 16:31 - 16:33 (00:02) dpryor console Fri Mar 3 12:01 - 10:43 (4+22:41) simsong ftp pleasant.cambrid Fri Mar 3 08:40 - 08:56 (00:15) simsong ttyp2 pleasant.cambrid Thu Mar 2 20:08 - 21:08 (00:59) ... In this display, you can see that five login sessions have been active since March 7th: simsong, dpryor, devon, dpyror (again), and simsong (again). Two of the users (dpryor and devon) logged on to the computer console. The main user of this machine is probably the user dpryor. (In fact, this computer is a workstation sitting on dpryor's desk.) The terminal name ftp indicates that dpryor was logged in for FTP file transfer. Other terminal names may also appear here, depending on your system type and configuration; for instance, you might have an entry showing pc-nfs as an entry type. The last command allows you to specify a username or a terminal as an argument to prune the amount of information displayed. If you provide a username, last displays logins and logouts only for that user. If you provide a terminal name, last displays logins and logouts only for the specified terminal. % last dpryor dpryor ttyp3 std.com Sat Mar 11 12:21 - 12:24 (00:02) dpryor console Wed Mar 8 10:47 - 17:41 (2+06:53) dpryor ftp mac4 Fri Mar 3 16:31 - 16:33 (00:02) dpryor console Fri Mar 3 12:01 - 10:43 (4+22:41) dpryor ftp mac4 Mon Feb 27 10:43 - 10:45 (00:01) dpryor ttyp6 std.com Sun Feb 26 01:12 - 01:13 (00:01) dpryor ftp mac4 Thu Feb 23 14:42 - 14:43 (00:01) dpryor ftp mac4 Thu Feb 23 14:20 - 14:25 (00:04) dpryor ttyp3 mac4 Wed Feb 22 13:04 - 13:06 (00:02) dpryor console Tue Feb 21 09:57 - 12:01 (10+02:04) You may wish to issue the last command every morning to see if there were unexpected logins during the previous night. On some systems, the wtmp file also logs shutdowns and reboots. 21.1.6.4 Pruning the wtmp fileThe wtmp file will continue to grow until you have no space left on your computer's hard disk. For this reason, many vendors include shell scripts with their Unix releases that zero the wtmp file automatically on a regular basis (such as once a week or once a month). These scripts are run automatically by the cron program. For example, some monthly shell scripts contain a statement that looks like this: # Zero the log file. cat /dev/null >/var/adm/wtmp Instead of this simple-minded approach, you may wish to make a copy of the wtmp file first, so you'll be able to refer to logins in the previous month. To do so, you must locate the shell script that zeros your log file and add the following lines: # Make a copy of the log file and zero the old one. mv /var/adm/wtmp /var/adm/wtmp.old cp /dev/null /var/adm/wtmp chmod 600 /var/adm/wtmp Most versions of the last command allow you to specify a file to use other than wtmp by using the -f option. For example: % last -f /var/adm/wtmp.old Some versions of the last command do not allow you to specify a different wtmp file to search through. If you need to check this previous copy and you are using one of these systems, you will need to momentarily place the copy of the wtmp file back into its original location. For example, you might use the following shell script to do the trick: #!/bin/sh mv /var/adm/wtmp /var/adm/wtmp.real mv /var/adm/wtmp.old /var/adm/wtmp last $* mv /var/adm/wtmp /var/adm/wtmp.old mv /var/adm/wtmp.real /var/adm/wtmp This approach has a serious problem: any logins and logouts will be logged to the wtmp.old file while the command is running. 21.1.7 loginlog FileIf you are using a System V-based version of Unix (including Solaris), you can log failed login attempts in a special file called /var/adm/loginlog. To log failed login attempts, you must specifically create this file with the following sequence of commands: # touch /var/adm/loginlog # chmod 600 /var/adm/loginlog # chown root /var/adm/loginlog After this file is created, Unix will log all failed login attempts to your system. A "failed login attempt" is defined as a login attempt in which a user tries to log into your system but types a bad password five times in a row. Normally, System V Unix hangs up on the caller (or disconnects the Telnet connection) after the fifth attempt. If this file exists, Unix will also log the fact that five bad attempts occurred. The contents of the file look like this: # cat /var/adm/loginlog simsong:/dev/pts/8:Mon Oct 7 00:42:14 2002 simsong:/dev/pts/8:Mon Oct 7 00:42:20 2002 simsong:/dev/pts/8:Mon Oct 7 00:42:26 2002 simsong:/dev/pts/8:Mon Oct 7 00:42:39 2002 simsong:/dev/pts/8:Mon Oct 7 00:42:50 2002 #  |
|
|
