12.1 Understanding Unix Internet Servers and Services

Most Unix network services are provided by individual programs called servers. For a server to operate, it must be assigned a protocol (e.g., TCP or UDP), be assigned a port number, and somehow be started.

12.1.1 The /etc/services File

As we saw in the last chapter, most Internet services are assigned a specific port for their exclusive use. When a client opens a connection across the network to a server, the client uses the port to specify which service it wishes to use. These ports are called well-known ports because they need to be known in advance by both the client and the server. Unix uses the /etc/services file as a small local database; for each service this file specifies the service's well-known port number and notes whether the service is available as a TCP or UDP service. The /etc/services file is distributed as part of the Unix operating system.

The information in the /etc/services file is derived from Internet RFCs^[1] and other sources. Some of the services listed in the /etc/services file are no longer in widespread use; nevertheless, their names still appear in the file.

^[1] RFC stands for Request For Comment. The RFCs describe many of the standards, proposed standards, and operational characteristics of the Internet. There are many online sources for obtaining the RFCs. The official copies of RFCs are located at http://www.rfc-editor.org/.

The following is an excerpt from the /etc/services file that specifies the ports for the Telnet, SMTP, and Network Time Protocol (NTP) services:

# /etc/services
#
 . . . 
telnet 23/tcp
smtp   25/tcp mail
time   37/udp timeserver
 . . .

Each line gives the canonical name of the service, the port number and protocol, and any aliases for the service name. As you can see, the SMTP service uses TCP on port 25, and also goes by the alias "mail".

12.1.1.1 Calling getservbyname( )

Most Unix servers determine their port numbers by looking up each port in the /etc/services file using the getservbyname ( ) library call. The /etc/services file can be supplemented or replaced by distributed database systems such as NIS, NIS+, Netinfo, DCE, or an LDAP-based service. Most of these distributed databases patch the system's getservbyname ( ) function, so the use of the network database is transparent to applications running on most Unix systems.

Trusted(?) Ports

On Unix systems, TCP and UDP ports in the range 0 to 1023 are sometimes referred to as trusted ports. Unix requires that a process have superuser privileges to be able to start listening for incoming connections on such a port or to originate connections to a remote server using one of these ports as the source port. (Note that any user can connect to a trusted port from an untrusted port.)

Trusted ports were intended to prevent a regular user from obtaining privileged information. For example, if a regular user could write a program that listened to port 23, that program could masquerade as a telnet server, receive connections from unsuspecting users, and obtain their passwords.

This idea of trusted ports is a Unix convention. It is not part of the Internet standard, and manufacturers of other TCP/IP implementations are not bound to observe this protocol. In particular, there are no restrictions that prohibit nonprivileged users and processes on Windows-based machines from originating or accepting connections on so-called trusted ports.

Some network servers bypass the getservbyname ( ) function and simply hardcode the service number into their programs. Others allow a port number to be specified in a configuration file. Still other servers listen simultaneously to several ports! Thus, if you make a change to a program's port number in the /etc/services file, the server may or may not change the port to which it is listening. This can result in significant problems if it becomes necessary to change the port used by a service; fortunately, well-known services seldom change their ports.

12.1.1.2 Ports cannot be trusted

It's important to remember that port assignments are standards, but they are not set in stone. Servers can be run on ports that are unassigned or are assigned to other protocols. This is especially problematic for organizations that wish to block some kinds of protocols from leaving their organizations while allowing others through—if you allow the packets for any specific IP port to travel unrestricted from the inside of your organization to the outside, then a malicious insider can effectively use that hole to tunnel any protocol through your defenses.

For example, because the SSL protocol cannot be effectively proxied, many organizations allow TCP connections on port 443 to travel from inside their organization to the outside. This is because attempts to proxy the SSL protocol are effectively man-in-the-middle attacks and are specifically detected by the SSL protocol. In the Spring of 2001, one of the authors had to spend two days at the offices of a major consulting firm. Their firewall was configured to allow packets through on port 443 but not packets on port 22 (SSH). The reason, allegedly, was "security": the network administrator had made a determination that SSH was too dangerous a protocol to allow from the inside of the organization to the outside. To get around this minor inconvenience, the author simply telephoned a friend and asked him to set up an SSH server running on port 443. A few moments later, the author used the ssh command on his laptop to connect to that remote SSH server. On top of this SSH connection the author tunneled a variety of other protocols, including POP, SMTP, IMAP, HTTP, and X. So much for the restrictive firewall!

Most network analysis tools cannot detect a protocol that is being run on an unexpected port: making this determination requires that each TCP connection be reassembled from the individual IP packets and then analyzed. If the contents are encrypted, even reassembly combined with content analysis may not be sufficient to determine the protocol being used.

12.1.2 Starting the Servers

There are fundamentally two kinds of network servers on Unix systems:

Servers that are always running: These servers are started automatically when the operating system starts up. Servers started at boot time are usually the servers that should provide rapid responses to user requests, must handle many network requests from a single server process, or both. Servers in this category include nfsd (the Network Filesystem daemon), httpd (the Apache web server), and sendmail.
Servers that are run only when needed: These servers are usually started from inetd , the Unix "Internet" daemon, and handle a single request. inetd is a flexible program that can listen to dozens of Internet ports and automatically start the appropriate daemon as needed. Servers started by inetd include popper (the Post Office Protocol daemon) and fingerd (the finger daemon). This greatly reduces the system load if there are many daemons that are infrequently used.

The location for network servers has changed as Unix has evolved. Older systems may keep them in /etc or /usr/etc, but modern Unix systems typically place them in /usr/sbin or /usr/libexec.

12.1.2.1 Startup on different Unix systems

Servers that are always running are usually started by the Unix system at startup. Unfortunately, there are many, many different strategies that different Unix systems use for deciding which servers to launch when the system starts. Old versions of Unix launched servers that were listed in a single shell script, /etc/rc. To provide for local customization, the last line of /etc/rc ran a second shell script, /etc/rc.local, if that script was present.

System V-based systems, including Solaris and Linux, have a complex startup system that uses multiple directories and a variety of run levels. Individual servers are started by scripts located in the /etc/init.d/ and /etc/rcn.d/ directories, in which n is the appropriate run level; servers can be enabled by placing executable scripts in these directories. (More specifically, they are placed in the /etc/init.d directory and linked into the run level directory, where they are run in alphabetical order by filename.)

Modern BSD-based systems start up servers that are located in the /usr/local/etc/rc.d/ directory. Some scripts execute the shell scripts /etc/rc.conf and /etc/defaults/rc.conf; these scripts set shell variables that are used by the startup scripts to determine which daemons should be run.

Mac OS X implements yet another startup system, based on startup packages located in the /System/Library/StartupItems directory.

It is vitally important that you know all of the different ways that processes can be run by your system when it starts up so that you can properly audit your system. People who break into computers frequently leave behind their own network servers or daemons that can be used to retake control of the system at a later point in time. Unfortunately, the power of Unix means that an attacker can easily set up such a server—in some cases, by making a single-line modification to a file on a running system.

12.1.2.2 Startup examples

The lines in an /etc/rc file that start up the Simple Mail Transfer Protocol (SMTP) server might look like this:

if [ -f /usr/lib/sendmail -a -f /etc/sendmail/sendmail.cf ]; then
 /usr/lib/sendmail -bd -q1h && (echo -n ' sendmail') > /dev/console
fi

This example checks for the existence of /usr/lib/sendmail and the program's control file, /etc/sendmail/sendmail.cf. If the two files exist, /etc/rc runs the sendmail program and prints the word sendmail on the system console.

Chapter 12 is what a startup script for sendmail looks like on SuSE Linux, which uses System V-style initialization scripts.

Example 12-1. Sample sendmail startup script

#! /bin/sh
# Copyright (c) 1996-99 SuSE Gmbh Nuernberg, Germany.
#
# Author: Florian La Roche <florian@suse.de>, 1996, 1997
#         Werner Fink <werner@suse.de>, 1996, 1999
#

. /etc/rc.config

test -s /etc/rc.config.d/sendmail.rc.config && \
      . /etc/rc.config.d/sendmail.rc.config

# Determine the base and follow a run-level link name.
base=${0##*/}
link=${base#*[SK][0-9][0-9]}

# Force execution if not called by a run-level directory.
test $link = $base && SMTP=yes
test "$SMTP" = yes || exit 0

# The echo return value for success (defined in /etc/rc.config).
return=$rc_done
case "$1" in
    start)
        echo -n "Initializing SMTP port. (sendmail)"
        startproc /usr/sbin/sendmail -bd -q1h || return=$rc_failed
        echo -e "$return"
        ;;
    stop)
        echo -n "Shutting down SMTP port:"
        killproc -TERM /usr/sbin/sendmail || return=$rc_failed
        echo -e "$return"
        ;;
    restart)
        $0 stop  &&  $0 start  ||  return=$rc_failed
        ;;
    reload)
        echo -n "Reload service sendmail"
        killproc -HUP /usr/sbin/sendmail || return=$rc_failed
        echo -e "$return"
        ;;
status)
        echo -n "Checking for service sendmail: "
        checkproc /usr/sbin/sendmail && echo OK || echo No process
        ;;
    *)
        echo "Usage: $0 {start|stop|status|restart|reload}"
        exit 1
esac

# Inform the caller not only verbosely and set an exit status.
test "$return" = "$rc_done" || exit 1
exit 0

This script is maintained in /etc/init.d/sendmail and symlinked to /etc/rc2.d/S80sendmail and /etc/rc2.d/K20sendmail. During the boot process, when the system enters run level 2, each script in /etc/rc2.d that begins with "S" will be run with the "start" argument. During the shutdown process, scripts beginning with "K" are run with the "stop" argument. On SuSE Linux, the insserv program is used to establish these links automatically.^[2]

^[2] Even among systems that use this kind of boot process, the script paths, the utility for setting up links, and the details of the scripts themselves vary widely from system to system. Consult your system's manual for details.

No matter how sendmail is started, after the program is running, sendmail will bind to TCP/IP port number 25 and listen for connections.^[3] Each time the sendmail program receives a connection, it uses the fork( ) system call to create a new process to handle that connection. The original sendmail process then continues listening for new connections.

^[3] The option -bd makes the sendmail program "be a daemon" while the option -q1h causes the program to process the mail queue every hour.

12.1.3 The inetd Program

Originally, BSD Unix set a different server program running for every network service. As the number of services grew in the mid 1980s, Unix systems started having more and more server programs sleeping in the background, waiting for network connections. Although the servers were sleeping, they nevertheless consumed valuable system resources such as process table entries and swap space. Perhaps more importantly, configuring these servers was somewhat difficult, as each server was started up in a different way and had a different syntax for defining which port they should bind to and which UID they should use when running.

Today's Unix systems use the Internet daemon, inetd, to centralize the handling of lightweight Internet services.^[4] The Internet daemon listens and accepts connections on many network ports at the same time.^[5] When a connection is received, inetd automatically starts up the appropriate TCP-based or UDP-based server running under the appropriate UID. The Internet daemon also simplifies the writing of application-specific daemons themselves, as each daemon can be written so that it reads from the network on standard input and writes back to the network on standard output—no special calls from the Berkeley socket library are required.

^[4] Some Unix systems use an alternative Internet daemon called xinetd. Instead of locating all of its configuration in a single inetd.conf file, xinetd typically requires a separate configuration file for each service in the directory /etc/xinetd.d. If your system uses xinetd, read the manual pages for details on configuration; most of the same issues apply as with inetd.

^[5] inetd uses the bind( ) call to attach itself to many network ports and then uses the select( ) call to determine which of these ports is the one that has received a connection.

The inetd daemon is run at boot time as part of the startup procedure. When inetd starts executing, it examines the contents of the /etc/inetd.conf file to determine which network services it is supposed to manage. The program will reread its configuration file if it is sent a HUP signal (see Appendix B for more details about signals).

A sample inetd.conf file is shown in Example 12-2. Note that in this example, services that are not considered "secure" have been disabled.

Example 12-2. A sample inetd.conf file

# Internet server configuration database
#
ftp       stream tcp nowait root    /usr/sbin/ftpd ftpd
#telnet   stream tcp nowait root    /usr/sbin/telnetd telnetd
#shell    stream tcp nowait root    /usr/sbin/rshd rshd
#login    stream tcp nowait root    /usr/sbin/rlogind rlogind
#exec     stream tcp nowait root    /usr/sbin/rexecd rexecd
#uucp     stream tcp nowait uucp    /usr/sbin/uucpd uucpd
#finger   stream tcp nowait nobody  /usr/sbin/fingerd fingerd
#tftp     dgram  udp wait   nobody  /usr/sbin/tftpd tftpd
#comsat   dgram  udp wait   root    /usr/sbin/comsat comsat
talk      dgram  udp wait   root    /usr/sbin/talkd talkd
ntalk     dgram  udp wait   root    /usr/sbin/ntalkd ntalkd
#echo     stream tcp nowait root    internal
#discard  stream tcp nowait root    internal
#chargen  stream tcp nowait root    internal
#daytime  stream tcp nowait root    internal
#time     stream tcp nowait root    internal
#echo     dgram  udp wait   root    internal
#discard  dgram  udp wait   root    internal
#chargen  dgram  udp wait   root    internal
#daytime  dgram  udp wait   root    internal
#time     dgram  udp wait   root    internal

Each line of the inetd.conf file contains at least six fields, separated by spaces or tabs:

Service name: Specifies the service name that appears in the /etc/services file. inetd uses this name to determine which port number it should listen to. If you are testing a new service or developing your own daemon, you may wish to put that daemon on a nonstandard port. Unfortunately, inetd requires that the service name be a symbolic value such as smtp, rather than a numeric value such as 25.
Socket type: Indicates whether the service expects to communicate via a stream or on a datagram basis.
Protocol type: Indicates whether the service expects to use TCP- or UDP-based communications. TCP is used with stream sockets, while UDP is used with dgram, or datagrams.
Wait/nowait: If the entry is "wait," the server is expected to process all subsequent connections received on the socket. If "nowait" is specified, inetd will fork( ) and exec( ) a new server process for each additional datagram or connection request received. Most UDP services are "wait," while most TCP services are "nowait," although this is not a firm rule. Although some manpages indicate that this field is used only with datagram sockets, the field is actually interpreted for all services.
User: Specifies the UID that the server process will be run as. This can be root (UID 0), daemon (UID 1), nobody (often UID -2 or 65534), or any other user of your system. This field allows server processes to be run with fewer permissions than root to minimize the damage that could be done if a security hole is discovered in a server program.
Command name and arguments: The remaining arguments specify the command name to execute and the arguments passed to the command, starting with argv[0].

Some services, like echo, time, and discard, are listed as "internal." These services are so trivial that they are handled internally by inetd rather than requiring a special program to be run. Although these services are useful for testing, they can also be used for denial of service attacks. You should therefore disable them.

You should routinely check the entries in the /etc/inetd.conf file and verify that you understand why each of the services in the file is being offered to the Internet. Sometimes, when attackers break into systems, they create new services to make future break-ins easier. If you cannot explain why a service is being offered at your site, you may wish to disable it until you know what purpose it serves. In many circumstances, it is better to disable a service that you are not sure about than it is to leave it enabled in an effort to find out who is using it at a later point in time: if somebody is using the service, they are sure to let you know! One easy way to list all of the services that are enabled is:

% grep -v "^#" /etc/inetd.conf
talk    dgram   udp     wait    root    /usr/sbin/tcpd  in.talkd
ntalk   dgram   udp     wait    root    /usr/sbin/tcpd  in.ntalkd
pop-3   stream  tcp     nowait  root    /usr/sbin/tcpd popper -c -C -p 2
auth    stream  tcp     nowait  nobody  /usr/sbin/tcpd identd -o -E -i

Because of the importance of the /etc/inetd.conf file, you may wish to track changes to this file using a source code control system such as RCS or CVS. You may also wish to use a consistency-checking tool such as Tripwire or detached PGP signatures to verify that all changes to the file are authorized and properly recorded.