Chapter 2 
The Internet and the World Wide Web

The Internet

The Internet and the World Wide Web are two of the principal building blocks that are used in the development of digital libraries. This chapter describes their main features. It also discusses the communities of engineers and computer scientists who created and continue to develop them. In contrast, the next chapter focuses on librarians, publishers, and other information professionals. Each chapter includes a discussion of how the community functions, its sense of purpose, the social conventions, priorities, and some of the major achievements.

As its name suggest, the Internet is not a single homogeneous network. It is an interconnected group of independently managed networks. Each network supports the technical standards needed for inter-connection - the TCP/IP family of protocols and a common method for identifying computers - but in many ways the separate networks are very different. The various sections of the Internet use almost every kind of communications channel that can transmit data. They range from fast and reliable to slow and erratic. They are privately owned or operated as public utilities. They are paid for in different ways. The Internet is sometimes called an information highway. A better comparison would be the international transportation system, with everything from airlines to dirt tracks.

Historically, these networks originated in two ways. One line of development was the local area networks that were created to link computers and terminals within a department or an organization. Many of the original concepts came from Xerox's Palo Alto Research Center. In the United States, universities were pioneers in expanding small local networks into campus-wide networks. The second source of network developments were the national networks, known as wide area networks. The best known of these was the ARPAnet, which, by the mid-1980s, linked about 150 computer science research organizations. In addition, there were other major networks, which have been almost forgotten, such as Bitnet in the United States. Their users are now served by the Internet.

Since the early networks used differing technical approaches, linking them together was hard. During the late 1980s the universities and the research communities converged on TCP/IP, the protocols of the ARPAnet, to create the Internet that we know today. A key event was the 1986 decision by the National Science Foundation to build a high-speed network backbone for the United States and to support the development of regional networks connected to it. For this backbone, the foundation decided to build upon the ARPAnet's technical achievements and thus set the standards for the Internet. Meanwhile, campus networks were using the same standards. We built one of the first high-speed networks at Carnegie Mellon, completing the installation in 1986. It accommodated several methods of data transmission, and supported competing network protocols from Digital, Apple, and IBM, but the unifying theme was the use of TCP/IP.

Future historians can argue over the combination of financial, organizational, and technical factors that led to the acceptance of the ARPAnet technical standards. Several companies made major contributions to the development and expansion of the Internet, but the leadership came from two U.S. government organizations: DARPA and the National Science Foundation. Chapter 4 describes the contribution that these organizations continue to make to digital libraries research.

An introduction to Internet technology

The technology of the Internet is the subject of whole books. Most of the details are unimportant to users, but a basic understanding of the technology is useful when designing and using digital libraries. Panel 2.1 is a brief introduction to TCP/IP, the underlying protocols that define the Internet.

Panel 2.1
An introduction to TCP/IP

The two basic protocols that hold the Internet together are known as TCP/IP. The initials TCP and IP are joined together so often that it is easy to forget that they are two separate protocols.


IP, the Internet Protocol, joins together the separate network segments that constitute the Internet. Every computer on the Internet has a unique address, known as an IP address. The address consists of four numbers, each in the range 0 to 255, such as Within a computer these are stored as four bytes. When printed, the convention is to separate them with periods as in this example. IP, the Internet Protocol, enables any computer on the Internet to dispatch a message to any other, using the IP address. The various parts of the Internet are connected by specialized computers, known as "routers". As their name implies, routers use the IP address to route each message on the next stage of the journey to its destination.

Messages on the Internet are transmitted as short packets, typically a few hundred bytes in length. A router simply receives a packet from one segment of the network and dispatches it on its way. An IP router has no way of knowing whether the packet ever reaches its ultimate destination.


Users of the network are rarely interested in individual packets or network segments. They need reliable delivery of complete messages from one computer to another. This is the function of TCP, the Transport Control Protocol. On the sending computer, an application program passes a message to the local TCP software. TCP takes the message, divides it into packets, labels each with the destination IP address and a sequence number, and sends them out on the network. At the receiving computer, each packet is acknowledged when received. The packets are reassembled into a single message and handed over to an application program.

This protocol should be invisible to the user of a digital library, but the responsiveness of the network is greatly influenced by the protocol and this often affects the performance that users see. Not all packets arrive successfully. A router that is overloaded may simply ignore some packets. This is called, "dropping a packet." If this happens, the sending computer never receives an acknowledgment. Eventually it gets tired of waiting and sends the packet again. This is known as a "time-out" and may be perceived by the user as an annoying delay.

Other protocols

TCP guarantees error-free delivery of messages, but it does not guarantee that they will be delivered punctually. Sometimes, punctuality is more important than complete accuracy. Suppose one computer is transmitting a stream of audio that another is playing immediately on arrival. If an occasional packet fails to arrive on time, the human ear would much prefer to lose tiny sections of the sound track rather than wait for a missing packet to be retransmitted, which would be horribly jerky. Since TCP is unsuitable for such applications, they use an alternate protocol, named UDP, which also runs over IP. With UDP, the sending computer sends out a sequence of packets, hoping that they will arrive. The protocol does its best, but makes no guarantee that any packets ever arrive.

Panel 2.1 introduced the Internet addresses, known as IP addresses. Another way to identify a computer on the Internet is to give it a name such as Names of this form are known as domain names and the system that relates domain names to IP addresses is known as the domain name system or DNS. Domain names and their role in digital libraries are studied in Chapter 12.

Computers that supports the TCP/IP protocols usually provide a standard set of basic applications. These applications are known as the TCP/IP suite. Some of the most commonly used are listed in Panel 2.2.

Panel 2.2
The TCP/IP suite

The TCP/IP suite is a group of computer programs, based on TCP/IP, that are provided by most modern computers. They include the following.

Terminal emulation. Telnet is a program that allows a personal computer to emulate an old-fashioned computer terminal that has no processing power of its own but relies on a remote computer for processing. Since it provides a lowest common denominator of user interface, telnet is frequently used for system administration.

File transfer. The basic protocol for moving files from one computer to another across the Internet is FTP (the file transfer protocol). Since FTP was designed to make use of TCP it is an effective way to move large files across the Internet.

Electronic mail. Internet mail uses a protocol known as Simple Mail Transport Protocol (SMTP). This is the protocol that turned electronic mail from a collection of local services to a single, world-wide service. It provides a basic mechanism for delivering mail. In recent years, a series of extensions have been made to allow messages to include wider character sets, permit multi-media mail, and support the attachment of files to mail messages.

The Internet community

The technology of the Internet is important, but the details of the technology may be less important than the community of people who developed it and continue to enhance it. The Internet pioneered the concept of open standards. In 1997, Vinton Cerf and Robert Kahn received the National Medal of Technology for their contributions to the Internet. The citation praised their work on the TCP/IP family of protocols, but it also noted that they, "pioneered not just a technology, but also an economical and efficient way to transfer that technology. They steadfastly maintained that their internetworking protocols would be freely available to anyone. TCP/IP was deliberately designed to be vendor-independent to support networking across all lines of computers and all forms of transmission."

The Internet tradition emphasizes collaboration and, even now, the continuing development of the Internet remains firmly in the hands of engineers. Some people seem unable to accept that the U.S. government is capable of anything worthwhile, but the creation of the Internet was led by government agencies, often against strong resistance by companies who now profit from its success. Recently, attempts have been made to rewrite the history of the Internet to advance vested interest, and for individuals to claim responsibility for achievements that many shared. The contrast is striking between the coherence of the Internet, led by far-sighted government officials, and the mess of incompatible standards in areas left to commercial competition, such as mobile telephones.

An important characteristic of the Internet is that the engineers and computer scientists who develop and operate it are heavy users of their own technology. They communicate by e-mail, dismissing conventional mail as "snail mail." When they write a paper they compose it at their own computer. If it is a web page, they insert the mark-up tags themselves, rather than use a formatting program. Senior computer scientists may spend more time preparing public presentations than writing computer programs, but programming is the basic skill that everybody is expected to have.

Panel 2.3
NetNews or Usenet

The NetNews bulletin boards, also known as Usenet, are an important and revealing examples of the Internet community's approach to the open distribution of information. Thousands of bulletin boards, called news groups, are organized in a series of hierarchies. The highest level groupings include "comp", for computer-related information, "rec" for recreational information, the notorious "alt" group, and many more. Thus "" is a bulletin board for discussing musicals. (The British spelling of "theatre" suggests the origin of this news group.)

The NetNews system is so decentralized that nobody has a comprehensive list of all the news groups. An individual who wishes to post a message to a group sends it to the local news host. This passes it to its neighbors, who pass it to their neighbors and so on.

If we consider a digital library to contain managed information, NetNews is the exact opposite. It is totally unmanaged information. There are essentially no restrictions on who can post or what they can post. At its worst the system distributes libel, hate, pornography, and simply wrong information, but many news groups work remarkably well. For example, people around the world who use the Python programming language have a news group, "comp.lang.python", where they exchange technical information, pose queries, and communicate with the developer.

Scientific publishing on the Internet

The publishing of serious academic materials on the Internet goes back many years. Panels 2.4 and 2.5 describe two important examples, the Internet RFC series and the Physics E-Print Archives at the Los Alamos National Laboratory. Both are poorly named. The letters "RFC" once stood for "Request for Comment", but the RFC series is now the definitive technical series for the Internet. It includes a variety of technical information and the formal Internet standards. The Los Alamos service is not an archive in the usual sense. Its primary function is as a pre-print server, a site where researchers can publish research as soon as it is complete, without the delays of conventional journal publishing.

Whatever the merits of their names, these two services are of fundamental importance for publishing research in their respective fields. They are important also because they demonstrate that the best uses of digital libraries may be new ways of doing things. One of the articles of faith within scholarly publishing is that quality can be achieved only by peer review, the process by which every article is read by other specialists before publication. The process by which Internet drafts become RFCs is an intense form of peer review, but it takes place after a draft of the paper has been officially posted. The Los Alamos service has no review process. Yet both have proved to be highly effective methods of scientific communication.

Economically they are also interesting. Both services are completely open to the user. They are professionally run with substantial budgets, but no charges are made for authors who provide information to the service or to readers who access the information. Chapter 6 looks more deeply at the economic models behind the services and the implications for other scientific publishing.

The services were both well-established before the emergence of the web. The web has been so successful that many people forget that there are other effective ways to distribute information on the Internet. Both the Los Alamos archives and the RFC series now use web methods, but they were originally built on electronic mail and file transfer.

Panel 2.4
The Internet Engineering Task Force and the RFC series

The Internet Engineering Task Force

The Internet Engineering Task Force (IETF) is the body that coordinates technical aspects of the Internet. Its methods of working are unique, yet it has proved extraordinarily able to get large numbers of people, many from competing companies, to work together. The first unusual feature is that the IETF is open to everybody. Anybody can go to the regular meetings, join the working groups, and vote.

The basic principle of operation is "rough consensus and working code". Anybody who wishes to propose a new protocol or other technical advance is encouraged to provide a technical paper, called an Internet Draft, and a reference implementation of the concept. The reference implementation should be software that is openly available to all. At the working group meetings, the Internet Drafts are discussed. If there is a consensus to go ahead, then they can join the RFC standards track. No draft standard can become a formal standard until there are implementations of the specification, usually computer programs, openly available for everybody to use.

The IETF began in the United States, but now acts internationally. Every year, one meeting is outside the U.S.. The participants, including the working group leaders, come from around the world. The IETF was originally funded by U.S. government grants, but it is now self-sufficient. The costs are covered by meeting fees.

The IETF as a community

The processes of the IETF are open to everybody who wishes to contribute. Unlike some other standards bodies, whose working drafts are hard to obtain and whose final standards are complex and expensive, all Internet Drafts and RFCs are openly available. Because of the emphasis on working software, when there is rivalry between two technical approaches, the first to be demonstrated with software that actually works has a high chance of acceptance. As a result, the core Internet standards are remarkably simple.

In recent years, IETF meetings have grown to more than two thousand people, but, because they divide into working groups interested in specific topics, the basic feeling of intimacy remains. Almost everybody is a practicing engineer or computer scientists. The managers stay at home. The formal meetings are short and informal; the informal meetings are long and intense. Many an important specification came from a late evening session at the IETF, with people from competing organizations working together.

Because of its rapid growth, the Internet is in continually danger of breaking down technically. The IETF is the fundamental reason that it shows so much resilience. If a single company controlled the Internet, the technology would be as good as the company's senior engineering staff. Because the IETF looks after the Internet technology, whenever challenges are anticipated, the world's best engineers combine to solve them.

Internet Drafts

Internet Drafts are a remarkable series of technical publications. In science and engineering, most information goes out of date rapidly, but journals sit on library shelves for ever. Internet Drafts are the opposite. Each begins with a fixed statement that includes the words:

"Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as work in progress."

The IETF posts online every Internet Draft that is submitted, and notifies interested people through mailing lists. Then the review begins. Individuals post their comments on the relevant mailing list. Comments range from detailed suggestions to biting criticism. By the time that the working group comes to discuss a proposal, it has been subjected to public review by the experts in the field.

The RFC series

The RFC series are the official publications of the IETF. These few thousand publications form a series that goes back almost thirty years. They are the heart of the documentation of the Internet. The best known RFCs are the standards track. They include the formal specification of each version of the IP protocol, Internet mail, components of the World Wide Web, and many more. Other types of RFC include informational RFCs, which publish technical information relevant to the Internet.

Discussions of scientific publishing rarely mention the RFC series, yet it is hard to find another set of scientific or engineering publications that are so heavily reviewed before publication, or are so widely read by the experts in the field. RFCs have never been published on paper. Originally they were available over the Internet by FTP, more recently by the web, but they still have a basic text-only format, so that everybody can read them whatever computing system they use.


Panel 2.5
The Los Alamos E-Print Archives

The Physics E-print Archives are an intriguing example of practicing scientists taking advantage of the Internet technology to create a new form of scientific communication. They use the Internet to extend the custom of circulating pre-prints of research papers. The first archive was established in1991, by Paul Ginsparg of Los Alamos National Laboratory, to serve the needs of a group of high-energy physicists. Subsequent archives were created for other branches of physics, mathematics, and related disciplines. In 1996, Ginsparg reported, "These archives now serve over 35,000 users worldwide from over 70 countries, and process more than 70,000 electronic transactions per day. In some fields of physics, they have already supplanted traditional research journals as conveyers of both topical and archival research information."

The operation of the archives

The primary function of the archives is for scientists to present research results. Frequently, but not always, these are papers that will be published in traditional journals later. Papers for the archives are prepared in the usual manner. Many physicists use the TeX format, which has particularly good support for mathematics, but other formats, such as PostScript, or HMTL are also used. Graphs and data can be embedded in the text or provided as separate files.

The actual submission of a paper to an archive can be by electronic mail, file transfer using the FTP protocol, or via a web form. In each case, the author provides the paper, a short abstract, and a standard set of indexing metadata. Processing by the archive is entirely automatic. The archives provide a search service based on electronic mail, a web-based search system, and a notification service to subscribers, which also uses electronic mail. The search options include searching one or many of the archives. Readers can search by author and title, or search the full text of the abstracts.

As this brief description shows, the technology of the archives is straightforward. They use the standard formats, protocols, and networking tools that researchers know and understand. The user interfaces have been designed to minimize the effort required to maintain the archive. Authors and readers are expected to assist by installing appropriate software on their own computers, and by following the specified procedures.

The economic model

This is an open access system. The cost of maintaining the service is funded through annual grants from the National Science Foundation and the Department of Energy. Authors retain copyright in their papers. Indeed, about the only mention of copyright in the archive instructions is buried in a page of disclaimers and acknowledgments.

As Ginsparg writes, "Many of the lessons learned from these systems should carry over to other fields of scholarly publication, i.e. those wherein authors are writing not for direct financial remuneration in the form of royalties, but rather primarily to communicate information (for the advancement of knowledge, with attendant benefits to their careers and professional reputations)."

The World Wide Web

The World Wide Web, or "the web" as it is colloquially called, has been one of the great successes in the history of computing. It ranks with the development of word processors and spread sheets as a definitive application of computing. The web and its associated technology have been crucial to the rapid growth of digital libraries. This section gives a basic overview of the web and its underlying technology. More details of specific aspects are spread throughout the book.

The web is a linked collection of information on many computers on the Internet around the world. These computers are called web servers. Some of web servers and the information on them are maintained by individuals, some by small groups, perhaps at universities and research centers, others are large corporate information services. Some sites are consciously organized as digital libraries. Other excellent sites are managed by people who would not consider themselves librarians or publishers. Some web servers have substantial collections of high-quality information. Others are used for short term or private purposes, are informally managed, or are used for purposes such as marketing that are outside the scope of libraries.

The web technology was developed about 1990 by Tim Berners-Lee and colleagues at CERN, the European research center for high-energy physics in Switzerland. It was made popular by the creation of a user interface, known as Mosaic, which was developed by Marc Andreessen and others at the University of Illinois, Urbana-Champaign. Mosaic was released in 1993. Within a few years, numerous commercial versions of Mosaic followed. The most widely used are the Netscape Navigator and Microsoft's Internet Explorer. These user interfaces are called web browsers, or simply browsers.

The basic reason for the success of the web can be summarized succinctly. It provides a convenient way to distribute information over the Internet. Individuals can publish information and users can access that information by themselves, with no training and no help from outsiders. A small amount of computer knowledge is needed to establish a web site. Next to none is needed to use a browser to access the information.

The introduction of the web was a grass roots phenomenon. Not only is the technology simple, the manner in which it was released to the public avoided almost all barriers to its use. Carnegie Mellon's experience was typical. Individuals copied the web software over the Internet onto private computers. They loaded information that interested them and made it accessible to others. Their computers were already connected to the campus network and hence to the Internet. Within six months of the first release of Mosaic, three individuals had established major collections of academic information, covering statistics, English, and environmental studies. Since the Internet covers the world, huge numbers of people had immediate access to this information. Subsequently, the university adopted the web officially, with a carefully designed home page and information about the university, but only after individuals had shown the way.

One reason that individuals have been able to experiment with the web is that web software has always been available at no charge over the Internet. CERN and the University of Illinois set the tradition with open distribution of their software for web servers and user interfaces. The most widely used web servers today is a no-cost version of the Illinois web server, known as Apache. The availability of software that is distributed openly over the Internet provides a great stimulus in gaining acceptance for new technology. Most technical people enjoy experimentation but dislike bureaucracy. The joy goes out of an experiment if they need to issue a purchase order or ask a manager to sign a software license.

Another reason for the instant success of the web was that the technology provides gateways to information that was not created specifically for the web. The browsers are designed around the web protocol called HTTP, but the browsers also support other Internet protocols, such as file transfer (FTP), Net News, and electronic mail. Support for Gopher and WAIS, two other protocols which are now almost obsolete, allowed earlier collections of information to coexist with the first web sites. Another mechanism, the Common Gateway Interface (CGI), allows browsers to bridge the gap between the web and any other system for storing online information. In this way, large amounts of information were available as soon as Mosaic became available.

From the first release of Mosaic, the leading browsers have been available for the most common types of computer - the various versions of Windows, Macintosh and Unix - and browsers are now provided for all standard computers. The administrator of a web site can be confident that users around the world will see the information provided on the site in roughly the same format, whatever computers they have.

The technology of the web

Technically, the web is based on four simple techniques. They are: the Hyper-Text Mark-up Language (HTML), the Hyper-Text Transport Protocol (HTTP), MIME data types, and Uniform Resource Locators (URLs). Each of these concepts is introduced below and discussed further in later chapters. Each has importance that goes beyond the web into the general field of interoperability of digital libraries.


HTML is a language for describing the structure and appearance of text documents. Panel 2.6 shows a simple HTML file and how a typical browser might display, or render, it.

Panel 2.6
An HTML example

An HTML file

Here is a simple text file in HTML format as it would be stored in a computer. It shows the use of tags to define the structure and format of a document.


<h1>D-Lib Magazine</h1>
<img src = "logo.gif"> 

<p>Since the first issue appeared in July 1995,
<a href = "">D-Lib Magazine</a> has appeared monthly as a compendium of research, news, and progress in digital libraries.</p> 

<p><i>William Y. Arms 
<br>January 1, 1999</i></p>


The document displayed by a web browser

When displayed by a browser, this document might be rendered as follows. The exact format depends upon the specific browser, computer, and choice of options.

As this example shows, the HTML file contains both the text to be rendered and codes, known as tags, that describe the format or structure. The HMTL tags can always be recognized by the angle brackets (< and >). Most HTML tags are in pairs with a "/" indicating the end of a pair. Thus <title> and </title> enclose some text that is interpreted as a title. Some of the HTML tags show format; thus <i> and </i> enclose text to be rendered in italic, and <br> shows a line break. Other tags show structure: <p> and </p> delimit a paragraph, and <h1> and </h1> bracket a level one heading. Structural tags do not specify the format, which is left to the browser. 

For example, many browsers show the beginning of a paragraph by inserting a blank line, but this is a stylistic convention determined by the browser. This example also shows two features that are special to HMTL and have been vital to the success of the web. The first special feature is the ease of including color image in web pages. The tag: 

     <img src = "logo.gif"> 

is an instruction to insert an image that is stored in a separate file. The abbreviation "img" stands for "image" and "src" for "source". The string that follows is the name of the file in which the image is stored. The introduction of this simple command by Mosaic brought color images to the Internet. Before the web, Internet applications were drab. Common applications used unformatted text with no images. The web was the first, widely used system to combine formatted text and color images. Suddenly the Internet came alive. 

The second and even more important feature is the use of hyperlinks. Web pages do not stand alone. They can link to other pages anywhere on the Internet. In this example, there is one hyperlink, the tag:

     <a href = ""> 

This tag is followed by a string of text terminated by </a>. When displayed by as browser, as in the panel, the text string is highlighted; usually it is printed in blue and underlined. The convention is simple. If something is underlined in blue, the user can click on it and the hyperlink will be executed. This convention is easy for the both the user and the creator of the web page. In this example, the link is to an HTML page on another computer, the home page of D-Lib Magazine

The basic concepts of HTML can be described in a short tutorial and absorbed quickly, but even simple tagging can create an attractive document. In the early days of the web, everybody was self-taught. A useful characteristic of HTML is that its syntax is forgiving of small mistakes. Other computing languages have strict syntax. Omit a semi-colon in a computer program and the program fails or gives the wrong result. With HTML, if the mark-up is more or less right, most browsers will usually accept it. The simplicity has the added benefit that computers programs to interpret HTML and display web pages are straightforward to write.

Uniform Resource Locator (URL)

The second key component of the web is the Uniform Resource Locator, known as a URL. URLs are ugly to look at but are flexible. They provide a simple addressing mechanism that allows the web to link information on computers all over the world. A simple URL is contained in the HTML example in Panel 2.6:

This URL has three parts:

     http is the name of a protocol is the domain name of a computer 
     dlib.html is a file on that computer

Thus an informal interpretation of this URL is, "Using the HTTP protocol, connect to the computer with the Internet address, and access the file dlib.html."


In computing, a protocol is a set of rules that are used to send messages between computer systems. A typical protocol includes description of the formats to be used, the various messages, the sequences in which they should be sent, appropriate responses, error conditions, and so on. HTTP is the protocol that is used to send messages between web browsers and web servers.

The basic message type in HTTP is get. For example, clicking on the hyperlink with the URL:

specifies an HTTP get command. An informal description of this command is:

MIME types

A file of data in a computer is simply a set of bits, but, to be useful the bits need to be interpreted. Thus, in the previous example, in order to display the file "dlib.html" correctly, the browser must know that it is in the HTML format. The interpretation depends upon the data type of the file. Common data types are "html" for a file of text that is marked-up in HMTL format, and "jpeg" for a file that represents an image encoded in the jpeg format.

In the web, and in a wide variety of Internet applications, the data type is specified by a scheme called MIME. (The official name is Internet Media Types.) MIME was originally developed to describe information sent by electronic mail. It uses a two part encoding, a generic part and a specific part. Thus text/ascii is the MIME type for text encoded in ASCII, image/jpeg is the type for an image in the jpeg format, and text/html is text marked-up with HMTL tags. As described in Chapter 12, there is a standard set of MIME types that are used by numerous computer programs, and additional data types can be described using experimental tags.

The importance of MIME types in the web is that the data transmitted by an HTTP get command has a MIME type associated with it. Thus the file "dlib.html" has the MIME type text/html. When the browser receives a file of this type, it knows that the appropriate way to handle this file is to render it as HTML text and display it in the screen.

Many computer systems use file names as a crude method of recording data types. Thus some Windows programs use file names that end in ".htm" for file of HMTL data and Unix computers use ".html" for the same purpose. MIME types are a more flexible and systematic method to record and transmit typed data.

Information on the web

The description of the web so far has been technical. These simple components can be used to create many applications, only some of which can be considered digital libraries. For example, many companies have a web site to describe the organization, with information about products and services. Purchases, such as airline tickets, can be made from web servers. Associations provide information to their members. Private individuals have their own web sites, or personal home pages. Research results may be reported first on the web. Some web sites clearly meet the definition of a digital library, as managed collections of diverse information. Others do not. Many web sites meet the definition of junk.

In aggregate, however, these sites are all important for the development of digital libraries. They are responsible for technical developments beyond the simple building blocks described above, for experimentation, for the establishments of conventions for organizing materials, for the increasingly high quality of graphical design, and for the large number of skilled creators, users, and webmasters. The size of the web has stimulated numerous companies to develop products, many of which are now being used to build digital libraries. Less fortunately, the success of all these web sites frequently overloads sections of the Internet, and has generated social and legal concerns about abusive behavior on the Internet.

Panel 2.7
The World Wide Web Consortium

No central organization controls the web, but there is a need for agreement on the basic protocols, formats, and practices, so that the independent computer systems can interoperate. In 1994, recognizing this need, the Massachusetts Institute of Technology created the World Wide Web Consortium (W3C) and hired Tim Berners-Lee, the creator of the web, as its director. Subsequently, MIT added international partners at the Institut National de Recherche en Informatique et en Automatique in France and the Keio University Shonan Fujisawa Campus in Japan. W3C is funded by member organizations who include most of the larger companies who develop web browsers, servers, and related products.

W3C is a neutral forum in which organizations work together on common specifications for the web. It works through a series of conferences, workshops, and design processes. It provides a collection of information about the web for developers and users, especially specifications about the web with sample code that helps to promote standards. In some areas, W3C works closely with the Internet Engineering Task Force to promulgate standards for basic web technology, such as HTTP, HTML, and URLs.

By acting as a neutral body in a field that is now dominated by fiercely competing companies, W3C's power depends upon its ability to influence. One of its greatest successes was the rapid development of an industry standard for rating content, known as PICS. This was a response to political worries in the United States about pornography and other undesirable content being accessible by minors. More recently it has been active in the development of the XML mark-up language.

Companies such as Microsoft and Netscape sometimes believe that they gain by supplying products that have non-standard features, but these features are a barrier to the homogeneity of the web. W3C deserves much of the credit for the reasonably coherent way that the web technology continues to be developed.


The first web sites were created by individuals, using whatever arrangement of the information they considered most appropriate. Soon, conventions began to emerge about how to organize materials. Hyperlinks permit an indefinite variety of arrangements of information on web sites. Users, however, navigate most effectively through familiar structures. Therefore these conventions are of great importance in the design of digital libraries that are build upon the web. The conventions never went through any standardization body, but their widespread adoption adds a coherence to the web that is not inherent in the basic technology.

The web as a digital library

Some people talk about the web technology as though it were an inferior stop-gap until proper digital libraries are created. One reason for this attitude is that members of other professions having difficulty in accepting that definitive work in digital libraries was carried out by physicists at a laboratory in Switzerland, rather than by well-known librarians or computer scientists. But the web is not a detour to follow until the real digital libraries come along. It is a giant step to build on.

People who are unfamiliar with the online collections also make derogatory statements about the information on the web. The two most common complaints are that the information is of poor quality and that it is impossible to find it. Both complaints have some validity, but are far from the full truth. There is an enormous amount of material on the web; much of the content is indeed of little value, but many of the web servers are maintained conscientiously, with information of the highest quality. Finding information on the web can be difficult, but tools and services exist that enable a user, with a little ingenuity, to discover most of the information that is out there.

Today's web, however, is a beginning, not the end. The simplifying assumptions behind the technology are brilliant, but these same simplifications are also limitations. The web of today provides a base to build the digital libraries of tomorrow. This requires better collections, better services, and better underlying technology. Much of the current research in digital libraries can be seen as extending the basic building blocks of the web. We can expect that, twenty five years from now, digital libraries will be very different; it will be hard to recall the early days of the web. The names "Internet" and "web" may be history or may be applied to systems that are unrecognizable as descendants of the originals. Digital libraries will absorb materials and technology from many places. For the next few years, however, we can expect to see the Internet and the web as the basis on which the libraries of the future are being built. Just as the crude software on early personal computers has developed into modern operating systems, the web can become the foundation for many generations of digital libraries.

Last revision of content: January 1999
Formatted for the Web: December 2002
(c) Copyright The MIT Press 2000