From the 1999 manuscript of Digital Libraries, by William Arms, (c) 2000 M.I.T. Press.

[ Previous Chapter | Next Chapter | Contents | Index | Glossary ]

Chapter 3
Libraries and publishers

Computing in libraries

Faced with the management of huge collections over long periods of time, libraries have a proud tradition as early adopters of new technology, from microfilm, through online information services and CD-ROMs, to digital libraries. (The second typewriter imported into Brazil was for the National Library of Brazil to type library catalog cards.) The Internet and the web are the latest examples of technology that has been embraced by libraries and is being adapted to their needs, but far from the first.

Mainstream commercial products do not always fit the needs of libraries. The computing industry aims its product developments at the large markets provided by commercial businesses, science, medicine, and the military. In the United States, much of the basic research behind these products has been funded by the Department of Defense or companies such as IBM. Until recently, these large markets had little interest in managing the kinds of information that is found in digital libraries. Consequently, the products sold by the computing industry did not address important needs in information management. Therefore, libraries are accustomed to taking core technology from other fields and tailoring it to their needs. This chapter introduces several such areas, most of which are described more fully in later chapters.

Resource sharing and online catalogs

In libraries, the word "networking" has two meanings. Before modern computer networks, libraries had a wide variety of arrangements to share resources. These include inter-library lending, reciprocal reading privileges between institutions, and the exchange of photocopies. The term given to these arrangements was "networking" or "resource sharing". Resource sharing services, such as the photocopying and document delivery provided by the British Library, enhance the availability of information around the world. However, nothing in resource sharing has had an impact to rival computer networking.

Shared cataloguing was mentioned in Chapter 1 as an early use of computers to support libraries. Almost every library has a catalog with records of the materials in the collections. The catalog has several uses. It helps users find material in the library, it provides bibliographic information about the items, and it is an important tool in managing the collections.

Cataloguing is a area in which librarians use precise terminology, some of which may be unfamiliar to outsiders. Whereas a non-specialist may use the term catalog as a generic term, in a library it has a specific meaning as a collection of bibliographic records created according to strict rules. Rather than the everyday word "book", librarians use the more exact term monograph. Over the years, the information that is included in a monograph catalog record has been codified into cataloguing rules. English speaking countries use the Anglo-American Cataloguing Rules (AACR). The current version is known as AACR2.

The task of cataloguing each monograph is time consuming and requires considerable expertise. To save costs, libraries share catalog records. Major libraries that catalog large numbers of monographs, such as the Library of Congress and the major university libraries, make their catalog records available to others, free of charge. Rather than create duplicate records, most libraries look for an existing catalog record and create their own records only when they can not find another to copy.

Since the late 1960s, this sharing of catalog records has been computer based. Technically, the fundamental tool for distributing catalog records is the MARC format (an abbreviation for Machine-Readable Cataloging). MARC was developed by Henriette Avram and colleagues at the Library of Congress, initially as a format to distribute monograph catalog records on magnetic tape. To be precise, it is necessary to distinguish between AACR2, which provides the cataloguing rules, and MARC, which is a format for representing the resulting catalog records. In practice, the term "MARC cataloguing" is often used in a general sense to cover both the catalog records and the computer format in which they are stored. MARC cataloguing has been expanded far beyond monographs and is now used for most categories of library materials, including serials, archives, manuscripts, and many more. Panel 3.1 gives an example of a MARC record.

The development of MARC led to two important types of computer-based system. The first was shared cataloguing; the pioneer was OCLC, created by Fred Kilgour in 1967. OCLC has a large computer system which has grown to more 35 million catalog records in MARC format, including records received from the Library of Congress. When an OCLC member library acquires a book that it wishes to catalog, it begins by searching the OCLC database. If it finds a MARC record, it downloads the record to its own computer system and records the holding in the OCLC database. In the past it could also have ordered a printed catalog card. This is called "copy cataloguing". If the OCLC database does not contain the record, the library is encouraged to create a record and contribute it to OCLC. With copy cataloguing, each item is catalogued once and the intellectual effort shared among all libraries. MARC cataloguing and OCLC's success in sharing catalog records have been emulated by similar services around the world.

The availability of MARC records stimulated a second development. Individual libraries were are to create online catalogs of their holdings. In most cases, the bulk of the records were obtained from copy cataloguing. Today, almost every substantial library in the United States has its online catalog. Library jargon calls such a catalog an "OPAC" for "online public access catalog". Many libraries have gone to great efforts to convert their old card catalogs to MARC format, so that the online catalog is the record of their entire holdings, rather than having an online catalog for recent acquisitions, but traditional card catalogs for older materials, The retrospective conversion of Harvard University's card catalog to MARC format has recently been completed. Five million cards were converted at a cost approaching $15 million.

A full discussion of MARC cataloguing and online public access catalogs is outside the scope of this book. MARC was an innovative format at a time when most computer systems represented text as fixed length fields with capital letters only. It remains a vital format for libraries, but it is showing its age. Speculation on the future of MARC is complicated by the enormous investment that libraries have made in it. Whatever its future, MARC was a pioneering achievement in the history of both computing and libraries. It is a key format that must be accommodated by digital libraries.

Linking online catalogs and Z39.50

During the 1980s, universities libraries began to connect their online catalogs to networks. As an early example, by 1984 there was a comprehensive campus network at Dartmouth College. Since the computer that held the library catalog was connected to the network, anybody with a terminal or personal computer on-campus could search the catalog. Subsequently, when the campus network was connected to the Internet, the catalog became available to the whole world. People outside the university could search the catalog and discover what items were held in the libraries at Dartmouth. Members of the university could use their own computers to search the catalogs of other universities. This sharing of library catalogs was one of the first, large-scale examples of cooperative sharing of information over the Internet.

In the late 1970s, several bibliographic utilities, including the Library of Congress, the Research Libraries Group, and the Washington Libraries Information Network, began a project known as the Linked Systems Project, which developed the protocol now known by the name of Z39.50. This protocol allows one computer to search for information on another. It is primarily used for searching records in MARC format, but the protocol is flexible and is not restricted to MARC. Technically, Z39.50 specifies rules that allow one computer to search a database on another and retrieve the records that are found by the search. Z39.50 and its role in fostering interoperability among digital libraries are discussed in Chapter 11. It is one of the few protocols to be widely used for interoperation among diverse computer systems.

Abstracts and indexes

Library catalogs are the primary source of information about monographs, but they are less useful for journals. Catalogs provide a single, brief record for an entire run of a journal. This is of little value to somebody who wants to discover individual articles in academic journals. Abstracting and indexing services developed to help researchers to find such information. Typical services are Medline for the biomedical literature, Chemical Abstracts for chemistry, and Inspec for the physical sciences including computing. The services differ in many details, but the basic structures are similar. Professionals, who are knowledgeable about a subject area, read each article from a large number of journals and assign index terms or write abstracts. Sometimes services use index terms that are drawn from a carefully controlled vocabulary, such as the MeSH headings that the National Library of Medicine uses for its Medline service. Others services are less strict. Some generate all their own abstracts. Others, such as Inspec, will use an abstract supplied by the publisher.

Most of these services began as printed volumes that were sold to libraries, but computer searching of these indexes goes back to the days of batch processing and magnetic tape. Today, almost all searching is by computer. Some indexing services run computer systems on which users can search for a fee; others license their data to third parties who provide online services. Many large libraries license the data and mount it on their own computers. In addition, much of the data is available on CD-ROM.

Once the catalog was online, libraries began to mount other data, such as abstracts of articles, indexes, and reference works. These sources of information can be stored in a central computer and the retrieved records displayed on terminals or personal computers. Reference works consisting of short entries are particularly suited for this form of distribution, since users move rapidly from one entry to another and will accept a display that has text characters with simple formatting. Quick retrieval and flexible searching are more important than the aesthetics of the output on the computer screen.

As a typical example, here are some of the many information sources that the library at Carnegie Mellon University provided online during 1997/8.

     Carnegie Mellon library catalog
     Carnegie Mellon journal list
     Bibliographic records of architectural pictures and drawings
     Who's who at CMU
     American Heritage Dictionary
     Periodical Abstracts
     ABI/Inform (business periodicals)
     Inspec (physics, electronics, and computer science)
     Research directory (Carnegie Mellon University)

Several of these online collections provide local information, such as Who's who at CMU, which is the university directory of faculty, students, and staff. Libraries do not provide their patrons only with formally published or academic materials. Public libraries, in particular, are a broad source of information, from tax forms to bus timetables. Full-text indexes and web browsers allow traditional and non-traditional library materials to be combined in a single system with a single user interface. This approach has become so standard that it is hard to realize that only a few years ago merging information from such diverse sources was rare.

Mounting large amounts of information online and keeping it current is expensive. Although hardware costs fall continually, they are still noticeable, but the big costs are in licensing the data and the people who handle both the business aspects and the large data files. To reduce these costs, libraries have formed consortia where one set of online data serves many libraries. The MELVYL system, which serves the campuses of the University of California, was one of the first. It is described in Chapter 5.

Information retrieval

Information retrieval is a central topic for libraries. A user, perhaps a scientist, doctor, or lawyer, is interested in information on some topic and wishes to find the objects in a collection that cover the topic. This requires specialized software. During the mid-1980s, libraries began to install computers with software that allowed full-text searching of large collections. Usually, the MARC records of a library's holdings were the first data to be loaded onto this computer, followed by standard references works. Full-text searching meant that a user could search using any words that appeared in the record and did not need to be knowledgeable about the structures of the records or the rules used to create them.

Research in this field is at least thirty years old, but the basic approach has changed little. A user expresses a request as a query. This may be a single word, such as "cauliflower", a phrase, such as "digital libraries", or a longer query, such as, "In what year did Darwin travel on the Beagle?" The task of information retrieval is to find objects in the collection that match the query. Since a computer does not have the time to go through the entire collection for each search, looking at every object separately, the computer must have an index of some sort that allows information retrieval by looking up entries in indexes.

As computers have grown more powerful and the price of storage declined, methods of information retrieval have moved from carefully controlled searching of short records, such as catalog records or those used by abstracting and indexing services, to searching the full text of every word in large collections. In the early days, expensive computer storage and processing power, stimulated the development of compact methods of storage and efficient algorithms. More recently, web search programs have intensified research into methods for searching large amounts of information which are distributed across many computers. Information retrieval is a topic of Chapters 10 and 11.

Representations of text and SGML

Libraries and publishers share an interest in using computers to represent the full richness of textual materials. Textual documents are more than simple sequences of letters. They can contain special symbols, such as mathematics or musical notation, characters from any language in the world, embedded figures and table, various fonts, and structural elements such as headings, footnotes, and indexes. A desirable way to store a document in a computer is to encode these features and store them with the text, figures, tables, and other content. Such an encoding is called a mark-up language. For several years, organizations with a serious interest in text have been developing a mark-up scheme known as SGML. (The name is an abbreviation for Standard Generalized Markup Language.) HTML, the format for text that is used by the web, is a simple derivative of SGML.

Since the representation of a document in SGML is independent of how it will be used, the same text, defined by its SGML mark-up, can be displayed in many forms and formats: paper, CD-ROM, online text, hypertext, and so on. This makes SGML attractive for publishers who may wish to produce several versions of the same underlying work. A pioneer application in using SGML in this way was the new Oxford English Dictionary. SGML has also been heavily used by scholars in the humanities who find in SGML a method to encode the structure of text that is independent of any specific computer system or method of display. SGML is one of the topics in Chapter 9.

Digital libraries of scientific journals

The early experiments

During the late 1980s several publishers and libraries became interested in building online collections of scientific journals. The technical barriers that had made such projects impossible earlier were disappearing, though still present to some extent. The cost of online storage was coming down, personal computers and networks were being deployed, and good database software was available. The major obstacles to building digital libraries were that academic literature was on paper, not in electronic formats, and that institutions were organized around physical media, not computer networks.

One of the first attempts to create a campus digital library was the Mercury Electronic Library, a project that we undertook at Carnegie Mellon University between 1987 and 1993. Mercury was able to build upon the advanced computing infrastructure at Carnegie Mellon, which included a high-performance network, a fine computer science department, and the tradition of innovation by the university libraries. A slightly later effort was the CORE project at Cornell University to mount images of chemistry journals. Both projects worked with scientific publishers to scan journals and establish collections of online page images. Whereas Mercury set out to build a production system, CORE also emphasized research into user interfaces and other aspects of the system by chemists. The two projects are described in Panel 3.2.

Although both the Mercury and CORE projects converted existing journal articles from print to bit-mapped images, conversion was not seen as the long-term future of scientific libraries. It simply reflected the fact that none of the journal publishers were in a position to provide other formats. Printers had used computer typesetting for many years, but their systems were organized entirely to produce printed materials. The printers' files were in a wide variety of formats. Frequently, proof corrections were held separately and not merged with the master files, so that they were not usable in a digital library without enormous effort.

Mercury and CORE were followed by a number of other projects that explored the use of scanned images of journal articles. One of the best known was Elsevier Science Publishing's Tulip project. For three years, Elsevier provided a group of universities, which included Carnegie Mellon and Cornell, with images from forty three journals in material sciences. Each university, individually mounted these images on their own computers and made them available locally.

Projects such as Mercury, CORE, and Tulip were not long-term production systems. Each had rough edges technically and suffered from the small size of the collection provided to researchers, but they were followed by systems that have become central parts of scientific journal publishing. They demonstrated that the potential benefits of a digital library could indeed be achieved in practice. The next generation of developments in electronic publishing were able to take advantage of much cheaper computer storage allowing large collections of images to be held online. The emergence of the web and the widespread availability of web browsers went a long way towards solving the problem of user interface development. Web browsers are not ideal for a digital library, though they are a good start, but they have the great advantage that they exist for all standard computers and operating systems. No longer does every project have to develop its own user interface programs for every type of computer. Scientific journal publishers

Until the mid-1990s, established publishers of scientific journals hesitated about online publishing. While commercial publishing on CD-ROM had developed into an important part of the industry, few journals were available online. Publishers could not make a business case for online publishing and feared that, if materials were available online, sales of the same materials in print would suffer. By about 1995, it became clear that broad-reaching changes were occurring in how people were using information. Libraries and individuals were expanding their uses of online information services and other forms of electronic information much more rapidly than their use of printed materials. Print publications were competing for a declining portion of essentially static library budgets.

Chapter 6 explains that journal publishing is in the grip of a price spiral with flat or declining circulation. In the past, publishers have used new technology to reduce production costs, thus mitigating some of the problems of declining circulation, but these savings are becoming exhausted. They realize that, if they want to grow their business or even sustain its current level, they need to have products for the expanding online market. Electronic information is seen as a promising growth market.

Most of the major publishers of scientific journals have moved rapidly to electronic publishing of their journals. The approach taken by the Association for Computing Machinery (ACM) is described in Panel 3.4, but, with only minor differences, the same strategy is being followed by the large commercial publishers, including Elsevier, John Wiley, and Academic Press, by societies such as the American Chemical Society, and by university presses, including M.I.T. Press.

Electronic journals

The term electronic journal is commonly used to describe a publication that maintains many of the characteristics of printed journals, but is produced and distributed online. Rather confusingly, the same term is used for a journal that is purely digital, in that it exists only online, and for the digital version of a journal that is primarily a print publication, e.g., the ACM journals described in Panel 3.4.

Many established publishers have introduced a small number of purely online periodicals and there have been numerous efforts by other groups. Some of these online periodical set out to mimic the processes and procedures of traditional journals. Perhaps the most ambitions publication using this approach was the On-line Journal of Current Clinical Trials which was developed by the American Association for the Advancement of Science (AAAS) in conjunction with OCLC. Unlike other publications for which the electronic publication was secondary to a printed publication, this new journal was planned as a high-quality, refereed journal for which the definitive version would be the electronic version. Since the publisher had complete control over the journal, it was possible to design it and store it in a form that was tailored to electronic delivery and display, but the journal was never accepted by researchers or physicians. It came out in 1992 and never achieved its goals, because it failed to attract the numbers of good papers that were planned for. Such is the fate of many pioneers.

More recent electronic periodicals retain some characteristics of traditional journals but experiment with formats or services that take advantage of online publishing. In 1995, we created D-Lib Magazine at CNRI as an online magazine with articles and news about digital libraries research and implementation. The design of D-Lib Magazine illustrates a combination of ideas drawn from conventional publishing and the Internet community. Conventional journals appear in issues, each containing several articles. Some electronic journals publish each article as soon as it has is ready, but D-Lib Magazine publishes a monthly issue with strong emphasis on punctuality and rapid publication. The graphic design is deliberately flexible, while the constraints of print force strict design standards, but online publications can allow authors to be creative in their use of technology.

Research libraries and conversion projects

One of the fundamental tasks of research libraries is to save today's materials to be the long-term memory for tomorrow. The great libraries have wonderful collections that form the raw material of history and of the humanities. These collections consist primarily of printed material or physical artifacts. The development of digital libraries has created great enthusiasm for converting some of these collections to digital forms. Older materials are often in poor physical condition. Making a digital copy preserves the content and provides the library with a version that it can make available to the whole world. This section looks at two of these major conversion efforts.

Many of these digital library projects convert existing paper documents into bit-mapped images. Printed documents are scanned one page at a time. The scanned images are essentially a picture of the page. The page is covered by an imaginary grid. In early experiments this was often at 300 dots per inch, and the page was recorded as an array of black and white dots. More recently, higher resolutions and full color scanning have become common. Bit-mapped images of this kind are crisp enough that they can be displayed on large computer screens or printed on paper with good legibility. Since this process generates a huge number of dots for each page, various methods are used to compress the images which reduces the number of bits to be stored and the size of files to be transmitted across the network, but even the simplest images are 50,000 bytes per page.

Since the driving force in electronic libraries and information services has come from the scientific and technical fields, quite basic needs of other disciplines, such as character sets beyond English, have often been ignored. The humanities have been in danger of being left behind, but a new generation of humanities scholars is embracing computing. Fortunately, they have friends. Panel 3.6 describes JSTOR, a project of the Andrew W. Mellon Foundation which is both saving costs for libraries and bringing important journal literature to a wider audience than would ever be possible without digital libraries.

These two examples, American Memory and JSTOR, are further examples of a theme that has run throughout this chapter. Libraries, by their nature, are conservative organizations. Collections and their catalogs are developed over decades or centuries. New services are introduced cautiously, because they are expected to last for a long time. However, in technical fields, libraries have frequently been adventurous. MARC, OCLC, the Linked Systems Project, Mercury, CORE, and the recent conversion projects may not have invented the technology they used, but the deployment as large-scale, practical systems was pioneering.

Chapter 2 discussed the community that has grown up around the Internet and the web. Many members of this community discovered online information very recently, and act as though digital libraries began in 1993 with the release of Mosaic. As discussed in this chapter, libraries and publishers developed many of the concepts that are the basis for digital libraries, years before the web. Combining the two communities of expertise provides a powerful basis for digital libraries in the future.

[ Previous Chapter | Next Chapter | Contents | Index | Glossary ]

[ Digital Libraries Home | About the Digital Libraries Web site ]

Last revision of content: January 1999
Formatted for the Web: December 2002
(c) Copyright The MIT Press 2000