Template

DEVELOPMENT OF AUTOMATION OF LIBRARY PROCESSES IN THE UNITED STATES: Historical Review

Robert M. Hayes

Graduate School of Library and Information Science
University of California at Los Angeles
Los Angeles, CA, USA
E-mail: rhayes@ucla.edu

In this brief summary, I am going to discuss seven timelines over the past 50 years, dividing them into three time periods, each of about 16 to 17 years: 1948-1964, 1964-1980, 1980-1997. The seven timelines are:

This period started with the seminal article, "As we may think", written by Vannevar Bush and published in the Atlantic Monthly in 1948. In a very real sense, it foretold virtually everything we have since seen in the developments that I will outline. It ended with the full-scale launching of the developments most relevant to the use of automation in libraries.

During this period, there was a continuing emphasis on the needs for information support for U.S. national interests. From the time of Vannevar Bush, as the Science Advisor to the President, the needs for information in science and technology also provided a continuing rationale for development of automated information management systems. The Science Advisor to the President became an advocate for the needs for science information. The National Science Foundation launched its Office of Science Information Service to support the development and implementation of automated systems and to support the necessary research; of special importance was its funding of the automation of Chemical Abstracts Services. The Department of Commerce established NTIA as one means to improve dissemination of scientific and technical information. The National Library of Medicine launched its efforts to automate the production of Index Medicus.

In this period, there was at best limited recognition of the role of information in the society of the United States. Its use within companies was solely for the purpose of internal accounting, and the amount of data involved was low. The number of transactions handled, while large in terms of the processing capabilities of the time, was tiny compared with the situation today. And there was at best limited use, if any, of external information, although industrial special libraries did serve as means for access to it.

For the consumer, information was a very little used commodity. Movies were the dominant means for entertainment, and television was just beginning to pervade our life.

It is with a sense almost of shock that I recall the nature of the technology during the 16-year period, 1948 to 1964, and compare it with what we have today. At the beginning of it, data processing meant punched-cards and key-operated accounting machines, with punched tape (like teletype tape) as the "common language" of communication among machines. Computers were limited in capabilities, in numbers, and in applications.

The first small-scale computers for application in business and similar operations (among which I would include libraries) appeared in the early 1950s, and larger systems -- "main-frame" computers -- became wide spread by the end of the period. But all of that equipment suffered from the lack of adequate means for input, storage, display, and output of data. Punched cards and punched tape were the only means for input, and they operated at data rates roughly equivalent to 10 characters per second -- the speed of teletype! Vacuum tubes (cathode ray tubes or CRTs, as they were called) and then, later, magnetic cores were the means for storage of operating programs and data while immediately being processed, but both were exceptionally expensive and limited in capacity. For large-scale data storage, we had magnetic tapes and magnetic drums -- each slow and with inherent limitations in the ways they could be used. For display, we had the most primitive of CRT units, with low resolution and presenting only limited amounts of data. And the means for output were punched card tabulators, with upper-case only fonts and operating again at the equivalent of 10 characters per second. Thus, even though the computers were fast, there wasn't much they could do, given the small internal memory; they were inherently limited by the capabilities for storage of large files and the slow speed of input and output.

During this period, publishing was much as it always had been. Composition was essentially manual; though there were an few, isolated experiments with use of technology, they were completely outside the mainstream of commercial publishing.

The form of publication was simply print, using the traditional means for doing so. Xerographic means for duplication was just beginning to have an impact toward the end of this period.

Where then were libraries during this period? Indeed, there was increasing recognition of the importance of libraries, both as part of the activities to which I have referred and independent of them. The Library of Congress had for many years been the primary center for production and distribution of catalog data, through its Card Production Service.

Most significant was the fact that during this period, Congress passed the Library Services Act (later expanded into the Library Services and Construction Act) which, among other things, fostered the creation of library networks that have been of vital importance as automated systems became important to libraries. Of similar importance, Congress created the National Library of Medicine and the National Agricultural Library.

Given the requirements for information storage and access, computers were seen as a potential means for meeting them. Despite the limitations during this period, the expectations were that the technologies would steadily improve, so there were many efforts to solve the technical and theoretical problems in this field.

The support from NSF's OSIS to these developments, especially that to the Chemical Abstracts Service, to which I have alluded already, was crucial. But for the library field, there was another even more vital source for funding -- the Council on Library Resources. It played crucial roles in the successive stages in development of library automation, but during this first period, they were of special importance in providing support to the National Library of Medicine in its effort to automate the production of Index Medicus. More generally, they were concerned with alternative means for producing catalogs and making catalog data available.

In that respect, one of the important attempts to apply computers to access to catalog data was the production of book-form catalogs, especially for union catalogs. While those efforts were essentially dead-ends in the context of later developments, they were important steps in the general progress.

The final thread in the set of timelines is the development of automation for internal, technical processing in libraries. During this period, progress in this respect was essentially nil. Oh, there were many abortive efforts to apply punched-card equipment to management of circulation records. There were similarly abortive efforts to deal with serial records, again using punched-card equipment. But for the core technical service functions -- acquisition and cataloging, aside from book-form catalog production, there were not even any abortive efforts; the problems were too great, especially with respect to the number of catalog entries involved and the overwhelming costs in converting them to machine-processible form.

During this period something fundamental and ultimately revolutionary became a driving force -- the transition of the United States and, at a slower pace, other developed countries from being "industrial" economies to being "information" economies. Today we see the impact of that transition in its effect on every component of the U.S. economy and U.S. society, but it was during this period that the changes began to become evident. The report by Marc Porat, The Information Economy, clearly identified what was happening and pointed out the fact that by the mid-1970s over fifty percent of the U.S. national workforce was engaged in information work. Today it is far more even than that.

It is important to note that, although many identify the appearance of the information economy as being created by and driven by information technology, in my view the cause is much more fundamental. It is the imperative in development of the society that has created what we see, and the technology merely feeds and serves that imperative, making it possible, yes, but not causing it.

Indeed, it was during this period that the importance of information began to be recognized by commercial and industrial companies. They began to install centralized management information systems and to experiment with access to external databases. While the level of information use was still small in comparison with today, it was clearly growing.

Among the developments during this period were several that paralleled those in the publishing and library field that I will be discussing in a moment. They were systems that provided online access to a variety of commercial databases, such as those for checking the credit status of individuals and organizations, those for airline reservations management, and those for stock market quotations. The technologies involved, combining as they did computers with telecommunications, were essentially the same as those for reference database access. The software, of course, was different since it involved much simpler record structures, much simpler retrieval criteria, but far greater numbers of transactions, by orders of magnitude.

The consumer use of information was also a area of growth, especially for the various entertainment media - movies, network TV and cable TV, and sports. In the period from 1964 through 1980, expenditures for consumer information quadrupled, increasing from $20 billion per annum to $80 billion; of that, sales of leisure books, newspapers, and magazines increased from about $10 billion to $40 billion.

During most of this period, computing was centered on large main frame computers. "Grosch's Law" (named after Herbert R. J. Grosch, who formulated it) governed most of the decisions about computing installations. It states that "computing power goes up as the square of the cost"; the implication of which is crystal clear: The bigger the better! (Parenthetically, Grosch's law has a theoretical basis in the physical structure of core-based internal memory systems; typically, they are structured in square arrays, for which the associated electronics increases linearly as the size increases as the square.)

In parallel, there was extensive development of networks among computers which became the backbone of today's Internet. These computer networks were made operationally feasible by the implementation of satellite and fiber-optic communication systems and the resulting explosion of capacities for data transmission.

During this period, the processes for publishing underwent dramatic change, as computer photocomposition completely replaced the former manual methods. And xerography became a fact of daily life in business of every kind. From the standpoint of this history, though, the important development in publishing was the onset of electronic formats. The beginning was the creation of databases, initially as a result of efforts by the U.S. federal government which paved the way for the commercial database access services to become established and then for commercial companies to enter into the business of producing databases to be mounted on those services.

Toward the end of this period, the CD-ROM (compact disk, read-only memory) had become a significant electronic format. In the time since then, publication in this format has doubled every two years.

A development of some importance for electronic formats was a standard language for encoding of documents (SGML - Standard Generalized Markup Language, now embodied in HTML for use on the Internet).

During this period, there were increasing efforts to establish cooperative networks among libraries -- not so much in the form of communication networks as in the form of administrative networks. Each of the states, under the stimulus of the Library Services and Construction Act, created its own multi-type state library network. The National Library of Medicine, under the mandate of the Medical Library Services Act, created the regional medical library system as a network And a variety of other networks were formed among groups of other types of libraries, such as academic ones. OCLC and the Research Library Network were created as cooperative network to solve the generic problem of retrospective conversion of catalog records.

Of course, cooperation in the area of cataloging was made possible only because of the crucial contribution of Henriette Avram, at the Library of Congress, in establishing the MARC format as the de facto national (and now international) standard for exchange of catalog data; without it, shared cataloging would have been intolerably difficult.

The most significant developments with respect to library services derived directly from the development and distribution of databases. Of special importance were the online services (DIALOG, for example) for access to the reference databases (such as CAS, ERIC, and Medline) and the OPACs for access to catalog databases. Less dramatic but still important was the implementation of ILL services by OCLC and RLIN.

It is in this area that perhaps the most dramatic change took place within this time period. With the stimulus of the MARC format and the use of OCLC and RLIN as economic solutions to the catalog conversion task, systems to support cataloging could be developed within institutions. A couple of them (DOBIS and NOTIS) became commercial products. In parallel, modules were developed to provide other parts of internal, technical services -- circulation and collection management, serials records, acquisitions.

This has been a most remarkable 17 years! The effects of international events for our field are of exceptional importance as countries throughout the world each will need to deal with the information revolution. They face almost overwhelming problems in doing so, however. The developing countries, in particular, need to create technological, economic, political, and social infra-structures that will permit them to advance into the information age.

Of special importance, of course, are the technological infra-infrastructures in telecommunications and computers, for use by business and the general public. All of that faces almost insurmountable hurdles in the magnitude of capital investment required.

This period has seen a virtual revolution in the extent to which information resources are used throughout the United States society. Today, massive amounts of data are generated, transmitted, and consumed. The cellular telephone is ubiquitous; the Internet and World-Wide-Web are growing in use at a phenomenal rate; the entertainment and amusement industries are exploding.

Within companies, management information systems now function as decentralized services that bring data directly to the point of immediate need. Communication within companies and with their customers invariably starts with bringing up a display of an appropriate record from a file. The use of the Internet and other services brings external data directly into the process of decision-making.

For the consumer, the picture is completely comparable. Today, online banking from the home is a fact of life and the use of credit cards in every commercial venue has become commonplace. The entertainment and amusement industry continues to grow at a phenomenal rate, and the publishing of books and magazines has grown similarly. Indeed, the "super book stores" proliferate at an almost unbelievable frequency.

You will recall that during the first thirty years the decisions concerning computer acquisition were based largely on Grosch's Law: Bigger is better! But something happened toward the beginning of this period that was a fundamental revolution. The microcomputer totally reversed the law: Smaller is better! The resulting "law", which became known as "Moore's Law" (named after Dr. Gordon Moore, co-founder of Intel), was that the capabilities of microprocessors double every two years. The result is that today I can hold in the palm of my hand more computing power than even the largest of main frames of two decades ago. The result has been a distribution of computing power that puts the PC, laptop, or palm top in the plane seat, the police car, the fire engine, the personal auto, as well as in every home and office. Of course, Moore's Law has not meant that the cost of computers has been reduced; instead, their cost has remained almost constant, but their functionalities have dramatically increased, by orders of magnitude.

The revolution in information technology is far greater than just the computer itself, though it indeed is the centerpiece. Increases in telecommunication capacity in some respects equal those in computer capacity. Whereas thirty years ago, we were limited to 10 characters per second (roughly the equivalent of 100 baud), today we have baud rates 5000 times greater for use in our own homes. Whereas thirty years ago we had CRTs with minimal capability, today we have SVGA display with resolutions virtually the equivalent of the printed page and screen capacities that permit the most beautiful images imaginable and the ability to observe in real time the operation of the heart or the brain of a human being, in living color no less. Whereas thirty years ago we were happy to have 6000 bytes of internal memory, today we can have almost unlimited numbers of megabytes. And whereas thirty years ago the means for mass storage of data were limited in capacity and unbelievably slow, today we have gigabytes of capacity with rapid random access.

Turning to the library context in the United States, the major effect has been economic pressures on any and every public enterprise -- among which are libraries. The funding problems for both public and academic libraries are real and in some cases catastrophic. Yet, in the face of the economic travails, the use of libraries has grown dramatically, for both public and academic libraries. The information economy requires the kinds of resources and means for information access that libraries and only libraries provide.

The effects of automation on library services have been dramatic. The availability of OPACs in virtually every library makes the resources of the library and, in most cases, of the world readily available. The availability of CD-ROMs brings a wealth of materials into the library, not only available but in processible form. The Internet and the World Wide Web provide means for online communication and access that are changing the entire information economy.

During this time period, the bits and pieces that were created by individual institutions have been replaced by integrated library systems, commercially available and with the support of highly qualified professional staffs. They are operational on both main frames and personal computers; they function in both stand-alone and "client-server" modes; they will serve every type and size of library; they will function well in any country of the world.

It is very difficult to make predictions, especially about the future, but I am going to do so, adding a fourth time period - the coming five years rather than fifteen - to those I have already discussed.

I predict the revolution that has resulted in the "information economy" will continue and the pace will accelerate rather than decrease, at least in other countries of the world if not in the United States. The facts are that information products and services have become a world-wide phenomenon. It is estimated that it is a trillion dollar market. The Internet and World Wide Web are simply a manifestation of it. The entertainment industry is without a doubt a major component, but computer related software and information packages are of increasing importance.

The reason I predict that this phenomenon will continue to increase is both simple and complex. The simple fact is that it has become easy to create information-based products and services, and since the basis for doing so grows the capabilities increase exponentially. The more complex fact is that creating information products and services consumes almost no physical resources; this means that it is not only easy but highly economic to do so. One can create a product or service with almost no investment, except one's time, and easily test the market for it at minimal risk.

Of course, the ability to do so depends upon both the technological infrastructure - the telecommunications system and the availability of inexpensive computers - and the capability to use it. That means that societies and countries without that infrastructure and those skills will be less able to participate in the growth; the gap between the "haves" and the "have-nots" will grow, not decrease.

So, what is going to happen to the use of information in commerce, industry, and by the consumer during the coming five-year period? Frankly, awesome though the implications are, I think it will continue to grow as it has for the past period.

During the nearly fifty years covered by this review, the computer business has continued to grow at phenomenal rates. Each year, the capabilities have increased, the uses have increased, the effectiveness and efficiency have increased, the pervasiveness in our economy and society has increased.

Of that fifty year history, though, the most spectacular gains have been during the past twenty-five years when "microprocessors" - computers on chips - became the basis for the hardware. The first microprocessors were breathtaking innovations and they revolutionized not only the design of computers but their use throughout our world. But in the ensuing years, the capability of microprocessors continued to advance at a spectacular pace, with Moore's Law repeatedly re-confirmed.

Will the process continue? The answer is Yes and, while there are absolute barriers in the laws of physics, there are still means by which the process will continue. Most important among them in fact are those that relate to the role of software; future developments in that respect may well have even greater impact than those we have seen to date with hardware.

It is valuable to highlight publishing as one component of the information economy, one that has special importance to the future of libraries. What is the future for publishing, especially in the light of development of the information technologies? In particular, what will happen to publication in print formats? Let's look at the means for electronic publication:

First among them is the Internet and the World Wide Web (WWW); they are growing exponentially with a doubling virtually every three to six months. It is important to note, however, that the majority of current and future increases in these services and uses are highly commercial. While the Internet, academic in its origins and orientation, is growing it is doing so at a substantially more limited rate and is already very small in comparison to the WWW.

Currently, CD-ROM is the fastest growing form of publication, doubling in sales every two years. While it may be displaced at sometime soon by DVD ("digital video disc"), the trends will continue and be similar. Close to this rate of increase is likely to continue for the coming ten-year period, though with a steady leveling over time. It is important to note that, in the past, publications of importance to libraries have been of reference materials (indexes, abstracts, encyclopedias, data collections, etc.), which typically are made available through the CD-ROM LAN; those publications are likely to level off quite rapidly. The substantial increases will be in instructional materials (especially as represented by multi-media packages), in journals, especially scientific journals, and perhaps in other types of material. For purposes of general reading, however, this form of electronic publication is unlikely to displace the printed book or popular journal. Therefore, for library planning it is necessary to plan for continued acquisition of materials in print form.

Electronic document delivery is a reality today, as represented by the commercial services which provide it as a replacement for inter-library lending as well as for its own values in speed of delivery. In the future, general scholarly journals, especially those produced by commercial publishers, are likely to shift from current print publication of issues to future on-demand publication of articles. Print versions may even disappear.

While there is clear evidence for a shift in journal publication from printed issues to on-demand publication of articles, there is no evidence of comparable shift of monograph publication in the same way. And even though newspapers and popular journals are likely to publish online versions, distributed through the WWW, they will continue to publish their print versions. As with CD-ROM materials, therefore, for purposes of general reading online electronic publication is unlikely to displace the printed book or popular journal.

Digitized imaging is a form of publication that will be of increasing importance in specific areas. It has become a powerful tool in research of all kinds. It is the basis for publishing of materials such as maps and may well replace print as the means for publication of them. Academic libraries, especially, should plan for management of digitized images.

So, what will happen to libraries? There are persons who forecast their demise, in the perception that they will be replaced by the wealth of resources becoming available through the information technologies; such voices have been heard for at least the past three to four decades. It is a fact that during the past decade libraries have faced enormous economic pressures; they have had to operate within the constraints of reduced budgets at the same time that the costs of acquisitions - especially of journals - have been escalating at exponential rates of 15% to 20% per annum. At the same time, they have needed to make continuing investments in automated systems and to deal with the array of computer-based forms of publication.

My own perception, though, is very different from that of those who wish to get rid of libraries. My view is that libraries are essential and will continue to be so in the foreseeable future. Instead of being overwhelmed by technologies, they have absorbed them, made them economic and effective, and served as the basis for testing and proving them.

It is also a fact that the use of libraries of every type has been increasing, not decreasing. Indeed, the effect of electronic information resources is to increase not decrease the use of the library. The various forms of publication are complementary and mutually supportive rather than being substitutes for each other. The use of any of them leads to increased use of the others, and the library serves as the agency for access to all of them.

I have every expectation that the library will continue not only to exist but to thrive and to play its historic leadership role in the coming decades. Underlying that expectation is my view that, while electronic publication will be increasingly important, it will not replace print in the foreseeable future. And libraries will be the means of access to both print and electronic formats.

So what will happen to library services? The crucial point to me is that the widening scope of information resources increases the importance of library services. In this respect, it is important to recognize that the library is more than simply a collection of materials, valuable though that is and will continue to be. It is also more than simply a means for access to those materials and the information contained in them, again valuable though that is and will continue to be. The library is the agency that serves as the means for SELECTION. It does so when acquisition librarians make decisions about what materials are worth adding to the collection. It does so when reference librarians help patrons in locating and selecting from the wealth of resources those that will meet needs. It does so when library-based information specialists select from retrieved information and analyze the results.

The library is also a means for users to learn how to manage information resources. Library services in teaching are therefore of special significance, and the increasing wealth of resources adds greatly to its importance.

My perception of internal, technical processing is that the dramatic changes resulting from automation of them have occurred and that the future will not significantly add to them. The bibliographic utilities, OCLC especially, are now well established, economically viable operations; their scope of coverage is becoming increasingly international. Cataloging and acquisitions work can depend upon their online union catalogs (OLUC, as OCLC refers to it).