Benchmarking Network Performance: Measures and Behaviors

Over the last year, Günter and Ricky have been examining models of collaboration across the cultural heritage community, working with RLG partners to identify the most significant obstacles and incentives to effective library-archive-museum partnerships. This is part of a program of work exploring cross-domain convergence in organizational structures and service requirements. Günter has reported on some of this work here (May ’08), here (November ’07) and here (July ’07). A final report from this project is expected soon. It should help us to understand how cooperation contributes to improved institutional performance, including greater discoverability of collections, better integration of functions, and increased operational efficiencies. All of these objectives are important to research institutions that want to participate fully in the network information economy.

In a related vein, Dennis recently queried members of the long-running SHARES inter-lending partnership about the hallmarks of ‘high-performance’ sharing partners. A new Working Group on High Performing Lenders has drafted a survey to

learn what qualities [SHARES participants] value in a supplier and which … partners consistently display those characteristics

I was interested to see that the criteria under consideration include some social behaviors and delivery options that respond to expectations that have been shaped (or at least sharpened) by the larger network environment. These include [with my annotations]:

  • Quality of scanning/copying [surrogates should meet or exceed quality of original
  • Willingness to supply rare or hard-to-find materials [if content is discoverable, it is assumed to be available]
  • Quality of holdings data in WorldCat [supply chains rely on accurate and reliable disclosure]

The library’s ability to meet end-user expectations is dependent upon the performance of its service providers — including its inter-lending partners. This is increasingly true in an environment where ‘local’ holdings are assumed to be continuous with the collective collection of library content. The larger information network imposes a set of collaborative imperatives that cross organizational, institutional and geographic boundaries.

All this has led me to wonder, as the ARL Library Assessment Conference (Seattle, 4-7 August) approaches, about the kinds of metrics we use to assess the value and performance of cultural heritage institutions, especially those that serve the research community. My colleague John MacColl attended a recent meeting of representatives from some of the organizations involved with with establishing and monitoring research library metrics, including ARL (123 institutions in North America), SCONUL (172 institutions the UK) and CAUL (41 institutions in Australia). There is evidently some interest in harmonizing these measures so that research libraries can begin to benchmark their collections and services against global indices. This suggests that libraries are seeking to situate themselves in an expanding network of information service providers in which performance standards reflect common operational requirements and social norms.

There are a couple of measures in the current array of library performance indicators that can be used to assess inter-institutional cooperation on a system-wide scale. Resource sharing statistics provide a gauge of network participation and institutional co-dependence. Annual expenditures on shared infrastructure are another useful index of collaboration. Since the mid 1990’s the National Center for Educational Statistics has tracked US library investments in ‘bibliographic utilities, networks and consortia’ — an interesting combination of social and technological support systems — as part of overall library operational expenditures. ARL added this category to its statistical measures in 2004. An acknowledgement, perhaps, that institutional performance and achievement in the research library sector is increasingly dependent upon collaboration.

In other sectors — notably supply chain management — the tangible benefits of cooperation have been studied more carefully. [References] Some fascinating work has been done to model instruments for measuring collaboration amongst “chain members” to enable reliable benchmarking and performance assessment. [Simatupang, T. M., & Sridharan, R. (2005).] The LAM community might benefit from a similar assessment framework, which acknowledges the multi-dimensional character of cooperation (information sharing, decision synchronization, alignment of incentives) and highlights its real operational value.