Taking the measure of bibliometrics and research impact services in libraries

Bibliometrics and research impact (BRI) services are a rapidly growing and evolving area of interest for many universities, as more and more uses are found for data-driven analysis of the research enterprise. While bibliometrics enjoys a long tradition in libraries, today’s BRI librarians are employing new skills and tools to meet both established and emerging needs in areas like strategic decision support, benchmarking, reputation analysis, support for funding requests, and understanding research performance.

The OCLC Research Library Partnership (RLP) Research Support Interest Group recently convened a discussion session on BRI services in libraries. The session was an opportunity for RLP members to connect with and learn from others in the network, and the result was a rich and informative conversation blending many perspectives. This blog post recaps some of the highlights from the discussion.

The session was led by two BRI leaders from the University of Waterloo – Laura Bredahl (Bibliometrics and Research Impact Librarian) and Alison Hitchens (Associate University Librarian, Collections, Technology, and Scholarly Communication). Laura and Alison set the stage with a brief overview of how BRI services developed at Waterloo, reprising a more detailed account presented during their recent Works in Progress webinar for the RLP membership (and discussed in this recent blog post by my OCLC colleague, Rebecca Bryant).

Institutional interest in BRI at Waterloo emerged from the university’s strategic plan that emphasized a transformational research theme, which in turn required methods for assessing progress. Increased use of bibliometrics by external agencies, such as university ranking bodies, was also a driver. In response, the provost formed a cross-unit working group to develop a white paper that would help campus stakeholders understand bibliometrics and their responsible use. The group also conducted an extensive review of the assessment tools that the university was currently licensing, as well as a detailed review of existing indicator assessment activities at Waterloo. The working group brought together representatives from across the campus, including the library, the office of research, the institutional analysis and planning unit, faculties, and research institutes. Strong working relationships were developed among these partners, particularly between the library, office of research, and the institutional analysis and planning unit (an exemplar of social interoperability!). The spirit of collaboration embodied in the group was extended to the establishment of a campus-wide BRI Community of Practice, led by Laura, aimed at supporting those around campus working in bibliometrics and other forms of research impact analysis.

In hearing Laura and Alison’s story, three things stood out:

• The importance of university administration in driving use cases for, and serving as consumers of, BRI services.
• The vital role of cross-campus relationship-building and collaboration as a means of assessing and meeting campus-wide BRI needs.
• The crystallization of campus-wide interests in BRI in the form of a new position (the role that Laura now holds) responsible for maintaining expertise in BRI tools and analysis, as well as working with senior administration, research institutes, departments, and faculty.

Establishing campus communities of practice

The subsequent discussion touched on a variety of themes, one of which was participants’ experiences in establishing cross-campus connections in the area of BRI. Many participants reported that they were still in the relationship-building stage, trying to make connections with other BRI stakeholders around campus. Those that have established relationships noted the importance of partnerships with campus units like the office of research, central planning units, schools and colleges, and research institutes. However, several participants cautioned those just starting to figure out who is engaged in BRI activities on campus not to confine attention to job titles with “research” in them. The range of individuals engaged in BRI is extensive, and in many cases, responsibilities for evaluating bibliometric data and examining research impact might have been added to existing roles. One participant noted that the marketing department on their campus was very interested in bibliometrics, demonstrating that interest in BRI activities can spring up in unexpected places.

Establishing connections with other campus units interested in BRI can take place in a variety of ways. Some participants described the emergence of informal, peer-to-peer linkages, which helped with knowledge sharing and community building. However, another noted that lack of a top-down approach to collaboration, in the form of cross-unit working groups or committees, meant that such interaction that did take place tended to be sporadic and unsustained, and ultimately inefficient for the purpose of building a robust community of practice across campus. A more formal, top-down approach helped to bridge silos and avoid costly fragmentation in how BRI is undertaken around campus. In general, participants agreed that both approaches – bottom-up and top-down – were necessary to build lasting working relationships across campus units.

Establishing cross-campus connections in BRI can occur through new groups set up for the purpose, such as the working group at Waterloo, but existing groups may work just as well: for example, one participant noted that an existing cross-unit group tasked with supporting grant proposals on campus would likely serve as a good venue for discussing BRI. But challenges remain: budget uncertainties can make starting new cross-unit initiatives difficult, while frequent transitions in unit leadership or staffing might lead to a repetitive process of re-establishing existing partnerships.

One participant noted that campus-wide interest in the United Nations Sustainable Development Goals helped “get a foot in the door” in talking to campus stakeholders about BRI, given that the SDGs were included in the university’s strategic planning, cut across many disciplines, and had a metrics-related component.[1] Similarly, another participant highlighted the importance of having a white paper or briefing document – such as the one produced by the Waterloo working group – as a shared reference point for cross-campus discussions on BRI.

Use cases for BRI

Another discussion theme touched on use cases for BRI services. Use cases are particularly important as a means of understanding the aspects of BRI campus stakeholders are really interested in, such as discovering partners for research collaborations or matching researchers to grant funding opportunities. Research institutes often put together annual reports, requiring metrics to demonstrate influence and impact. One participant observed that, rather than individual researchers, their biggest use case was supporting directors and administrators requesting help in demonstrating research impact.

Research impact extends to open access and reaching the general public with research findings. As one participant phrased it, the important thing used to be who you know; now, it is who knows you. In this sense, maximizing research impact requires scholars to have profiles created, updated, and discoverable in institutional research information management systems.

Responsible use of metrics

While participants were able to articulate a host of use cases for BRI services on their campuses, one cautioned that it was essential to understand the limits of bibliometric analysis and communicate this to stakeholders. Additionally, participants pointed out that the ability of BRI services to meet campus needs depended on access to complete, high quality data to compute meaningful metrics. As one participant put it, we often have the tools, but lack the ingredients (data); this has been and continues to be a significant challenge. For example, several participants noted that they have been struggling with translating BRI services to the arts and humanities, where traditional impact measures like number of publications or citations are poor proxies of research output and impact. A participant shared that they were conducting a survey on campus with arts and architecture faculty to determine what tools they used to measure impact, as a first step toward developing services around those tools and practices.

An interesting strand of the discussion touched on responsible use of BRI analysis. Because a great deal of data and many analytical tools are readily available online, almost anyone can dip into BRI – and even construct their own metrics. Participants noted some danger here, especially when “handcrafted” metrics are used to support important decision-making, such as tenure. Campus stakeholders in BRI need to be educated about this, and the library may have an important role as an advocate for the responsible use of research metrics, such as encouraging the campus to support efforts like the Declaration on Research Assessment (DORA).

BRI Tools

BRI services can require a range of sophisticated (and usually expensive) tools. Participants cited a wide variety of tools that they employed at their institutions, such as SciVal, InCites, Dimensions, Lens.org, and Pure; APIs from Scopus, Unpaywall, and Web of Science; visualization tools like VOSviewer; and general data tools like Excel or OpenRefine. The landscape of BRI tools is currently very dynamic; one participant observed that new tools in areas such as data visualization are appearing all the time.

Tools needed will depend on use cases, and in some instances, collaborative partners and the tools they use – for example, university ranking services or research assessment exercise bodies. But adopting new tools can be delayed by a lack of use cases or infrequent need: as one participant observed, their BRI services were still “niche”, and therefore many BRI tools were too expensive and/or too difficult to learn on the basis of infrequent use.

Our discussion of BRI tools prompted several questions.

• Should libraries acquire BRI expertise and supply BRI services, or should their efforts be focused on training others to perform BRI analysis? One factor in this decision may be the extent to which other campus units have the capacity to retain BRI expertise – for example, will BRI analysis be done by student workers with a high rate of turnover?
• Should libraries focus on becoming a center of expertise on high-end tools, which may be more efficient than trying to duplicate this expertise across many campus units?
• Should BRI tools be funded by the library, or through centralized budgets, given that the tools often support campus-wide needs?

While these questions remain open, participants did agree that as BRI librarians acquire more expertise, they will be able to conduct more and more sophisticated and valuable analyses with BRI tools. They also emphasized that the essential first step in deciding what BRI tools are needed is to understand what needs to be done with them – what are the use cases?

Other topics

The discussion ranged across many other topics, including data stewardship issues: what should be done with data that is created through BRI analysis? Where should it be stored? Who has access? How can it be brought, in the words of one participant, “under the enterprise data governance umbrella”? Participants also noted potential roadblocks, such as securing necessary permissions, in combining data for BRI analysis from different sources around campus.

We also discussed relevant expertise to bring into newly created BRI positions. One participant suggested that aspiring BRI experts should “not be afraid of data analysis and spreadsheets”, and have a curiosity about data, noting that previous experience as a liaison librarian proved helpful. Likely skill gaps for new BRI librarians may include experience in using APIs, R, Python, and other programming/development competencies. Participants agreed that establishing a budget line to support training is vital to ensure the success of people coming into BRI roles.

The session closed with some sage advice from our discussion leaders. Asked what advice they would give to other institutions thinking about creating a BRI position in the library, Alison and Laura underscored the importance of cross-campus connections and collaboration:

• Have conversations about the position not just in the library but also with those around campus who would potentially partner with this person in their work.
• Invite staff from other units to be on the hiring committee for the BRI position.
• Start developing your campus partnerships early.
• Do not try to solve all of your institution’s BRI problems at once; choose your initial use cases carefully.

We thank Alison and Laura, and all of the participants in the session, for a rich and engaging conversation. It was a unique opportunity to pool the collective experiences, challenges, and questions of the RLP membership in the emerging area of BRI services in libraries.

My thanks to Rebecca Bryant, Alison Hitchens, and Laura Bredahl for reviewing a draft of this post!

---

[1] OCLC is also supporting the SDGs: see https://www.oclc.org/go/en/sustainable-development-goals.html