The following post is part of a series related to the provision of bibliometrics and research impact services at OCLC Research Library Partnership institutions.
Research libraries increasingly provide support and education on the topic of research impact. But what information are they providing users about the biases baked into frequently used research metric resources?
According to a recent research study by Sheila Craft-Morgan at The Ohio State University, not nearly enough.
Sheila recently shared the findings from her investigation into the content included in LibGuides on research impact topics in a webinar presentation for the OCLC Research Library Partnership. Sheila previously wrote about the issue of bias in research metrics in a May 2023 Hanging Together blog post, co-authored with Allegra Swift from University of California San Diego (UCSD).
What does bias in research metrics look like?
Sheila described some of the practices that create bias in the scholarly communications lifecycle, as documented in the literature[1]. For example:
- Men engage in self-citation more often than women
- Women are less likely to be cited than men
- Scholars from the most prestigious institutions receive more citations
Sheila emphasized that this bias is both systemic and structural, often with deep historical roots. For instance, she provided a powerful example of uneven citation practices between the Journal of the American Medical Association (JAMA) and the Journal of the National Medical Association (JNMA), where JNMA authors cited JAMA articles, but not vice versa. This disparity has a historical origin: the NMA and its journal were both founded as a professional venue for African American physicians who were prohibited from joining the AMA due to Jim Crow laws and customs.[2]
The implicit and explicit biases inherent in scholarly communications can limit content discovery, create an inaccurate view of the scholarly record, and further reinforce inequalities. Uncritical reliance on research metrics can potentially cause harm to individuals and groups. Scholarly communications, including metrics, must become more open and equitable, as advocated for in a 2019 ACRL report and research agenda, Open and Equitable Scholarly Communications: Creating a More Inclusive Future.
Key findings
Sheila’s research project focused on the research impact practices at a subgroup of 50 ARL libraries—a sample of elite US institutions which notably does not include any Historically Black Colleges and Universities (HBCUs). In particular, she examined the LibGuides on research impact at this group of libraries. She found that:
- Over 90% of these LibGuides contained information about article, journal, or author metrics, persistent identifiers, and researcher profiles.
- Nearly 50% included information about the responsible use of metrics.
- But only 14% (n=8) included information about bias or DEIJ (diversity, equity, inclusion, and justice) resources.
Exemplars: LibGuides that address bias
Sheila pointed to examples from three institutions that discussed bias and did it well:
- Penn State. The home page of the LibGuide on Citation and Journal Measures provides a comprehensive disclaimer discouraging the use of citation counts and research level metrics as a proxy for quality of research due to inherent bias, with specific examples.
- Brigham Young University. This guide calls out many of the pitfalls of research metrics, including subjectivity, disciplinary differences, and inherent bias.
- University of California San Diego. This guide directs users to DORA’s resource on Rethinking research assessment: Ideas for action, which includes information about unintended cognitive and system biases.
Call to action
Sheila concluded the webinar by encouraging others to take action to address issues of bias in research impact, by doing things like:
- Include information about implicit and explicit bias in webinars, online resources, and toolkits about research impact
- Advocate for change within our own library organizations
- Partner with colleagues in other areas of the library, such as teaching and learning librarians who are educating students in inclusive citation
- Just begin!
I’ve focused on Sheila’s findings on LibGuides in this blog post, but her webinar revealed other interesting findings about research impact services in ARL libraries. I invite you to watch the webinar presentation and review the terrific bibliography in her slides.
[1] Chawla, D. Men cite themselves more than women do. Nature (2016). https://doi.org/10.1038/nature.2016.20176
Kahalon, R., Klein, V., Ksenofontov, I., Ullrich, J., & Wright, S. C. (2022). Mentioning the Sample’s Country in the Article’s Title Leads to Bias in Research Evaluation. Social Psychological and Personality Science, 13(2), 352–361. https://doi.org/10.1177/19485506211024036
Liu, F., Holme, P., Chiesa, M. et al. Gender inequality and self-publication are common among academic editors. Nat Hum Behav (2023). https://doi.org/10.1038/s41562-022-01498-1
Ross, M.B., Glennon, B.M., Murciano-Goroff, R. et al. Women are credited less in science than men. Nature 608, 135–145 (2022). https://doi.org/10.1038/s41586-022-04966-w
Skopec, M., Issa, H., Reed, J. et al. The role of geographic bias in knowledge diffusion: a systematic review and narrative synthesis. Res Integr Peer Rev 5, 2 (2020). https://doi.org/10.1186/s41073-019-0088-0
Teixeira da Silva, J.A. The Matthew effect impacts science and academic publishing by preferentially amplifying citations, metrics and status. Scientometrics 126, 5373–5377 (2021). https://doi.org/10.1007/s11192-021-03967-2
Teich, E.G., Kim, J.Z., Lynn, C.W. et al. Citation inequity and gendered citation practices in contemporary physics. Nat. Phys. 18, 1161–1170 (2022). https://doi.org/10.1038/s41567-022-01770-1
[2] Escobar Jones, Cherice, Reid, Gwendolynne, Poe, Mya. “Leading American medical journal continues to omit Black research, reinforcing a legacy of racism in medical knowledge” — https://theconversation.com/leading-american-medical-journal-continues-to-omit-black-research-reinforcing-a-legacy-of-racism-in-medical-knowledge-185111
Rebecca Bryant, PhD (she/her), previously worked as a university administrator and as community director at ORCID. Today she applies that experience in her role as Senior Program Officer with the OCLC Research Library Partnership, conducting research and developing programming to support 21st century libraries and their parent institutions.
