Implementing an AI reference chatbot at the University of Calgary Library

In 2021, the University of Calgary Libraries launched a multilingual reference chatbot by leveraging a commercial product that combines a large language model (LLM) with retrieval-augmented generation (RAG) technology. The chatbot is trained on the library’s own web content, including LibGuides and operating hours, and is accessed from the library’s website.  

In a Works in Progress webinar hosted by the OCLC Research Library Partnership (RLP) on 20 November 2024, University of Calgary Library staff discussed the creation and implementation of the AI reference chatbot and shared lessons learned. Kim Groome, Information Specialist; Leeanne Morrow, Associate University Librarian, Student Learning and Engagement; and Paul Pival, Research Librarian–Data Analytics presented. 

This blog post provides a summary of the webinar’s key points, but for a deeper dive, you can watch the full recording here: 

Project genesis 

Like many research libraries, the University of Calgary Libraries has offered live chat to users since the early 2010s, with information specialists staffing the service daily from 9 a.m. to 5 p.m. The pandemic catalyzed discussions about an AI chatbot, with many factors driving the conversation:  

• Surging demand for chat services. Chat usage spiked dramatically during the pandemic. While the library typically handled 500–900 live chats per month in 2019, this number skyrocketed to 3,077 in September 2020. 
• Staffing constraints. The increased volume required additional staff and staff time to keep up with demand. 
• Limited service hours. Staffed by humans, live chat was available during extended business hours, but this still left students without support during the late evenings or early mornings. 
• Improved convenience. Even students visiting the library in person utilized the chat reference service. It was convenient and helped them maintain their study space during peak hours.  
• Automation potential for many questions. An analysis of live chat questions revealed a significant percentage of questions that were well-suited for automated responses. 
• Alignment with institutional priorities. Implementing an AI chatbot aligned with the university’s commitment to student-centered initiatives and its strategic focus on enhancing student success. 

The library team looked across the Canadian library ecosystem for examples but found limited adoption among other libraries.[i] Instead, the library found that at UCalgary, the Office of the Registrar had already implemented a chatbot named “Rex,” leveraging technology provided by Ivy.ai. By building on this preexisting campus project, the library accelerated its own chatbot initiative, benefiting from shared resources and institutional experience.  

Implementation 

Assessing the usefulness of an AI chatbot 

Initial work included conducting an analysis of past reference chat questions to evaluate the automation potential of an AI chatbot. Kim Groome described exporting approximately 3,000 chatbot interactions recorded over a one-month period during the pandemic and coding the questions into themes like study/workspace, printing, and borrowing requests. Through this analysis, which took approximately 30 hours, the library determined that 14-24% of reference chat inquiries were directional (e.g., “where is …?”) and could potentially be handled by a chatbot. The coding was performed using Excel; use of Python by experienced coders could further expedite the work.  

Training and testing 

After identifying a core set of common questions that could be effectively addressed by the chatbot, an eight-person library team began training and testing in April 2021, working with the vendor to increase consistency and quality of the chatbot answers. Testing was extended to other library staff members in July 2021. Recognizing the potentially infinite scope of user questions, the team avoided scope creep by initially focusing on a defined set of about fifty questions identified during their analysis.  

Go-live 

The library’s chatbot launched on 16 August 2021, branded as “T-Rex” to differentiate it from the preexisting “Rex” chatbot offered by the registrar (note that the Rex is the official mascot of UCalgary Dinos teams). Today T-Rex is one of six chatbots on the UCalgary campus, each operating on separate knowledge bases and answering questions 24/7. 

Continuous improvement and maturity 

Quality monitoring  

Kim described how the library team continually assesses the quality of chatbot responses using anonymous weekly reports. The team rates the bot’s answers on a 1 to 5 scale, where 5 represents a perfect response.  

Examples of the rating process 

Participants asked many questions we were unable to address during the webinar due to time constraints. Following the webinar, Kim offered these examples to address questions about the rating process: 

• 5/5 response: If patron asks, “Do you have databases for nursing”, the bot provides an accurate answer, earning a perfect score.  
• 4/5 – 5/5 response: If patron asks for the specific nursing database, such as, “Where can I find CINAHL” (even if spelled incorrectly in six ways), the bot delivers an excellent response.  
• 2/5 response: But if the patron asks a question like, “I want to find articles on the social effect of the opioid crisis what database would I use,” the bot will struggle. Even though the bot didn’t answer the question, the team would still scale it at 2/5 because the specific topic is not on the website—and therefore not in the “bot brain.” Since the chatbot hasn’t been trained on the topic, it cannot answer the question. If the bot interprets the phrase “find articles” and offers a response like “to find articles, please enter the topic in the library search box…”, then the response would be rated at 3/5 or even 4/5.

Developing custom responses 

The team monitored user questions following go-live, identifying questions that would benefit from customized answers. For example,  

• A frequent user query about access to the Harvard Business Review couldn’t initially be answered because access to the resource was embedded in a search tool outside the RAG’s scope. 
• Misspellings were also common, such as for resources like PsycInfo.  

If any question was asked more than three times weekly in distinct transactions, the team would create a custom response to address it. Over the first year, the team spent about 5 hours each week creating 10-15 custom responses to these questions, incrementally improving the chatbot. For misspellings like the PsycInfo example, the team incorporated common misspellings like Psychinfo, pyscinfo, and psychinfo. 

Monitoring the chat is important for identifying and immediately correcting any wrong responses. For example, a patron once asked, “Can I return a book that has already been declared lost,” and the bot responded, “No, you cannot return a library book that was recently declared lost.” This is obviously incorrect, and the response occurred because of information missing from the library website. But the issue is also complex, with at least fifteen different circumstances surrounding a lost item; adding a series of complex scenarios to the website was an imperfect solution. Instead, the team created a rule where any question that includes words like “lost” and “book” receives a customized qualifying statement: “If you need to contact library staff about a lost a book, or the lost book charge, please email: <email address>.” Similarly, for questions that mention the term “recall,” the bot will respond, “Recalls are a very special circumstance. Here is an FAQ for more information.” 

Maturity 

The chatbot was further improved in February 2023 when a new GPT layer was added by the vendor, enabling the tool to generate its own responses to complement existing custom responses. Today the chatbot offers fast, consistent, 24/7 support that is accessible to a wide range of users and is WCAG 2.11 AA compliant. It knows over 2 million words—each form of a word is a new word (i.e., renew and renewing are two separate words)—and has over 1000 custom responses. T-Rex is very accurate for directional questions, and 50% of all questions receive a rating of at least 4/5.  

T-Rex has exceeded expectations. Before launch, the implementation team estimated that the chatbot could answer 14-24% of reference chat questions, but today the chatbot answers about 50% of all questions with a rating of at least 4/5. This deflects half of all questions from live reference chat. This has been significant, as 1.5 FTE of staff time has been redirected to support more strategic, higher-level tasks. As a result, the library has reduced staffed desk hours, instead encouraging users to rely on the 24/7 chatbot for immediate assistance. There have been no staff reductions, just higher productivity.  

Now that the chatbot is mature, it takes only about one hour per week to supervise and monitor the chatbot, primarily to confirm that it continues to work as expected. Updates, such as changes to library URLs, are efficiently managed using a simple Excel spreadsheet.  

The implementation of T-Rex was the library’s first AI effort. More recently, the library has collaboratively established the Centre for Artificial Intelligence Ethics, Literacy and Integrity (CAELI). Located within a campus branch library, CAELI supports student success by fostering strong digital and information literacy skills among UCalgary students. 

Lessons learned 

The UCalgary team shared several key insights from the project:  

• Use library web pages as the system of record. One of the very first lessons learned after go-live was that the chatbot would be unable to answer a question if the library didn’t have a webpage or FAQ that addressed the topic. While it could be tempting to update the chatbot’s responses directly, Kim advised against this approach because it would create duplicate maintenance points. Instead, she urged participants to consider the website as the system of record for chatbot content.  
• Leverage a team-based approach. Implementing the chatbot with a team-based approach increased resilience and reduced points of failure for the project.  
• Identify and respond to user expectations. Users preferred answers that connected them directly to the source they were looking for, rather than being directed to a webpage that required further navigation. Over time, the team refined responses to reduce the number of clicks required to reach specific information. 
• Expect non-library questions. The team discovered that users would ask the chatbot many questions that the library RAG was unable to answer, such as, “When can I register for the spring semester?” In many cases, the bot can direct the user to one of the other relevant chatbots on campus (registrar, admissions, financial aid, career services, etc.) for appropriate answers. This is a significant benefit of an enterprise approach to adopting chatbot technology.  
• Think creatively about addressing non-library questions. The Calgary library recognized its role in supporting academic integrity, and it analyzed the chatbot data to learn more about the types of academic integrity questions students were asking. The library found that students were asking questions about reference styles, citation managers, plagiarism and detection software, and academic policies. These questions often arose late at night when live support was unavailable. In collaboration with the campus academic integrity coordinator, the library developed custom responses and added relevant campus content to its website, enhancing the chatbot’s ability to support student success.  
• Anticipate that there will be non-adopters. Some people prefer to interact directly with other humans and are unlikely to adopt chatbot technology. About 12-15% of library chatbot users still ask to speak to a human, even in cases where the chatbot could likely answer their question. Users can click through to “Connect to a Person” directly from T-Rex during regular service hours.  

Library use of AI chatbots 

To understand webinar participants’ own use of and experiences with chatbots, we polled attendees during the presentation. Their responses provide anecdotal insights about library adoption of chatbots.  

While webinar participants were clearly interested in chatbots, they weren’t necessarily strong users. Only about 40% of participants reported using chatbots on a daily or weekly basis; 27% reported never using GenAI chatbots.  

Relatedly, nearly 50% of participants reported that they didn’t enjoy interacting with GenAI chatbots, although nearly as many had mixed feelings.  

Implementation of library chatbots  

This webinar was useful for our RLP participants because we learned that few libraries had implemented an AI chatbot, but nearly 50% were considering it.  

Is your library implementing an AI chatbot? Share a comment below or send me an email. I’m eager to learn more. Special thanks to the UCalgary Library team for generously sharing their experiences and insights so we can all learn from their innovative work. 

---

[i] Julia Guy et al., “Reference Chatbots in Canadian Academic Libraries,” Information Technology and Libraries 42, no. 4 (December 18, 2023), https://doi.org/10.5860/ital.v42i4.16511.