Our library has created an AI working group and we are working towards setting up formal evaluation of several tools (we're still putting that list together). But first, we wanted to create a values statement and a rubric (and perhaps a test guide?) to inform our evaluations and help us articulate our observations and how they relate to any decisions as to whether or not to enable generative AI features in products.

We've been asking for these features to be disabled until we are ready to perform evaluations. We're trying to be ready to evaluate the various AI tools over the coming summer.

I would imagine that this conversation could be very helpful in giving us various perspectives to consider when we're putting everything together, so thank you!

One of my concerns is the perception that AI summaries and/or ranking are somehow more authoritative than if the user did the work themselves and might curtail further investigation into a search. Even if the AI does a good job of relevance ranking and summarizing, it's not exactly performing a reference interview so does the user know what it does and doesn't know about what the user is actually looking for? These tools are designed to craft a response that meets the expectations of the user based on the question asked, so whereas the abstracts themselves are query-neutral and can provide the user with a sense of how close a particular result is to what they want, the crafted summary will be word-smithed to satisfy the question asked and may mask any disconnect between the query and the selected results. This problem actually gets worse as the hallucination problem gets better. The more reasonable the response, the easier it is to trust that the response is correct.

Sorry, I don't mean to derail the conversation with hypotheticals. I look forward to hearing everyone's observations.

Keith

Hi Everyone,

I wanted to begin our initial Topic Discussion series with the following:

What Clarivate/ProQuest/Ex Libris AI Products/Tools are you or your university currently investigating? What have you liked about the products and have you seen any Initial problems or concerns regarding the products?

If you have seen a significant concern, what case have you submitted, if any, and the case number associated with the issue.

I hope to track the information on a spreadsheet and provide it to the related working groups/steering.

We have a number of Ex Libris developers on this list so it will also assist with pointing out issues as well as sharing what people are finding helpful.

Thanks again for your contributions,

Davin Pate, M.L.S.

Assistant Director for Scholarly Communications and Collections

(972) 883-2908 |davin.pate@utdallas.edu

http://www.utdallas.edu/library/

The University of Texas at Dallas

Book time to meet with me