Generative AI

We can evaluate AI-generated content in much the same ways as we evaluate other sources. Tests such as the CCOW method of source evaluation can be helpful in determining whether the generated information you've found is reliable.

However, some of the questions we typically ask about sources may be more difficult to answer when working with generative AI, because AI tools do not disclose the processes used or the information consulted to arrive at the answers it provides.

Here are some important areas to consider when evaluating AI-generated content:

Accuracy

• Is the information provided true and accurate?
• Is the output logical?
• Are you able to verify claims?

Authority and Bias

• Are you able to verify that information output comes from a trustworthy source? If not, are you able to verify the information output from another source on your own?:
  • Hallucinations are common, and some tools are more likely to hallucinate than others. AI tools should not be used to generate sources or citations, and all cited information in output should be verified.
• Does the output include any statements that promote stereotypes or lack nuance?

Currency

• How up-to-date is the data your tool has been trained on? Is that information available?
• How important is currency of information for your research?
• Do you need to seek out more current resources elsewhere?

Check out the video below for a more in-dept explanation of evaluating AI-generated content:

Being AI Literate does not mean you need to understand the advanced mechanics of AI. It means that you are actively learning about the technologies involved and that you critically approach any texts you read that concern AI, especially news articles. 

The ROBOT test is a tool you can use when reading about AI applications to help evaluate the legitimacy of a particular AI technology.

 
Reliability

• How reliable is the information available about the AI technology?
• If it’s not produced by the party responsible for the AI, what are the author’s credentials? Bias?
• If it is produced by the party responsible for the AI, how much information are they making available? 
  • Is information only partially available due to trade secrets?
  • How biased is they information that they produce?

Objective

• What is the goal or objective of the use of AI?
• What is the goal of sharing information about it?
  • To inform?
  • To convince?
  • To find financial support?

Bias

• What could create bias in the AI technology?
• Are there ethical issues associated with this?
• Are bias or ethical issues acknowledged?
  • By the source of information?
  • By the party responsible for the AI?
  • By its users?

Owner

• Who is the owner or developer of the AI technology?
• Who is responsible for it?
  • Is it a private company?
  • The government?
  • A think tank or research group?
• Who has access to it?
• Who can use it?

Type

• Which subtype of AI is it?
• Is the technology theoretical or applied?
• What kind of information system does it rely on?
• Does it rely on human intervention?

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

To cite in APA: Hervieux, S. & Wheatley, A. (2020). The ROBOT test [Evaluation tool]. The LibrAIry. https://thelibrairy.wordpress.com/2020/03/11/the-robot-test