Generative AI

We can evaluate AI-generated content in much the same ways as we evaluate other sources. Tests such as the CCOW method of source evaluation can be helpful in determining whether the generated information you've found is reliable.

However, some of the questions we typically ask about sources may be more difficult to answer when working with generative AI, because AI tools do not disclose the processes used or the information consulted to arrive at the answers it provides.

Tests such as the SIFT method of source evaluation (or the Four Moves) can be helpful in determining whether the generated information you've found is reliable. 

Stop to think critically

Before sharing or using information, stop to check your emotional response to the information. Also be sure to consider:

• What do you already know about the topic?
• What do you know about the source? What is it's reputation?
• Could your search strategy be affecting the results you receive?

Investigate the source

According to digital literacy expert and creator of the SIFT method, Mike Caulfield, "you want to know what you’re reading before you read it" to make the most of your time when engaging in the research process. Knowing about the source can help you better understand the trustworthiness of the information. Take a moment to consider:

• Who is the author or creator? What do you know about them/the organization?
• What is the mission of the creator? Do they have specific motives for sharing information? 
• Does the author have authority in the area?

You can use lateral reading to investigate what other trusted sources say about the source you're interested in. Google and Wikipedia can be helpful for this step!

Find better coverage

Lateral reading can be helpful for this step, too. Look for other sources that 

• Can you find other coverage of information on the topic?
• Is there coverage that is more trusted, more in-depth, or maybe just more varied?

Trace claims, quotes, and media to their original context

It is good practice to try to locate the original source and context of quotes and information because much of what exists on the internet has been intentionally stripped of context. 

• Has the claim, quote, or media been fairly represented?
• Has information been cherry-picked to support an agenda?
• Has information been taken out of context?

Check out the video below for a more in-dept explanation of evaluating AI-generated content:

Being AI Literate does not mean you need to understand the advanced mechanics of AI. It means that you are actively learning about the technologies involved and that you critically approach any texts you read that concern AI, especially news articles. 

The ROBOT test is a tool you can use when reading about AI applications to help evaluate the legitimacy of a particular AI technology.

 
Reliability

• How reliable is the information available about the AI technology?
• If it’s not produced by the party responsible for the AI, what are the author’s credentials? Bias?
• If it is produced by the party responsible for the AI, how much information are they making available? 
  • Is information only partially available due to trade secrets?
  • How biased is they information that they produce?

Objective

• What is the goal or objective of the use of AI?
• What is the goal of sharing information about it?
  • To inform?
  • To convince?
  • To find financial support?

Bias

• What could create bias in the AI technology?
• Are there ethical issues associated with this?
• Are bias or ethical issues acknowledged?
  • By the source of information?
  • By the party responsible for the AI?
  • By its users?

Owner

• Who is the owner or developer of the AI technology?
• Who is responsible for it?
  • Is it a private company?
  • The government?
  • A think tank or research group?
• Who has access to it?
• Who can use it?

Type

• Which subtype of AI is it?
• Is the technology theoretical or applied?
• What kind of information system does it rely on?
• Does it rely on human intervention?

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

To cite in APA: Hervieux, S. & Wheatley, A. (2020). The ROBOT test [Evaluation tool]. The LibrAIry. https://thelibrairy.wordpress.com/2020/03/11/the-robot-test