As we have all seen and heard, there is a global rise in the spread of misinformation on social media, and there has been some behavioural research on “fake news.” What is fake news, you ask? They are false news headlines that are presented as if legitimate.
These studies often present participants with news content that varies on dimensions like true v. false, politically consistent v. inconsistent, etc. and ask participants to make judgments for accuracy or choices.
This guide is intended to help researchers navigate the unique challenges that come with this type of research. One of the main issues is that content being spread on social media is a moving target that reflects current affairs. We must consider that the selection of the content to include can be highly consequential for the study’s outcome, and researcher biases can result in biases in a stimulus set. We advocate for pretesting materials report our own pretest of 225 recent true and false news headlines relating to U.S. political issues and the COVID-19 pandemic.
Step 1 – Selecting your misinformation
When one refers to fake news, it is actually a reference to whatever happens to exist as the present form of misinformation occurring in the world. If you wish to create your own fake news stimuli (as has been done in some studies; Pereira et al., 2020), you need to take extra care to make sure that your content is similar to what is seen in everyday life.
Creating your own headlines means that you don’t know if they are t he sort that would actually spread on social media. You should get a sense of what fake news looks like in the context that you are interested in. For example, if you would like to run a study on misinformation in Brazil, it would not make sense to use examples of fake news headlines from the United States.
Fortunately, there are numerous fact-checking organizations around the world that keep records of false or misleading content that is spread on the internet. Some of them are below.
Many falsehoods on these websites are not examples of “fake news” again; it is up to you to decide what type of misinformation you would like to research.
Step 2 – Selecting true content
Determining how people distinguish between true and false content is very important but if you’re only interested in testing misinformation without “true content,” then skip this step. Finding misinformation is easy as there is a clear way to discover “misinformation” that fact-checkers have identified as false. There is a much larger world of possibilities with true content, and there are reliable mainstream sources such as The New York Times or Washington Post.
When searching for content, it is also important to sample as many different sources as possible. Our preferred approach is to select headlines that would make sense for someone to share several months after the headline was published because they describe events that could happen.
Step 3 – Complete a pretest
A pretest is optional, but if one is set, you can select 20 headlines that are Pro Democratic and 20 that are Pro Republican; the Pro Democratic headlines may end up being more Pro Democratic than the Pro Republican headlines are Pro Republican (or vice versa), depending on who selects them (or even random chance).
The vast majority of the empirical studies on fake news that we have published have been backed up with pretesting, which requires extra time and money.
Step 4 – Creating your experiment
It is time to create your experiment. The selection of your content is absolutely central to the validity of your study. For this study, it was essential to have headlines that were highly political as those are the type of headlines that should facilitate partisan bias. We, therefore, used a pretest to confirm this. We found that using more than 30 headlines is probably too much, and we suggest running smaller batches of participants initially. You can randomize across a large set of headlines. If you are committed to studying fake news, then this may be a good strategy
Conclusion
It is necessary for social scientists to form a better understanding of why and how falsehood has spread so easily on social media platforms. Fortunately, there are steps that can be taken to build robust surveys and experiments using “real world” examples of news content. If we can build a collective knowledge base around the issue, it will be easier for meaningful interventions to impact the quality of content that people share on social media
Sources
