If you pay attention to news about science or social media, then chances are you’ve already heard about how back in 2012 Facebook carried out a giant psychological experiment on hundreds of thousands of its users without informing them.
The goal of the experiment was to learn more about how a person’s mood is affected by the types of posts they see in their News Feed. And so for one week, Facebook subtly manipulated what appeared in users’ News Feeds and then analyzed the emotional tone of those users’ subsequent status updates.
Needless to say, public furor ensued.
Since the findings of the study were published in the June 2014 issue of Proceedings of the National Academy of Sciences (PNAS), reactions to the now famous “Facebook mood-manipulation experiment” have ranged from suspicion about Facebook’s true motives for conducting the research to anger over whether the experiment violated ethical guidelines for conducting research on human participants.
Given the attention this story has received in the news and on social media, I decided to take a close look at the actual PNAS paper.
So in this post, I’ll describe the methods and findings of the Facebook experiment and offer a critical assessment of what I see as being the major implications of this study.
Let’s begin with a brief rundown on what actually happened back in 2012.
What actually happened in Facebook’s “Mood-Manipulation Experiment?”
From January 11-18, 2012, members of Facebook’s Core Data Team collaborated with researchers from the University of California, San Francisco and Cornell University to conduct an experiment on a psychological phenomenon called emotional contagion.
Emotional contagion is the idea that some emotions – such as happiness, sadness, and anger – behave like infection diseases, spreading from person to person as though they are actually contagious.
What the researchers involved in the Facebook experiment wanted to know was whether emotions can spread from person to person even in the absence of direct human contact. If so, they reasoned that seeing positive emotional posts in your Facebook News Feed (e.g., seeing a friend post about being excited to start a new job) should lead you to experience a more positive mood, whereas seeing negative emotional posts in your News Feed (e.g., seeing a friend post about being annoyed that they have to work on Saturday) should lead you to experience a more negative mood.
So the researchers randomly selected 689,0003 Facebook users based on their user ID and for one week covertly manipulated the number of positive and negative posts that appeared in their News Feeds.
Positive posts were defined as those that contained at least one positive emotion word (e.g., love, happy, etc.), and negative posts were defined as those that contained at least one negative emotion word (e.g., hate, angry, etc.).
During the experiment, whenever users logged on to Facebook they were randomly assigned to either an “experimental condition” or a “control condition.” During the experimental condition, either positive or negative posts were omitted from a user’s News Feed. In the control condition, a similar percentage of posts were omitted from a user’s News Feed entirely at random (i.e., not based on emotional content).
How did they assess people’s mood?
The researchers assessed the mood of each user in the study by looking at the number of positive and negative words contained in their status updates following the mood manipulation procedure.
Status updates containing a greater number of positive emotion words were assumed to reflect a positive mood, whereas status updates containing a greater number of negative emotion words were assumed to reflect a negative mood.
To maintain compliance with Facebook’s Data Use Policy, status updates were collected and coded via an automated data collection and filtering system so that no text was actually ever seen by any of the researchers.
What did the experiment find?
The researchers found that when positive posts were omitted from News Feeds, Facebook users went on to post status updates that contained fewer positive words and more negative words, suggesting the experience of a more negative mood. The opposite occurred when negative posts were omitted from News Feeds – Facebook users went on to post status updates that contained fewer negative words and more positive words, suggesting the experience of a more positive mood.
Figure 1 from the original paper is presented below, showing the main results of the study.
So the findings of this experiment suggest that we unwittingly adopt the same emotional state we see others convey online.
When we see friends share posts expressing positive emotion on Facebook, we tend to follow along and share positive status updates of our own. Unfortunately, the reverse is also true. Seeing friends share posts expressing negative emotion on Facebook leads us to join in and share status updates conveying negative emotion as well.
Now that I’ve described the methods and major findings of Facebook’s mood-manipulation experiment, let’s move on to a critical assessment of what these findings actually mean and what their implications might be.
Are the findings of the Facebook experiment important, and should we worry about Facebook controlling our moods?
As I see it, there are two ways of looking at the results of the Facebook study to gauge how important and relevant they are to the real-world.
The first is to consider the size of the effect of Facebook’s News Feed manipulation on people’s moods, and the second is to think about the number of people whose behavior and mood might be affected by seeing fewer positive posts (or fewer negative posts) in their News Feed.
The latter approach is especially important considering the enormous size of Facebook, currently estimated at 1.28 billion users worldwide.
1. How Large is the Effect of Facebook’s Newsfeed Manipulation on People’s Moods?
As it turns out, even though reducing the number of positive and negative posts in users’ News Feeds did indeed have a statistically significant effect on mood (again, as measured by the number of positive and negative words people used in subsequent status updates of their own), the size of the effect was extremely small.
How small exactly?
I’ll demonstrate in two different ways:
A) First, consider that psychological scientists generally classify effect sizes in their research (symbolized here by the letter d) as follows:*
Small Effects: d = 0.2 or lower
Medium Effects: d = around 0.5
Large Effects: d = 0.8 or higher
Effect sizes reported in the Facebook study meanwhile ranged from a puny 0.001 to a slightly less puny 0.02! In fact when positive posts were omitted from News Feeds, the number of positive status updates from users decreased by less than half of 1% (0.10% specifically).
So removing positive and negative posts from a person’s News Feed doesn’t just have a small effect on mood. It has a miniscule effect.
*This is the conventional classification of a specific measure of effect size called Cohen’s d, which is the measure of effect size that was used in the Facebook study. Note that an effect size of d = 0 would mean an experimental manipulation had absolutely no effect on behavior.
Need more convincing? Consider the following hypothetical scenario:
B) Assuming your News Feeds was manipulated so as to omit positive posts, what’s the probability that you would show signs of a worse mood than someone whose News Feed had not been manipulated?
Given the largest effect size reported in the paper (d = 0.02), the answer is 50.56% – essentially a coin toss! *
*Known as the common language effect size (CL), this estimate was derived from Cohen’s d (the measure of effect size used in the Facebook study) using the following formula: CL = Φ (δ/√2), where Φ is the cumulative distribution function of the standard normal distribution, and δ is the value for Cohen's d.
So although manipulating the number of positive and negative posts that appear in a person’s News Feed certainly does affect mood (or at least how a person chooses to express their mood online), the size of the effect is tiny, and probably too small to be of much concern to a single individual.
So there’s no reason to lose sleep over the fear that Facebook might be controlling your mood by manipulating the contents of your News Feed!
In fact, if you were one of the 689,003 people in the Facebook experiment, there would only have been a 0.8% chance that you would have shown signs of a negative mood after having positive posts omitted from your News Feed.*
*This number was calculated as follows: [Estimated number of people affected by the News Feed manipulation] / 689,003. See below for details on calculating Estimated number of people affected by the News Feed manipulation.
Importantly, the authors of the Facebook study fully acknowledge that omitting positive and negative posts from users’ News Feeds has only a tiny effect on mood.
Nonetheless, they conclude their paper by saying the following:
“Although these data provide, to our knowledge, some of the first experimental evidence to support the controversial claims that emotions can spread throughout a network, the effect sizes from the manipulations are small (as small as d = 0.001). These effects nonetheless matter given that the manipulation of the independent variable (presence of emotion in the News Feed) was minimal whereas the dependent variable (people’s emotional expressions) is difficult to influence given the range of daily experiences that influence mood (10). More importantly, given the massive scale of social networks such as Facebook, even small effects can have large aggregated consequences (14, 15): For example, the well-documented connection between emotions and physical well-being suggests the importance of these findings for public health. Online messages influence our experience of emotions, which may affect a variety of offline behaviors. And after all, an effect size of d = 0.001 at Facebook’s scale is not negligible: In early 2013, this would have corresponded to hundreds of thousands of emotion expressions in status updates per day.”
*bolded text added by me
The portions of this text that I’ve highlighted raise a very interesting question, and point to the need to evaluate the importance of this research a second way, besides simply considering effect size as we’ve already done.
We need to think about the number of people whose behavior and mood would be affected by seeing fewer positive and negative posts in their News Feed, something that is especially important given the enormous size of Facebook.
2. Given the present size and scale of Facebook, how many people might be affected by seeing fewer positive posts (or fewer negative posts) in their News Feed?
Even though the contents of your News Feed has just a tiny effect on how you choose to express your mood online (as already discussed above), decreasing the number of positive and negative posts that show up in the News Feeds of a large enough group of users could affect the behavior of a tremendous number of people, given the enormous size of Facebook.
But how many people exactly?
Assuming the number of positive and negative posts in a person’s News Feed has just a small effect on how a person chooses to express their mood online (Cohen’s d = 0.02, as reported in the study) and that the baseline probability of posting a positive status update on Facebook is 46.8% (based on the finding reported in the Facebook study that 46.8% of posts contained positive words before the start of the experiment), I estimate that if positive posts were omitted from the News Feeds of a quarter of all Facebook users (25% of 1.28 Billion = 320,000,000), this would lead to a subsequent decrease in the number of positive status updates from 2,546,887 people worldwide.*
*This number was calculated as [Number of FB users with positive posts reduced in their News Feeds (320,000,000)] / NNT. NNT refers to the number of patients one would need to treat in an experiment to produce one more favorable outcome in the treatment group compared to the control group. It was calculated using the following formula devised by Furukawa and Leucht (2011): NNT = 1 / [Φ (δ – Ψ (CER)) – CER]; where Φ is the cumulative distribution function of the standard normal distribution and Ψ is its inverse, CER is the control group's event rate (in this case, the baseline probability of posting a positive status update to Facebook or 0.468) and δ is Cohen's d (in this case, 0.02). NNT was determined to be 125.64.
And even though that’s only a tiny fraction of the group of people whose News Feeds would have been manipulated (0.8% of the original group of 320 million), let’s make sure we keep this number in perspective.
We’re talking about 3.29x the population of Charlotte, NC and 26% of the entire population of North Carolina.
That’s a lot of people posting fewer positive status updates on Facebook.
And although it might not spell disaster or the end of the world, one has to wonder about the real-world, offline consequences if 2.5 million people suddenly started sharing with friends and family on Facebook a slightly more jaded and pessimistic outlook on life.
So are the findings from the Facebook experiment important and relevant to the real-world?
I would argue the answer is yes.
People are more closely connected now than ever before, thanks to the internet, social media, and Facebook in particular. As such, the average individual’s sphere of influence is almost certainly larger today than at any other time in human history. With email, Twitter, and Facebook, we now have the capability to influence the opinions, feelings, and behaviors of people even thousands of miles away from us, and to do so almost instantly in real time.
So even though reducing the number of positive and negative posts in a person’s News Feed has only a tiny effect on an individual’s mood, tiny effects on the behavior of individuals can potentially add up to effects of great consequence when implemented in a large and highly integrated social network, especially one that comprises close to 14% of the world’s population.
Furukawa, T. A., & Leucht, S. (2011). How to obtain NNT from Cohen’s d: comparison of two methods. PloS one, 6(4).
Kramer, A.D., Guillory, J.E., & Hancock, J.T. (2014). Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, 111 (24), 8788-8790.
For a guide to different measures of effect size related to Cohen’s d, see this excellent interactive visualization by Kristoffer Magnusson.