Download data and study materials from OSF
University of California, Berkeley
Amy E. Lerman
University of California, Berkeley
University of California, Santa Barbara
Sample size: 3510
Field period: 06/25/2019-03/15/2020
Governments, interest groups, and the mass media routinely use quantitative information to depict the severity of problems facing the nation. When doing so, they can choose to report the (unacceptably low) percentage of people experiencing good outcomes—as in the percentage of high school students obtaining the degree—or the (unacceptably high) percentage of people experiencing bad outcomes—as in the percentage of high school students dropping out. Likewise, policy initiatives to improve matters can be framed as increasing the incidence of good outcomes or as decreasing the incidence of bad ones. Despite the ubiquity of these rhetorical choices in political discourse, there is no research into whether they matter to public opinion. Existing studies of negativity bias, loss aversion, and attribute framing do, however, support the expectation that they will matter.
We carried out two sets of experiments to open up this line of inquiry. Each set consists of four experiments focusing, respectively, on drinking water quality, hospital deserts, high school achievement, and marine debris. The first set presents problem information, randomizing whether it is framed negatively or positively, and the second presents a policy solution, again with the frame randomized. All four experiments within each set showed clear and consistent results. When problems were negatively (vs. positively) framed, people expressed more anger and worry, and judged the problem more serious and more worthy of government attention. The public’s reaction to policy solutions, by contrast, were utterly unaffected by the frame.
In this study, respondents were given either a problem or a policy question. The problems and policies were assigned randomly from a list of 4, which included:
The chance of surviving a life-threatening medical emergency can depend on how long it takes to get to a hospital. According to a recent report, [20% / 80%] of Americans live [too far from / close enough to] a hospital to get adequate care in case of an emergency.
The policy experiment treatments first introduced the problem and then described a policy initiative designed to ameliorate it. In the negative condition, the problem was described in terms of the prevalence of bad outcomes (20%) and the policy was portrayed as reducing that from 20% to 10%. In the positive condition, the problem was described in terms of the prevalence of good outcomes (80%) and the policy was portrayed as increasing that from 80% to 90%. To illustrate, this is the text of the Hospital Deserts conditions.
The chance of surviving a life-threatening medical emergency can depend on how long it takes to get to a hospital. According to a recent report, [20%/80%] of Americans live [too far from/close enough to] a hospital to get adequate care in case of an emergency.
Representatives in Congress have proposed a policy that would decrease the number of Americans that live [too far from / close enough to] a hospital to get adequate care. The policy proposal calls for a coordinated effort across the 50 states, the creation of public-private partnerships, and a combination of tax incentives, regulatory reforms and outreach initiatives. The price tag is estimated to be $10 billion/year. The policy is expected to [decrease / increase] the percentage of Americans living [too far from / close enough to] hospitals from [20% to 10% / 80% to 90%] by 2025.
Overall, the design was a 4x2x2 factorial—questions regarding four topic areas, each framed either negatively or positively, and either focused on reactions to problems or reactions to policy solutions. Respondents were randomized into one of the 16 possible conditions.
Problem Dependent Variables
For each of the dependent variables shown below, the order of the response options was randomized (e.g. Not angry – Extremely angry/Extremely angry – Not angry). The order in which the four questions were asked was also randomized.Anger
Policy Dependent Variables
For each of the dependent variables shown below, the order of the response options was randomized (e.g. Strongly Disapprove to Strongly Approve vs. Strongly Approve to Strongly Disapprove). The order in which the questions were presented was fixed, as given below.Opinion
In addition to the core problem or policy DV questions, we also asked a standard set of demographics, plus questions concerning party identification. We also asked questions to measure two potential moderators: Risk Attitude and Need for Cognition. These questions were asked at the start of the survey, prior to the experiment. After the last survey question, we offered an optional but full debriefing. The debriefing page included study information as well as source information and links for each of the problem depictions.
Results were in the expected direction across all four experiments and all four dependent variables (DVs). Negatively framed problems tended to generate more anger and worry, and to be judged more serious and more worthy of government attention. In 10 of the 16 comparisons (four issues, four DVs each), the treatment effect was substantively and statistically significant (p<.05, two-tailed). If the data for the four experiments were stacked and analyzed with fixed effects for problem area and DV order randomizations, treatment effects were significant for all four dependent variables at p<.001. However, the effect size was significantly larger for Anger (b=.66, with Y scored 0-1, beta=.17) than for Worry (b=.44, beta=.12), Seriousness (b=.44, beta=.13), and Government Priority (b=.26, beta=.08). There was some evidence of anger carry-over effects, in that the effect sizes for Worry, Seriousness, and Government Priority were larger if the respondent was first asked about the extent to which the problem made him or her angry. Effect sizes were not moderated by Need for Cognition or Risk Attitude. They were larger among college-educated respondents (vs. the less educated) and among political Independents (vs. among Democrats or Republicans), but these differences did not reach conventional levels of statistical significance.
Results were consistent across all four experiments and all three dependent variables. In no case was there a substantively significant difference in opinions by frame, and 11 of the 12 tests of framing effects (four issues, three DVs each) yielded a p-value greater than .40. Moreover, we found no evidence of significant effects among subgroups defined by potential moderators (Need for Cognition, Risk Atittude, Education, Age, or Partisanship).