Improving Measurement of Self-Reported Health: Effects of Question Context and Response Order

Download data and study materials from OSF

Principal investigator:

Dana Garbarski

University of Wisconsin-Madison

Email: dgarbars@ssc.wisc.edu

Homepage: www.danagarbarski.com


Sample size: 2696

Field period: 09/29/2012-02/21/2013

Abstract
The self-rated health question ("would you say your health in general is excellent, very good, good, fair, or poor?") is widely used to study health because of its ability to predict morbidity and mortality. However, it is unclear from prior research how the distribution of answers to self-rated health (SRH) and its association with other health measures depend on (1) whether SRH is administered before or after more specific health-related items or (2) the order in which the response options are presented.
SRH is often asked before other questions about health and disease, following the common survey practice of placing more general items before more specific items. However, preceding SRH with health-related questions may define and stabilize the meaning of "health" as a summarizing question. Results are mixed for the few studies that examine whether SRH's distribution is affected by being placed before or after other health items. Further, no study examines whether the correlation between SRH and other health items changes depending on SRH's placement.
Because research indicates respondents often select the first option they find acceptable, question writers often suggest beginning with the least desirable response option. However, SRH is overwhelmingly administered beginning with "excellent," the most positively valenced option, and little research examines whether this ordering affects responses.
We analyze results from a 2-by-2 factorial experiment. We examine several outcomes including effects on response distributions and correlations with other health measures.

Hypotheses
Hypothesis 1: Previous research on the ordering of response options finds that respondents are likely to choose the first response option that is perceived to be acceptable. As a result, we expect that mean levels of self-rated health will be lower when the response options are ordered from "poor" to "excellent" compared to when they are ordered from "excellent" to "poor."
Hypothesis 2: We expect that administering SRH after domain-specific health questions primes respondents with specific information about health that communicates the meaning of the self-rated health question as a summary measure. Thus, we hypothesize that the associations between SRH and each of the domain-specific health items will be stronger when SRH follows the more-specific health items, consistent with assimilation effects.
Hypothesis 3: We expect question order effects will depend on the respondent's health status: those in better health will have more positively rated health when SRH follows a list of domain-specific health questions compared to when SRH precedes such a list, while those in worse health will have more negatively rated health when SRH follows a list of domain-specific health questions compared to when SRH precedes domain-specific health questions.
Hypothesis 4: We expect that the effects of question order and response option order on SRH vary across sociodemographic and prior health covariates.

Experimental Manipulations
The experiment follows a 2-by-2 factorial design in which participants are randomly assigned to one of two levels for each factor. For the first factor, the response categories are ordered as "excellent, very good, good, fair, or poor" versus "poor, fair, good, very good, or excellent." For the second factor, the administration of SRH either precedes or follows the administration of a set of domain-specific health questions.

Outcomes
Self-rated health

Summary of Results
This study documents how question order and response option order work independently and together to predict self-rated health. We find that self-rated health will be worse (that is, more evenly distributed across the response categories rather than clustered toward excellent) when the response options are ordered from "poor" to "excellent" compared to when they are ordered from "excellent" to "poor." We also find evidence that placing SRH after domain-specific health items is subject to an overall assimilation effect. There are conditional context effects in which respondents who have worse health adjust their assessment of their health downward after being reminded of the various domains in which their health is not good; these effects are more pronounced for experimental conditions in which the response options are ordered from "excellent" to "poor." Finally, the effects of question order and response option order depend on at least two sociodemographic covariates: marital status and education.

References

Gabarski, Dana, Nora Cate Schaeffer, and Jennifer Dykema. 2014. "The effects of response option order and question order on self-rated health." Quality of Life Research 24(6): 1443-1453.

Garbarski, Dana, Nora Cate Schaeffer, and Jennifer Dykema. 2016. "The effect of response option order on self-rated health: a replication study." Quality of Life Research.