That’s the title of a new working paper by Wahed Rahman, Jeff Bloem, and me in which we randomly place the module asking survey respondents about their assets either near the beginning (treatment) or at the end (control) of the survey to see whether the latter introduces classical (i.e., noise) or non-classical (i.e., bias) measurement error.
On average, we have a null finding. That is, whether we ask respondents about their assets early or late in the survey introduces neither classical nor non-classical measurement error. But we do find some interesting treatment heterogeneity in that respondents from larger households (i.e., households with more than four individuals) and with a low level (i.e., fewer than six years) of formal education tend to underreport assets when asked about them later in the survey.
One caveat: We are assuming that this is happening because of survey fatigue, and so that the “right” number of assets and the “right” value of those assets is given by respondents who are asked about them earlier. Unfortunately, we have no way of testing whether survey fatigue is the mechanism here. The fact that we find a null finding on average, combined with the fact that our survey was relatively short (i.e., 75 minutes) lends credence to the idea that survey fatigue is what is driving our sub-sample results.
Here is the abstract:
Social and economic policy and research rely on the accurate measurement of welfare. In nearly all instances, measuring welfare requires collecting data via long household surveys that are cognitively taxing on respondents. This can lead to measurement error, both classical (i.e., noisier responses) and non-classical (i.e., biased responses). We embed a survey ordering experiment in a relatively short survey, lasting just over 75 minutes on average, by asking half of our respondents about their assets near the beginning of the survey (treatment) and asking the remainder of our respondents about their assets at the end of the survey (control). We find no evidence that survey ordering introduces classical or non-classical measurement error in either the number of reported assets or the reported asset value in the full sample. But in sub-samples of respondents who (i) are from larger (i.e., more than four individuals) households, or (ii) have low levels (i.e., fewer than six years) of education, we find evidence of differential reporting due to survey ordering. These results highlight important heterogeneity in response bias which, despite the null effect in the full sample, can be meaningful. For example, for respondents from larger households, placing the asset module near the beginning of the survey leads to a 23 percent increase in the total reported asset value relative to placing the same module at the end of the survey.