Do you understand your survey non-response rates?
Response rates to traditional social surveys have been steadily dropping over the last few decades. This can pose a serious issue for those of us responsible for producing estimates of population level characteristics such as household income.
The threat
Falling response rates increase the risk of non-response bias, if there are fundamental differences between those people who complete a survey (responders) and those who don’t (non-responders).
Non-response bias is a particular type of data collection bias that typically happens when people are unwilling or unable to respond to a survey due to a characteristic or factor, that distinguishes them from those who do respond. This can mean that different groups may be over or under-represented in the resulting survey estimates.
Dealing with non-response bias
There are methods that can be used to make adjustments to the estimates, such as calibrating estimates to population totals, but bias can still exist if the factors that are influencing someone’s response to a survey are not captured by these auxiliary sources.
The best way to test for non-response bias, and therefore to understand whether you need to make any adjustment for non-response in your sample, is to compare characteristics of responders with non-responders. How do you do this though, with no survey data for the non-responders?
Census non-response link study
For the last five decades, the Office for National Statistics (ONS) has undertaken a census non-response link study alongside the ONS Census for England and Wales. This study has linked data from social surveys administered around census day, with census data, to allow for a test of potential non-response bias in the social survey data.
We link the addresses of responding and non-responding households in the social survey datasets to the same addresses in census dataset. We are then able to use the rich census data to explore the demographic characteristics that explain the profiles of responders and non-responders.
This allows us to assess whether any bias has been introduced into our survey estimates, where we can compare variables that are in common between the census and the survey. It also allows us to evaluate the effectiveness of our current non-response models and use the detailed census data to update these models where needed.
Linking social survey and census data can help with a more bottom-up approach, too: by identifying those households more likely to be defined by non-response, we can develop optimal data collection strategies to target these households for the next sampling phase.
Data linkage project for the 2021 Census
We are again planning to undertake a social survey and census data linkage project alongside the 2021 Census, with a set of ONS social surveys identified for inclusion in the first phase. We intend to undertake analysis on non-response within these social surveys and, also, analysis of mode effects for the common core of questions between the online-first 2021 Census and the social surveys.
These analyses will allow us to consider our non-response models and inform our future transformation of social surveys. The linked dataset will also be a valuable resource for our colleagues in the census team, as part of the evidence base for understanding census coverage and quality.
Do you know a GSS survey that could benefit?
For the second phase of the project, planned to begin mid-autumn 2021, we’re interested in whether GSS social surveys could benefit from a similar approach and would like to work with you to understand how such a study could be used.
If you would like to learn more about the ONS social surveys to census data linkage project or have a GSS survey of your own you would like to discuss, please contact methodology@ons.gov.uk (please put ‘Census Non-Response Link Study – CNRLS’ in the subject line).
Katie O’Farrell