Tips for maintaining quality when designing surveys at pace

Policy details

Introduction

When designing a survey, we would normally conduct a lot of up-front research and pre-testing when developing new questions and respondent materials. This includes iterative rounds of qualitative testing, such as cognitive and usability testing.

However, at times the GSS needs to respond to urgent requests to gather data with little to no time for pre-testing.

In these instances, we must be sure to maintain the quality of our surveys, so we have published some top tips for doing this.

Please note: before commissioning a new survey or adding questions to existing surveys, make sure you have established a clear need and have checked the data you require is not available from other data sources.

Tip 1: Don’t forget what you know about best practice for questionnaire design

You may be given data requirements which are challenging to meet, and the time constraints for development may leave you feeling under pressure to compromise.

It is important that you draw on your existing knowledge and experience. Providing previous testing evidence during counter-discussions is a helpful way to steer the design back in the right direction.

Some quick hints include:

• avoid double-barrelled questions
• avoid leading questions
• avoid loaded questions
• use plain English and online readability checkers. – think to yourself, “How would I ask a friend this question?” – and use those conversational terms
• be consistent with phrases and words that government is using in order to make concepts more recognisable for respondents (Style.ONS and the GDS style guide can help with this)
• avoid absolutes in question wording
• remember to use five-point or seven-point Likert scale (depending on unipolar or bipolar concepts)
• keep sentences, question stems and response lists short
• avoid lengthy questionnaires on the same topic – this can lead to fatigue and poor data quality

Tip 2: Be guided by the Government Design Principles

Remember the Government Design Principles. We’ve highlighted the most relevant principles to fast paced development.

Do less

This is about sharing what has been developed. It is useful to see how others are tackling collecting data on similar challenging concepts and learning from them. It avoids duplication of effort and can inspire your designs. You may be able to recycle questions or wording for materials that have been pre-tested by working in this way.

This is for everyone

It is vital that we design inclusively to ensure that everyone in society can take part in our surveys. This applies to respondent communication materials and the questionnaires. We need everyone to be represented in our data.

This can be challenging, and it may not be wholly possible due to the pace of development. But keeping the principles of inclusive design in mind, drawing upon existing knowledge, and applying approved patterns will help to create accessible products.

Understand context

Think about the context and circumstances that people may be completing the survey in. Have empathy for them and the situation they may find themselves in. Be sensitive in your question design and communications. Always consider the ethics of what you are saying, asking and to whom.

Iterate and then iterate again

If you can add the questions or materials to a future cognitive testing round then do so. This will allow you to gather insights on the question performance. If the survey is continuous, you can consider making changes to the wording of future releases based on these findings.

Tip 3: Carry out questionnaire scenario testing

Ensure you carry out scenario testing before going live. As a research team, collectively think of different scenarios and parts of society then develop a testing specification for them. You can use personas if you have them or keep in mind previous test participants you’ve spoken to. This is particularly important when time constraints mean you cannot carry out cognitive testing.

Then, assign these scenarios to others to run through the questionnaire to ensure the flow and wording makes sense for their circumstances. Testing is also used to check that the questionnaire is free of errors before collecting data from the public.

Tip 4: Use what you already know

If you have established wording or other components that works well from other projects, then make sure you recycle them here. Likewise, avoid wording or designs that you know are not well understood.

This is why involving other qualitative researchers in the scenario and quality testing is useful. It acts as catch all exercise in absence of pre-testing with the public. They can make useful suggestions for edits to the wording and designs based on their insights.

Furthermore, the GSS Harmonisation Team develop and maintain a library of recommended questions and guidance on measuring a broad range of key concepts. If you are required to develop a new question from scratch, then check the library of harmonised principles or email GSSHelp@statistics.gov.uk as there may already be a high quality question which you can use with confidence.

Tip 5: Ask for data requirements rather than draft questions

It is often harder to deconstruct a draft question or try to interpret the research question than to simply ask for it directly. This will save you and them time; they can tell you what they need to know and then you can get to designing a good question to deliver on those needs.

Tip 6: Use the survey data to refine the survey

Once the survey is live, you can make use of survey data itself and other data (such as paradata) to understand if there are any issues. For example, you may see that one question experiences high drop off rates. This is an indicator that you may need to return to this question and rethink the design or its inclusion in the survey.

You can also explore:

• whether the routing and validation are working as expected – ideally, you’d detect these during the scenario testing, but it is worth running extra checks on the survey data
• free text box entries – if you can identify themes in the responses given, then consider creating a new response option in the next release, this will reduce the respondent burden and the data processing burden.
• the number of “don’t know” and “not applicable” options being selected  – this will help you detect possible issues

By exploring these sorts of things, you can make changes to the next release to improve the respondent experience and data quality.

Tip 7: Less is more

Keep the programming requirements and routing simple to help reduce risk, error and development time.

Complicated routing takes time to develop and test, which may risk the delivery of your rapid or pulse survey. Instead, try to keep things simple. Don’t forget, it may be possible to derive outputs from other questions in the survey post collection. This may help to manage requests for complicated questions. As the question designer, you can help analysts identify where this is possible.