To enable the GSS to effectively carry out all aspects of Quality Management the following tools have been developed to aid producers in their production of official statistics.
Process for Producing Outputs
Process followed by statistical producers when producing outputs: Generic Statistical Business Process Model (GSBPM).
- GSBPM is the process followed by ONS to produce statistical outputs. In autumn 2012, the ONS Design Authority made the decision to adopt GSBPM terminology for its business processes which brought us in line with other National Statistical Institutes (NSIs).
- GSPBM (previously known as the Statistical Value Chain (SVC)) was produced by UN/OECD/Eurostat. It ensures the same terminology is used by all when referring to statistical business processes. A GSBPM revision Task Force was initiated in 2013 with representatives from a range of NSIs, including ONS. There are limited changes to the model itself, but many more improvements, additions and clarifications to the supporting documentation.
Risk Assessment (Stage 1)
An approach for producers to identify, assess and address potential risks with their statistical outputs.
- Carrying out risk assessment of statistical outputs is an essential part of identifying, prioritising and addressing risks. This approach is known in industry as “Value Engineering” and can be used to assign a red to green status for statistical outputs, providing a high level picture of the risks. These assessments enable producers to gain a strong view on strengths and vulnerabilities to help prioritise strategic developments including process improvements, methods reviews and investment in systems.
- It is expected that risk assessment should be carried out as part of developing a new statistical output, together with being a regular yearly process for existing official statistics.
- Risk assessment is also carried out for the following reasons:
- to have a broad picture of where the highest risks are in terms of outputs to assist in prioritising / directing resources for improvements in methods, processes and systems
- to inform the prioritisation of quality and methods reviews
- to provide a top down approach to planning/bidding for any future systematic programme of improvements
- to provide a strategic approach to identifying process improvement initiatives.
Quality Reviews (Stage 2)
Provides information and guidance on carrying out high level quality reviews of statistical outputs.
- Establishing user requirements
- Use of administrative data or data sourced from other organisations
- Classifications and harmonised standards
- Specific focuses on methodological areas such as editing and imputation, weighting and estimation, index number construction, confidentiality etc.
- Carrying out statistical quality reviews are important for continuing to ensure official statistics are fit for purpose, whether an issue has been previously identified or not. Reviews are part of the cycle of improvement, are a way to check compliance with the Code and are an extremely useful component of a wider approach to quality management. This means that both users and producers can be assured of the continued quality of our statistics.
- One of the main approaches to carrying out quality reviews in the GSS has been through the use of the Quality Methods and Harmonisation Tool (QMHT), a self assessment questionnaire designed to help official statistics producers review their surveys and outputs. As the title suggests, the tool is mainly focussed on quality, methods and harmonisation, not necessarily the process of production itself. It has 15 areas of focus which roughly follow the key stages of the Generic Statistical Business Process Model (GSBPM) , a way of ensuring consistency in the approach to producing official statistics. Sections include:
- QMHT is very detailed, with a particular focus on statistical processes. As such it can be used both as a complete tool and as a way of focussing on a specific area of known concern.
In-Depth Quality Reviews (Stage 3)
Provides information and guidance on carrying out in-depth quality reviews. This includes focuses on specific methodology, bringing in external experts and carrying out National Statistician Quality Reviews (NSQRs).
- Statistical methods:
- Whether they are consistent with scientific principles and internationally recognised good practice.
- Whether they are prepared for emerging methodological and data developments.
- Conceptual issues – broaded conceptual issues such as whether the statistics are measuring the right thing, and whether they are prepared for emerging methodological and data developments.
- Assurance – Assurance to users and key stakeholders is essential. If there is significant risk then it is often important to involve an independent external expert in the process.
- In-depth quality reviews should be used where significant risks or issues have been identified to an official statistics product. These risks can represent a number of issues (both statistical and process related) but they will be high profile, high impact risks where the need for intervention outweighs any associated costs (both in terms of time and resource). These risks can be real or perceived but are likely to have a significant impact on reputation if not addressed.
- National Statistics Quality Reviews (NSQRs) have been designed for when there is significant identified risk to an official statistics product and when the need for investment outweighs the associated costs. It is recommended that NSQRs are carried out on a single output or set of outputs which either use similar methodologies or are viewed collectively. Although official statistics producers are not required to follow the process set out for NSQRs, the normal risks where an NSQR might be considered can relate to:
- NSQRs should be carried out as a project and in stages. Planning should set out how the project will be managed including roles and responsibilities, project boards, timescales, costs and scope. A process similar to that set out above should then be followed. External experts should be involved in all aspects of the review, although their role may be more direct in some stages rather than others.
Provides guidance on how to quality assure statistical products.
- Quality Assurance is about anticipating and avoiding problems. It covers all procedures that aim to ensure quality requirements are met and that problems are anticipated. It requires processes and systems that are planned and tested to perform under all conditions, and self-correct or flag problems when they arise. The goal of quality assurance is to prevent errors in a statistical product and to get them right first time.
Regular Quality Reviews (RQR)
A facilitated meeting involving senior methodologists and output managers discussing methods and quality of an output” and include the following information:
- An RQR consists of a facilitated meeting, lasting between 1.5 and 2 hours, between the manager of a statistical output, a senior methodologist and a representative from Quality Centre. The meeting makes use of existing documentation, which forms the basis for a discussion of methods and quality, led by the senior methodologist. This review of methods covers all relevant stages of Generic Statistical Business Process Model and all five of the European Statistical System (ESS) dimensions of output quality (Eurostat, 2014). The collation of the documentation for the meeting, the correspondence and the scheduling of the meetings are all handled by Quality Centre.
- Each RQR is different; they are tailored by the senior methodologist to the individual statistical output. They ask questions of the output manager during the meeting and develop a set of recommendations for that statistical output. This means that the recommendations that are made by the senior methodologist are bespoke; this ensures that they are relevant to the statistical output
Guidance on how to carry out a Regular Quality Review
1. Identify participants and arrange for the meeting to take place
The meeting should involve the manager responsible for the statistical output and a senior methodologist or statistician that will lead the review. Although it can be useful to have someone to facilitate the meeting, this would not be a requirement. Where departments are small, the senior methodologist or statistician could come from another department or a different part of the organisation.
2.Collate background documentation
The information that forms the basis of the meeting should be compiled and circulated to all those involved in the meeting ahead of time. This should include any methodological articles or quality reviews.
3. Senior methodologist studies background information and identifies
The senior methodologist or statistician carrying out the review needs time to study the background material and to develop a set of questions in readiness for the review. Up to one day of time is set aside for this stage to be completed.
4. The senior methodologist or statistician leads the review
The meeting takes place and the senior methodologist or statistician works through the questions that have been prepared. Sometimes responses may need to be provided after the meeting. The success of the review depends on all parties being open and transparent about the practices and methods that are in place. The meeting should be an open forum; it should not be considered as an interview or a test. There are no right or wrong answers; the aim is simply to get the best value out of the recommendations that are made.
5 Recommendations made within a report and RAG status assigned
The senior methodologist or statistician then writes their recommendations in a report, which is sent back to the output manager. The output manager has a right to reply to ensure that the information recorded is accurate. The recommendations can be prioritised. This has proved to be a useful approach where many outputs are reviewed, both to monitor cross-cutting recommendations but also to ensure consistency when multiple reviewers are involved. A RAG status is applied to the statistical output which reflects the discussions and the recommendations made.