Downs, S.H., & Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomized and non-randomized studies of health care interventions. Journal of Epidemiology Community Health, 52, 377-384.
Relevance for Public Health
Downs & Black (1998) do not explicitly refer to public health practices; however, they do state that the “Checklist for Measuring Quality” could be applied to any study detailing a health care intervention. Therefore, this instrument is applicable to the critique of studies involving public health interventions.
The “Checklist for Measuring Quality” (Downs & Black, 1998) addresses the increasing demand for the use of evidence from systematic reviews and meta-analyses to support program and policy decisions in public health decision-making. This tool can be used to assess the quality of original or primary source research articles and to synthesize evidence from quantitative studies for public health practitioners, policy makers and decision-makers. The “Checklist for Measuring Quality” is discussed in a three page article and contains 27 ‘yes’-or-'no’ questions across five sections. The tool is easy to use and provides both an overall score for study quality and a numeric score out of a possible 30 points. The five sections include questions about:
1. Study quality (10 items) – the overall quality of the study;
2. External validity (3 items) – the ability to generalize findings of the study;
3. Study bias (7 items) – to assess bias in the intervention and outcome measure(s);
4. Confounding and selection bias (6 items) – to determine bias from sampling or group assignment; and
5. Power of the study (1 items) – to determine if findings are due to chance.
Administration of the tool can happen either within a systematic review process, or as a quality assessment tool for individual articles. Within a systematic review, a group guides the review and one or two people administer the tool.
Accessing the Method/Tool
Access to full PDF article requires full registration for archived content of BMJ Journals.
Implementing the Method/Tool
Time for Participation/Completion
30 to 60 minutes
Downs & Black (1998) estimate that it will take reviewers approximately 20 minutes to assess one paper (range 10-45 minutes).
Additional Resources and/or Skills Needed for Implementation
The checklist requires appraisers with specialized expertise in quality assessment and application to health care interventions. Other resources may be necessary but are not described by the authors of the tool.
Steps for Using Method/Tool
Who is involved
The “Checklist for Measuring Study Quality” is completed by those individuals who are interested and able to critically appraise research studies for quality and applicability to public health. Practitioners, decision-makers and/or policy makers who require fast, synthesized public health oriented research for program planning and decision-making may find this tool useful.
Conditions for Use
Copyright © 1998 Journal of Epidemiology and Community Health.
Evaluation and Measurement Characteristics
Has been evaluated.
This instrument has received pilot testing and a second testing after revisions.
Validity properties meet accepted standards.
Downs & Black (1998) describe assessing face, content and criterion validity. Face and content validity were assessed by three experienced reviewers, including two senior epidemiologists and a medical statistician. Modifications to the checklist were made based on the feedback of these reviewers. Criterion validity for this instrument was tested by comparing the total scores of the tool with another tool used only for randomized controlled trials. The resulting correlation between the two instruments was high (r=0.90).
Reliability properties meet accepted standards.
Reliability testing was completed during the pilot testing phase and after development of the new, modified version of the tool. Internal consistency reliability scores were high (Cronbach alpha > 0.69) on all subscales, except for the external validity subscale (Cronbach alpha=0.54). Test-retest reliability scores were high for all subscales when randomized and non-randomized studies were assessed (r: 0.69-0.90). The correlation score for external validity was low (r=-0.37). Inter-rater reliability scores were high for most of the subscales (r>0.70) with the exception of external validity (r=-0.14). The variation across scores for the external validity subscale was thought to be related to the rating ability of reviewers who did not have health care backgrounds and who may not have appreciated the applicability of the interventions to health care.
University of London
Method of Development
The development of the “Checklist for Measuring Study Quality” was based on epidemiologic principles, reviews of study designs and existing tools for the assessment of randomized trials.
University of London
Room 34, Keppel St
London, UK WC1E 7HT
Phone: (020) 7927-2228
Fax: (020) 7580-8183
These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.