Relevance for Public Health

Downs & Black (1998) do not explicitly refer to public health practices; however, they do state that the “Checklist for Measuring Quality” could be applied to any study detailing a health care intervention. Therefore, this instrument is applicable to the critique of studies involving public health interventions.

Description

The “Checklist for Measuring Quality” (Downs & Black, 1998) addresses the increasing demand for the use of evidence from systematic reviews and meta-analyses to support program and policy decisions in public health decision-making. This tool can be used to assess the quality of original or primary source research articles and to synthesize evidence from quantitative studies for public health practitioners, policy makers and decision-makers. The “Checklist for Measuring Quality” is discussed in a three page article and contains 27 ‘yes’-or-'no’ questions across five sections. The tool is easy to use and provides both an overall score for study quality and a numeric score out of a possible 30 points. The five sections include questions about: 1. Study quality (10 items) – the overall quality of the study; 2. External validity (3 items) – the ability to generalize findings of the study; 3. Study bias (7 items) – to assess bias in the intervention and outcome measure(s); 4. Confounding and selection bias (6 items) – to determine bias from sampling or group assignment; and 5. Power of the study (1 items) – to determine if findings are due to chance. Administration of the tool can happen either within a systematic review process, or as a quality assessment tool for individual articles. Within a systematic review, a group guides the review and one or two people administer the tool.

This instrument has received pilot testing and a second testing after revisions.

Downs & Black (1998) describe assessing face, content and criterion validity. Face and content validity were assessed by three experienced reviewers, including two senior epidemiologists and a medical statistician. Modifications to the checklist were made based on the feedback of these reviewers. Criterion validity for this instrument was tested by comparing the total scores of the tool with another tool used only for randomized controlled trials. The resulting correlation between the two instruments was high (r=0.90).

Reliability testing was completed during the pilot testing phase and after development of the new, modified version of the tool. Internal consistency reliability scores were high (Cronbach alpha > 0.69) on all subscales, except for the external validity subscale (Cronbach alpha=0.54). Test-retest reliability scores were high for all subscales when randomized and non-randomized studies were assessed (r: 0.69-0.90). The correlation score for external validity was low (r=-0.37). Inter-rater reliability scores were high for most of the subscales (r>0.70) with the exception of external validity (r=-0.14). The variation across scores for the external validity subscale was thought to be related to the rating ability of reviewers who did not have health care backgrounds and who may not have appreciated the applicability of the interventions to health care.

The “Checklist for Measuring Study Quality” is completed by those individuals who are interested and able to critically appraise research studies for quality and applicability to public health. Practitioners, decision-makers and/or policy makers who require fast, synthesized public health oriented research for program planning and decision-making may find this tool useful.

Downs & Black (1998) estimate that it will take reviewers approximately 20 minutes to assess one paper (range 10-45 minutes).

The checklist requires appraisers with specialized expertise in quality assessment and application to health care interventions. Other resources may be necessary but are not described by the authors of the tool.

Not specified.

Copyright © 1998 Journal of Epidemiology and Community Health.

This tool was developed by Sara H Downs and Nick Black at the Health Services Research Unit, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, University of London, UK.

The development of the “Checklist for Measuring Study Quality” was based on epidemiologic principles, reviews of study designs and existing tools for the assessment of randomized trials.

Dr. Nick Black Professor, London School of Tropical Medicine, University of London Room 34, Keppel Street, London WC1E 7HT, UK Tel: 020 7927 2228 Fax: 020 7580 8183 Email: nick.black@lshtm.ac.uk

Title of Primary Resource

File Attachment

Web-link

Reference

Downs, S.H., & Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomized and non-randomized studies of health care interventions. Journal of Epidemiology Community Health, 52, 377-384.

Type of Material

Format

Cost to Access

Language

Conditions for Use