National Collaborating Centre for Methods and Tools

Relevance for Public Health

This instrument was developed for use in public health, and can be applied to articles of any public health topic area. Some of these areas include the promotion of family and sexual health and the prevention of chronic disease, injuries and substance misuse. Various types of public health professionals would find this tool relevant to utilize sources of high quality literature to support the decision-making process, especially when designing, implementing and evaluating public health programs and policy.

Description

The “Quality Assessment Tool For Quantitative Studies” developed by the Effective Public Health Practice Project (EPHPP) is a tool for doing knowledge synthesis. This instrument, along with a user manual, provides a standardized means to assess study quality and develop recommendations for study findings. The quality appraisal tool was developed by the Effective Public Health Practice Project (EPHPP) as a discrete step within the systematic review process. The EPHPP began in 1998 with funding from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The goal of the project was to address the lack of high quality evidence to support public health interventions known as Ontario Mandatory Health Programs and Service Guidelines (MHPSG, 1997) as outlined in Ontario’s standards for public health. This standardized tool was developed in order to provide high quality systematic reviews to address the public health sector’s need for evidence to support practice. Final results of the “Quality Assessment Tool For Quantitative Studies” lead to an overall methodological rating of strong, moderate or weak in eight sections: 1. selection bias 2. study design 3. confounders 4. blinding 5. data collection methods 6. withdrawals and dropouts 7. intervention integrity 8. analysis A reviewer’s dictionary is provided to assist reviewers and maintain standardized results.

The tool can be applied after a research article has been selected for critical appraisal. After the article has been reviewed, the “Quality Appraisal Tool For Quantitative Studies” is completed according to the instructions provided in the accompanying companion manual.

This tool requires individuals to have specialized expertise in critical appraisal methods. Having knowledge of public health interventions and the ability to synthesize results into applicable practice recommendations is also important.

A standardized approach for the assessment of qualitative research has not been completed. Currently, the EPHPP uses qualitative research in order to explain and interpret the outcomes and interventions under review.

This tool has been evaluated for content and initial construct validity and inter-rater reliability.

Thomas, Ciliska, Dobbins & Miccucci (2004) assessed content validity using an iterative process. An expert group was formed to review the tool for consistency of coding, interpretation, and examination of evidence tables and/or recommendations. The validation process involved assessing the instrument’s content and categories for clarity, completeness and relevance, and an overall comparison with similar types of tools.

Test-retest reliability on this tool was calculated over two occasions using a random selection of primary studies and two reviewers. Results indicated that agreement for the two reviewers was Kappa 0.74 and Kappa 0.61 (Thomas, et al., 2004).

The Effective Public Health Practice Project (EPHPP), McMaster University, Faculty of Health Sciences, School of Nursing. The original authors are B.H. Thomas, Associate Professor, McMaster University School of Nursing, D. Ciliska, Professor, McMaster University School of Nursing, M. Dobbins, Associate Professor, McMaster University School of Nursing, and S. Micucci, Project Coordinator.

This tool was developed by reviewing available instruments in peer review literature and devising a list of components. Content validity was assessed by circulation to six experts in appraising study quality; comments were incorporated. A pilot version tested ten primary studies by four experts in critical appraisal and community health. The tool was further clarified with feedback from the reviewers. A detailed dictionary was developed, as well as guidelines for assessing the overall strength of the articles.

Project Coordinator Effective Public Health Practice Project 1685 Main St. W., Suite 302 Hamilton Ontario L8S 1G5 905-525-9140 www.ephpp.ca

Title of Primary Resource

File Attachment

Web-link

Reference

Effective Public Health Practice Project. (1998). Quality Assessment Tool For Quantitative Studies. Retrieved October 2008 from above web address. Corresponding article of interest: Thomas, B.H., Ciliska, D., Dobbins, M., & Micucci, S. (2004). A process for systematically reviewing the literature: Providing the research evidence for public health nursing interventions. Worldviews on Evidence-Based Nursing, 1(3), 176-184.

Type of Material

Format

Cost to Access

Language

Conditions for Use

Title of Supplementary Resource

File Attachment

Web-link

Reference

Effective Public Health Practice Project. (1998). Dictionary for the Effective Public Health Practice Project Quality Assessment Tool For Quantitative Studies. Retrieved October 2008 from above web address. Corresponding article of interest: Thomas, B.H., Ciliska, D., Dobbins, M., & Micucci, S. (2004). A process for systematically reviewing the literature: Providing the research evidence for public health nursing interventions. Worldviews on Evidence-Based Nursing, 1(3), 176-184.

Type of Material

Format

Cost to Access

Language

Conditions for Use

Title of Supplementary Resource

File Attachment

Web-link

Reference

Thomas, B.H., Ciliska, D., Dobbins, M., & Micucci, S. (2004). A process for systematically reviewing the literature: Providing the research evidence for public health nursing interventions. Worldviews on Evidence-Based Nursing, 1(3), 176-184.

Type of Material

Format

Cost to Access

Language

Conditions for Use