AMSTAR 2: Appraisal tool for systematic reviews of randomized and observational studies (non-randomized studies)

A Summary of

Shea, B., Reeves, B.C., Wells, G., Thuku, M., Hamel, C., Moran, J., et al. (2017). AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008.

Relevance for Public Health

AMSTAR (A Measurement Tool to Assess Systematic Reviews) was the original critical appraisal tool for systematic reviews of randomized studies. AMSTAR 2 is an update that accounts for both randomized studies and observational (non-randomized) studies. It is practical for use for health professionals and public health practitioners involved with program or policy decision making. The goal for AMSTAR 2 is to increase its applicability to include critical appraisal of a broad range of study designs on healthcare interventions used in systematic reviews.

Description

AMSTAR (A Measurement Tool to Assess Systematic Reviews), originally created in 2007, dealt with critical appraisal of systematic reviews. Since then there has been a shift in focus toward evidence-informed decision making using a broader scope of evidence, including observational studies or non-randomized studies. AMSTAR 2, an update to the original AMSTAR tool, was created to address the contemporary need for clinical and policy decision making that accounts for real-world observational evidence.

Accessing the Method/Tool


Language(s)

Not specified

Format(s)

Not specified

Cost

Not Specified

Implementing the Method/Tool


Time for Participation/Completion

10 to 30 minutes

Additional Resources and/or Skills Needed for Implementation

None

Steps for Using Method/Tool

AMSTAR 2 consists of 16 items in total, 10 of which are from the original AMSTAR. Each item allows for the following response options: Yes, Partial Yes, or No. AMSTAR 2 is not intended to be scored.

The following 16 items are covered in AMSTAR 2:

  1. Did the research questions and inclusion criteria for the review include the components of PICO?
  2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?
  3. Did the review authors explain their selection of the study designs for inclusion in the review?
  4. Did the review authors use a comprehensive literature search strategy?
  5. Did the review authors perform study selection in duplicate?
  6. Did the review authors perform data extraction in duplicate?
  7. Did the review authors provide a list of excluded studies and justify the exclusions?
  8. Did the review authors describe the included studies in adequate detail?
  9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?
  10. Did the review authors report on the sources of funding for the studies included in the review?
  11. If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results?
  12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?
  13. Did the review authors account for RoB in primary studies when interpreting/discussing the results of the review?
  14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?
  15. If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?
  16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?

Who is involved

AMSTAR 2 is practical for use by individuals involved in appraising research evidence from a clinical, public health or policy standpoint. Managers and decision makers should also be familiar with the components of the appraisal tool when they are using systematic reviews to inform the basis of evidence-informed decision making.

Conditions for Use

Not specified

Evaluation and Measurement Characteristics


Evaluation

Has been evaluated.

The tool has been evaluated.

Validity

Validity properties meet accepted standards.

The validity of the tool has been evaluated.

Reliability

Reliability properties meet accepted standards.

The reliability of the tool has been evaluated using three pairs of raters across three sets of reviews. Kappa scores were in the acceptable range.

Methodological Rating

Not applicable

Method/Tool Development


Developer(s)

Beverly Shea
Barnaby Reeves
George Wells
Micere Thuku
Candyce Hamel
Julian Moran
David Moher
Peter Tugwell
Vivian Welch
Elizabeth Kristjansson
David Henry

Method of Development

The original AMSTAR tool was developed in 2007. Since then, a need to update the tool was identified based on published critiques, user feedback received at workshops, and developments in the science of systematic reviews. The original AMSTAR tool was created based on a scoping review of rating instruments available at the time; the use of analysis validation techniques; and pilot testing, including reliability and user testing, external validation and content expert review.

AMSTAR 2 was developed based on information from an expert working group, review of current appraisal tools, review of extensive user feedback of the original tool, and a brainstorming session of ideas to create a list of items to consider. The draft instrument underwent pilot testing at workshops, feedback and further modifications, inter-rater reliability testing, and usability testing.

Release Date

2017

Contact Person/Source

Beverly Shea
School of Epidemiology and Public Health, Faculty of Medicine
University of Ottawa
Ottawa, ON
Email: bevshea@uottawa.ca

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Resources

Title of Supplementary Resource AMSTAR 2
File Attachment None
Web-link
Reference

None.

Type of Material Online tool
Format On-line Access
Cost to Access None.
Language English
Conditions for Use Not specified
Title of Primary Resource AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both
File Attachment None
Web-link
Reference

Shea, B., Reeves, B.C., Wells, G., Thuku, M., Hamel, C., Moran, J., et al. (2017). AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008.

Type of Material Journal article
Format Periodical
Cost to Access None.
Language English
Conditions for Use

© BMJ

Title of Supplementary Resource Online appendix 1: AMSTAR 2 guidance document
File Attachment None
Web-link
Reference

None

Type of Material Online document
Format On-line Access
Cost to Access None.
Language English
Conditions for Use Not specified
Title of Supplementary Resource Preferred reporting items for systematic reviews and meta-analysis: The PRISMA statement
File Attachment None
Web-link
Reference

Moher, D., Alessandro, L., Tetzlaff, J., Altman, D.G., and the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analysis: The PRISMA statement. Annals of Internal Medicine, 151(4), 264-269.

Type of Material Journal article
Format Periodical
Cost to Access None.
Language English
Conditions for Use

© 2009, The American College of Physicians

Title of Supplementary Resource Meta-analysis of observational studies in epidemiology: A proposal for reporting
File Attachment None
Web-link
Reference

Stroup, D.F., Berlin, J.A., Morton, S.C., Olkin, I., Williamson, D.G., Rennie, D., et al. (200). Meta-analysis of observational studies in epidemiology: A proposal for reporting. JAMA 283(15), 2008-2012.

Type of Material Journal article
Format Periodical
Cost to Access None.
Language English
Conditions for Use

© 2000 American Medical Association

Title of Supplementary Resource Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
File Attachment None
Web-link
Reference

Shea, B.J., Grimshaw, J.M., Wells, G.A., Boers, M., Andersson, N., Hamel, C., & Bouter, L.M. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7(10). doi: 10.1186/1471-2288-7-10

Type of Material Journal article
Format Periodical
Cost to Access None.
Language English
Conditions for Use

Copyright © 2007 Shea et al; licensee BioMed Central Ltd.

Have you used this resource? Share your story!