AMSTAR2: Appraisal tool for systematic reviews of randomized and observational studies (non-randomized studies)

A summary of

Shea, B., Reeves, B.C., Wells, G., Thuku, M., Hamel, C., Moran, J., et al. (2017). AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008.

How to cite this NCCMT summary:

National Collaborating Centre for Methods and Tools (2017). AMSTAR2: Appraisal tool for systematic reviews of randomized and observational studies (non-randomized studies). Hamilton, ON: McMaster University. (Updated 21 January, 2018) Retrieved from http://www.nccmt.ca/knowledge-repositories/search/307.

Categories: Tool, Appraise

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Relevance for Public Health

AMSTAR (A Measurement Tool to Assess Systematic Reviews) was the original critical appraisal tool for systematic reviews of randomized studies. AMSTAR2 is an update that accounts for both randomized studies and observational (non-randomized) studies. It is practical for use for health professionals and public health practitioners involved with program or policy decision making. The goal for AMSTAR2 is to increase its applicability to include critical appraisal of a broad range of study designs on healthcare interventions used in systematic reviews.

Description

AMSTAR (A Measurement Tool to Assess Systematic Reviews), originally created in 2007, dealt with critical appraisal of systematic reviews. Since then there has been a shift in focus toward evidence-informed decision making using a broader scope of evidence, including observational studies or non-randomized studies. AMSTAR2, an update to the original AMSTAR tool, was created to address the contemporary need for clinical and policy decision making that accounts for real-world observational evidence.

Implementing the Method/Tool
Steps for Using Method/Tool

AMSTAR2 consists of 16 items in total, 10 of which are from the original AMSTAR. Each item allows for the following response options: Yes, Partial Yes, or No. AMSTAR2 is not intended to be scored.

The following 16 items are covered in AMSTAR2:

  1. Did the research questions and inclusion criteria for the review include the components of PICO?
  2. Did the report of the review contain an explicit statement that the review methods were established prior to the conduct of the review and did the report justify any significant deviations from the protocol?
  3. Did the review authors explain their selection of the study designs for inclusion in the review?
  4. Did the review authors use a comprehensive literature search strategy?
  5. Did the review authors perform study selection in duplicate?
  6. Did the review authors perform data extraction in duplicate?
  7. Did the review authors provide a list of excluded studies and justify the exclusions?
  8. Did the review authors describe the included studies in adequate detail?
  9. Did the review authors use a satisfactory technique for assessing the risk of bias (RoB) in individual studies that were included in the review?
  10. Did the review authors report on the sources of funding for the studies included in the review?
  11. If meta-analysis was performed, did the review authors use appropriate methods for statistical combination of results?
  12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?
  13. Did the review authors account for RoB in primary studies when interpreting/discussing the results of the review?
  14. Did the review authors provide a satisfactory explanation for, and discussion of, any heterogeneity observed in the results of the review?
  15. If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?
  16. Did the review authors report any potential sources of conflict of interest, including any funding they received for conducting the review?
Who is involved

AMSTAR2 is practical for use by individuals involved in appraising research evidence from a clinical, public health or policy standpoint. Managers and decision makers should also be familiar with the components of the appraisal tool when they are using systematic reviews to inform the basis of evidence-informed decision making.

Conditions for Use
Not specified
Evaluation and Measurement Characteristics
Evaluation
Has been evaluated.

The tool has been evaluated.

Validity
Validity properties meet accepted standards.

The validity of the tool has been evaluated.

Reliability
Reliability properties meet accepted standards.

The reliability of the tool has been evaluated using three pairs of raters across three sets of reviews. Kappa scores were in the acceptable range.

Methodological Rating
Not applicable
Method/Tool Development
Developer(s)

Beverly Shea
Barnaby Reeves
George Wells
Micere Thuku
Candyce Hamel
Julian Moran
David Moher
Peter Tugwell
Vivian Welch
Elizabeth Kristjansson
David Henry

Method of Development

The original AMSTAR tool was developed in 2007. Since then, a need to update the tool was identified based on published critiques, user feedback received at workshops, and developments in the science of systematic reviews. The original AMSTAR tool was created based on a scoping review of rating instruments available at the time; the use of analysis validation techniques; and pilot testing, including reliability and user testing, external validation and content expert review.

AMSTAR2 was developed based on information from an expert working group, review of current appraisal tools, review of extensive user feedback of the original tool, and a brainstorming session of ideas to create a list of items to consider. The draft instrument underwent pilot testing at workshops, feedback and further modifications, inter-rater reliability testing, and usability testing.

Release Date
2017
Contact Person/Source

Beverly Shea
School of Epidemiology and Public Health, Faculty of Medicine
University of Ottawa
Ottawa, ON
Email: bevshea@uottawa.ca

Resources

Title of Supplementary Resource
AMSTAR2
File Attachment
None
Web-link
Reference

None.

Type of Material
Online tool
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use
Not specified

Title of Primary Resource
AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both
File Attachment
None
Web-link
Reference

Shea, B., Reeves, B.C., Wells, G., Thuku, M., Hamel, C., Moran, J., et al. (2017). AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ, 358, j4008.

Type of Material
Journal article
Format
Periodical
Cost to Access
None.
Language
English
Conditions for Use

© BMJ


Title of Supplementary Resource
Online appendix 1: AMSTAR2 guidance document
File Attachment
None
Web-link
Reference

None

Type of Material
Online document
Format
On-line Access
Cost to Access
None.
Language
English
Conditions for Use
Not specified

Title of Supplementary Resource
Preferred reporting items for systematic reviews and meta-analysis: The PRISMA statement
File Attachment
None
Web-link
Reference

Moher, D., Alessandro, L., Tetzlaff, J., Altman, D.G., and the PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analysis: The PRISMA statement. Annals of Internal Medicine, 151(4), 264-269.

Type of Material
Journal article
Format
Periodical
Cost to Access
None.
Language
English
Conditions for Use

© 2009, The American College of Physicians


Title of Supplementary Resource
Meta-analysis of observational studies in epidemiology: A proposal for reporting
File Attachment
None
Web-link
Reference

Stroup, D.F., Berlin, J.A., Morton, S.C., Olkin, I., Williamson, D.G., Rennie, D., et al. (200). Meta-analysis of observational studies in epidemiology: A proposal for reporting. JAMA 283(15), 2008-2012.

Type of Material
Journal article
Format
Periodical
Cost to Access
None.
Language
English
Conditions for Use

© 2000 American Medical Association


Title of Supplementary Resource
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
File Attachment
None
Web-link
Reference

Shea, B.J., Grimshaw, J.M., Wells, G.A., Boers, M., Andersson, N., Hamel, C., & Bouter, L.M. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7(10). doi: 10.1186/1471-2288-7-10

Type of Material
Journal article
Format
Periodical
Cost to Access
None.
Language
English
Conditions for Use

Copyright © 2007 Shea et al; licensee BioMed Central Ltd.


Have you used this resource? Share your story!