AMSTAR: assessing methodological quality of systematic reviews

A summary of

Shea, B.J., Grimshaw, J.M., Wells, G.A., Boers, M., Andersson, N., Hamel, C., & Bouter, L.M. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7(10). doi: 10.1186/1471-2288-7-10

How to cite this NCCMT summary:

National Collaborating Centre for Methods and Tools (2011). AMSTAR: assessing methodological quality of systematic reviews. Hamilton, ON: McMaster University. (Updated 21 July, 2011) Retrieved from http://www.nccmt.ca/resources/search/97.

Categories: Tool, Appraise

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Relevance for Public Health
This tool was developed for general use and may be applied to assess the methodological quality of systematic reviews of public health interventions. It should be noted, however, that its use has only been tested on systematic reviews of randomized controlled trial designs. AMSTAR is being used by a number of groups, including the Canadian Agency for Drugs and Technologies in Health and The Cochrane Effective Practice and Organization of Care Group (EPOC).
Description

The tool is an 11-item questionnaire that can be used to assess the methodological quality of systematic reviews by assessing the presence of:

  • an a priori design;
  • duplicate study selection and data extraction;
  • a comprehensive literature search;
  • the use of status of publication as an inclusion criteria;
  • a list of included/excluded studies;
  • characteristics of included studies;
  • documented assessment of the scientific quality of included studies;
  • appropriate use of the scientific quality in forming conclusions;
  • the appropriate use of methods to combine findings of studies;
  • assessment of the likelihood of publication bias; and
  • documentation of conflict of interest.

The AMSTAR tool was created by building on previous tools, empirical evidence and expert consensus. Over a decade has passed since the initial development of these types of tools, and more research has been conducted about potential sources of bias in systematic reviews. AMSTAR has incorporated these other sources of bias and will remain a "living document" subject to improvement as further advances in methodological research occur.

The instrument was developed using the following:

  1. the enhanced Overview Quality Assessment Questionnaire (OQAQ) by Oxman and Guyatt (1991)
  2. a checklist created by Sacks et. al (1987)
  3. three additional items recently judged to be of methodological importance:
    • language restriction
    • publication bias
    • publication status (inclusion of grey literature)

The tool was applied to 99 paper-based and 52 electronic systematic reviews. Exploratory factor analysis was used to identify underlying components. Methodological experts considered the results using a nominal group technique aimed at item reduction and design of an assessment tool with face and content validity.

Implementing the Method/Tool
Steps for Using Method/Tool
Users may print off a copy of the 11-item tool and use it to critically appraise a systematic review. This tool enables the user to qualitatively assess the quality of a systematic review.

AMSTAR determines the methodological quality of systematic reviews by assessing the presence of:

  • an a priori design;
  • duplicate study selection and data extraction;
  • a comprehensive literature search;
  • the use of status of publication as an inclusion criteria;
  • a list of included/excluded studies;
  • characteristics of included studies;
  • documented assessment of the scientific quality of included studies;
  • appropriate use of the scientific quality in forming conclusions;
  • the appropriate use of methods to combine findings of studies;
  • assessment of the likelihood of publication bias; and
  • documentation of conflict of interest.
Note: One study by another group of authors (Kung et al., 2010) has developed a revised tool called "R-AMSTAR" that uses a quantitative scoring method to assess the quality of systematic reviews.
Who is involved
Anyone who is critically appraising a systematic review could use this tool.
Conditions for Use
Not specified
Evaluation and Measurement Characteristics
Evaluation
Has been evaluated. Two reliability and validity evaluations of the AMSTAR tool have been conducted. Both evaluation studies include authors from the AMSTAR tool development study (Shea, Grimshaw, et al., 2007):

1) Shea, B.J., Bouter, L.M., Peterson, J., Boers, M., Andersson, N., et al. (2007). External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR). PLoS ONE, 2(12): e1350. doi:10.1371/journal.pone.0001350
  • AMSTAR was used to appraise 42 reviews focused on therapies for gastro-esophageal reflux, peptic ulcer disease and other acid-related diseases.
  • Two assessors applied the AMSTAR to each review.
  • Two other assessors, plus a clinician and/or a methodologist, independently applied a global assessment to each review.
  • Reported outcomes included reliability (inter-observer kappas) and construct validity.

2)
Shea, G.J., Hamel, C., Wells, G.A., Bouter, L.M., Kristjansson, E., Grimshaw, J,. Henry, D.A., Boers, M. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology. Oct; 62(10):1013-20. Epub 2009 Feb 20.

  • Thirty systematic reviews were randomly selected from a database of 151 reviews that were used in the development of AMSTAR.
  • Each review was assessed by two reviewers using the following:
1. the Overview of Quality Assessment Questionnaire (OQAQ)
2. Sack's instrument
3. AMSTAR
  • Reported outcomes included reliability (inter-observer kappas), intra-class correlation coefficients of the sum scores, construct validity and completion times.
Validity
Validity properties meet accepted standards. The following validity properties have been assessed:

1) Face validity—expert review for appropriateness; see method/tool development below

2) Content validity—extent to which a measure represents all facets of a given social concept; see method/tool development below

3) Construct (convergent) validity—ability of an instrument to measure an abstract concept or construct; assesses the overlap between two or more tests that presumably measure the same construct

In the 2009 Shea et al. study, construct validity was assessed by converting the mean total score of each of the 30 reviews to a percentage of the maximum score for each of the three instruments (AMSTAR, OQAQ and Sacks et al.). Intra-class correlation assessed convergence of the total scores between each pair of instruments (AMSTAR-OQAQ, AMSTAR-Sacks and OQAQ-Sacks). Similarly, in the 2007 Shea et al. study, construct validity was assessed by comparing AMSTAR with a global assessment tool.


Reliability
Reliability properties meet accepted standards. The AMSTAR tool was found to have high inter-rater reliability by measuring the kappa statistic in both evaluation studies (Shea, Bouter, et al., 2007; Shea, Hamel, et al., 2009). The kappa statistic measures the level of agreement between two observers that could be expected by chance. Kappa scores > 0.8 are considered to be almost perfect agreement.

In both evaluation studies, kappa scores ranged from moderate to almost perfect agreement for AMSTAR. In the 2007 Shea et al. study, nine out of 11 items scored a kappa > 0.75 and the overall scores had a kappa of 0.84. The 2009 Shea et al. study had an average kappa score of 0.70 for inter-rater agreement for individual items.

Methodological Rating
Strong
Method/Tool Development
Developer(s)
Beverley J Shea
EMGO Institute, VU University Medical Center, Amsterdam, the Netherlands
Community Information and Epidemiological technologies (CIET) Institute of Population Health, Ottawa, ON

Jeremy M Grimshaw
Institute of Population Health, Ottawa, ON

George A. Wells
Department of Epidemiology and Community Medicine, University of Ottawa,
Ottawa, ON

Maarten Boers
Department of Clinical Epidemiology and Biostatistics, VU University Medical Center,
Amsterdam, the Netherlands

Neil Andersson
Community Information and Epidemiological Technologies (CIET),
Ottawa, ON

Candyce Hamel
Community Information and Epidemiological Technologies (CIETcanada)
Ottawa, ON

Ashley C. Porter
Community Information and Epidemiological Technologies (CIETcanada)
Ottawa, ON

Peter Tugwell
Institute of Population Health, Ottawa, ON

David Moher
University of Ottawa, Ottawa, ON

Lex M. Bouter
EMGO Institute, VU University Medical Center, Amsterdam, the Netherlands
Method of Development

The AMSTAR tool was created by building on previous tools, empirical evidence and expert consensus. Initially, the tool created using the following:

  1. the enhanced Overview Quality Assessment Questionnaire (OQAQ) by Oxman and Guyatt (1991)
  2. a checklist created by Sacks et al. (1987)
  3. three additional items recently judged to be of methodological importance:
    • language restriction
    • publication bias
    • publication status (inclusion of grey literature)
    Exploratory factor analysis
  4. The tool was applied to assess the quality of 99 paper-based and 52 electronic systematic reviews. Exploratory factor analysis was conducted to discover the main dimensions of the tool by conducting a preliminary investigation of the correlations between all the identified variables. Items with low factor loadings tended to be weakly correlated with other items and were removed. The factor analysis made it possible to reduce the 37-item instrument to 29 items that measured 11 components.

    Nominal Group Technique

    Building on the results of the explanatory factor analysis, 11 international experts in the fields of methodological quality assessment and systematic reviews were convened to reduce the item pool and assess face and content validity using nominal group technique.

    Nominal group technique is generally used to aid decision making. The nominal technique involves experts, discussion and a consensus that is qualitative in nature. In this study, the expert panel reviewed the results of the factor analysis. Each participant recorded his or her ideas independently. Then, ideas were listed in a round-robin format with a facilitator until all ideas had been listed, and then a group discussion took place. Individuals again independently recorded their private judgments. These were aggregated to produce group judgments. The formulated tool was electronically circulated to the group for a final round of fine tuning after the meeting.

    The nominal group identified the items that most appropriately captured the 11 components identified through factor analysis and confirmed face and content validity of the tool. The resulting instrument is an 11-item questionnaire.

Release Date
2007
Contact Person/Source
Beverley J Shea
Community Information and Epidemiological technologies (CIET) Institute of Population Health
1 Stewart Street, Rm 319
Ottawa, Ontario K1N 6N5
Canada

Tel: +613-562-5800 ext. 8571
Fax: +613-562-5392
Email: bshea@ciet.org or bevshea@uottawa.ca

Resources

Title of Primary Resource
Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews
File Attachment
None
Web-link
Reference
Shea, B.J., Grimshaw, J.M., Wells, G.A., Boers, M., Andersson, N., Hamel, C., & Bouter, L.M. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7(10). doi: 10.1186/1471-2288-7-10
Type of Material
Journal article
Format
Periodical
Cost to Access
Language
English
Conditions for Use
Copyright © 2007 Shea et al; licensee BioMed Central Ltd.

Title of Supplementary Resource
External validation of a measurement tool to assess systematic reviews (AMSTAR)
File Attachment
None
Web-link
Reference

Shea, B.J., Bouter, L.M., Peterson J., Boers, M., Andersson, N., Ortiz, Z., & Grimshaw, J.M. (2007). External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE 2(12), e1350. doi:10.1371/journal.pone.0001350

Type of Material
Journal article
Format
Periodical
Cost to Access
Language
English
Conditions for Use
Copyright © 2007 Shea et al.

Title of Supplementary Resource
AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.
File Attachment
None
Web-link
Reference
Shea, B.J., Hamel, C., Wells, G.A., Bouter, L.M., Kristjansson, E., Grimshaw, J., & Boers, M. (2009). AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology, 62(10), 1013-20.
Type of Material
Journal article
Format
Periodical
Cost to Access
Journal article purchase
Language
English
Conditions for Use
Copyright © 2009 Elsevier Inc.

Title of Supplementary Resource
From systematic reviews to clinical recommendations for evidence-based health care: Validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance
File Attachment
None
Web-link
Reference
Kung, J., Chiappelli, F., Cajulis, O.O., Avezova, R., Kossan, G., Chew, L., Maida, C.A. (2010). From systematic reviews to clinical recommendations for evidence-based health care: Validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. The Open Dentistry Journal, 4, 84-91.
Type of Material
Journal article
Format
Periodical
Cost to Access
Language
English
Conditions for Use
Copyright © 2010 Kung et al; Licensee

Have you used this resource? Share your story!