user story

Critically appraising practice guidelines: The AGREE II instrument

A summary of Brouwers, M., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., Graham, I.D., Grimshaw, J., Hanna, S., Littlejohns, P., Makarski, J., Zitzelsberger, L. for the AGREE Next Steps Consortium. (2010). AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Canadian Medical Association Journal, 182(18), e839-e842. doi:10.1503/cmaj.090449

How to cite this NCCMT summary:

National Collaborating Centre for Methods and Tools (2011). Critically appraising practice guidelines: The AGREE II instrument. Hamilton, ON: McMaster University. (Updated 01 November, 2013). Retrieved from http://www.nccmt.ca/registry/view/eng/100.html.

Keywords: Appraise, Policy development

Tool Knowledge Doing

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Relevance for Public Health
This tool could be used by individual practitioners to critically appraise public health guidelines and by decision-makers to inform policy decisions.
Description
The AGREE II Instrument assesses the methodological rigour of how a clinical practice guideline was developed. A clinical practice guideline is a systematically developed statement to help people make decisions in clinical settings. Guidelines have also been developed for health policy formation at the system level. Since the quality of the development of these guidelines can vary considerably, a strategy is needed to choose which guidelines should be implemented.

An international team of guideline developers and researchers, known as the AGREE Collaboration (Appraisal of Guidelines, Research and Evaluation), was established to develop such a tool. The original AGREE Instrument was published in 2003. Refinements to the original tool resulted in a revised version, the AGREE II Instrument.

The AGREE Collaboration defines quality of guidelines as "the confidence that the potential biases of guideline development have been addressed adequately and that the recommendations are both internally and externally valid, and are feasible for practice."

The purpose of the AGREE II Instrument is to provide a framework to:
  • assess the quality of guidelines;
  • provide a methodological strategy for the development of guidelines; and
  • inform what information and how the information ought to be reported in guidelines.
The AGREE II Instrument consists of three major sections:
  • Introduction (overview, key resources and references)
  • User's Manual: Instructions for using the AGREE II
  • AGREE II Instrument
Implementing the Method/Tool
Steps for Using Method/Tool
The AGREE II Instrument is available as a PDF or in electronic form, which can be accessed from http://www.agreetrust.org/resource-centre/agree-ii/. Other resources can be accessed from the website, including:
  • AGREE II Training Tools (two online tutorials)
  • AGREE II Translations (translated versions of the instrument in six languages)
  • AGREE-related publications
  • Practice guidelines
  • The Original AGREE Instrument
The AGREE II Instrument consists of 23 items organized within six domains, followed by two global rating items for an overall assessment. Each domain captures a specific aspect of guideline quality.
  • Domain 1: Scope and Purpose—overall aim of the guideline, target group
  • Domain 2: Stakeholder Involvement—extent to which appropriate stakeholers were involved in developing the guideline and represents the views of its intended users
  • Domain 3: Rigour of Development—process of gathering and summarizing the evidence, methods used to develop recommendations
  • Domain 4: Clarity of Presentation—language, structure, format of guideline
  • Domain 5: Applicability—potential barriers and facilitators to implementation, strategies to improve uptake, resources needed to implement the guideline
  • Domain 6: Editorial Independence—biases due to competing interests
Overall assessment includes rating the overall quality of the guideline and whether the guideline would be recommended for use in practice.

Items are rated on a 7-point scale from 1 (Strongly Disagree) to 7 (Strongly Agree). A score of 1 is given when there is no information on that item or if it is poorly reported. A score of 7 is given if the quality of reporting is excellent and when full criteria have been met (in the User's Manual).

A quality score is calculated for each of the six domains, which are independently scored. Domain scores are calculated by summing up all the scores of items in the domain and by scaling the total as a percentage of the maximum possible score for that specific domain.

Who is involved
The AGREE II Instrument is intended to be used by the following groups:
  • frontline practitioners, to assess a guideline before adopting its recommendations in practice
  • guideline developers, to follow a structured methodology to ensure that their guidelines are methodologically sound
  • policy-makers, to inform decision making regarding policies and which guidelines could be implemented in practice
  • educators, to enhance critical appraisal skills amongst practitioners.
Conditions for Use
Not specified
Evaluation and Measurement Characteristics
Evaluation
Has been evaluatedHas been evaluated. The AGREE Next Steps Consortium conducted two studies to evaluate the usefulness, reliability (Brouwers, et al., 2010 - Part 1) and construct validity (Brouwers et al., 2010 - Part 2) of the AGREE II instrument.
Validity
Validity properties meet accepted standardsValidity properties meet accepted standards. Brouwers et al., 2010 (Part 2) conducted the first systematic  evaluation of the construct validity of the AGREE instrument. This study involved 30 participants: 25 people identified their primary role as a guideline developer or researcher, three described their primary role as a clinician and two individuals identified themselves as policy or decision-makers. 

The purpose of the tool is to distinguish between higher and lower quality guidelines. The study assessed the capacity of the AGREE items to distinguish among guideline content of known varying quality (guideline content was manipulated by the researchers to reflect either high or low quality content).

The researchers considered the construct validity results as promising. For all items assessed (NB: item 16 was not assessed because it was not applicable to the guideline used in the study), mean ratings were in the intended direction (i.e., content manipulated to be high quality tended to be rated higher by participants than content manipulated to be lower quality). Three of the items included in the analysis (items 10, 11 and 12) did not yield statistically significant differences between the high and low quality content. 

The authors noted that future validity studies employing larger and more diverse samples of stakeholders are warranted to increase confidence in the validity of the tool.

Reliability
Reliability properties meet accepted standardsReliability properties meet accepted standards. Brouwers et al., 2010 (Part 1) conducted an exploratory analysis of the internal consistency of the AGREE II instrument. The authors noted that further analysis of the measurement properties of the instrument is required.

Chronbach alpha scores measuring internal consistency of the six domains ranged from 0.64 to 0.89. Only two of the domains achieved an alpha value that met conventionally accepted standards for internal consistency (alpha value above 0.8). It should be noted, however, that the internal consistency of the domains is consistent with ranges reported for the original AGREE instrument (The AGREE Collaboration, 2003).

Inter-rater reliability was adequate. The number of appraisers required to reach a level of inter-rater reliability of 0.7 ranged from two to five across domains.
Methodological Rating
ModerateModerate
Method/Tool Development
Developer(s)
AGREE Next Steps Consortium

Dr. Melissa C. Brouwers
Principal Investigator, AGREE Next Steps Consortium
McMaster University, Hamilton ON, Canada

Consortium Members:
Dr. G.P. Browman, British Columbia Cancer Agency, Vancouver Island, Canada
Dr. J.S. Burgers, Dutch Institute for Healthcare Improvement CBO, The Netherlands
Dr. F. Cluzeau, Chair of AGREE Research Trust; St. George's Hospital Medical School, London UK
Dr. D. Davis, Association of American Medical Colleges, Washington DC, USA
Dr. G. Feder, University of Bristol, UK
Dr. B. Fervers, Cancer et Environement, Centre Leon Berard, France
Dr. I. Graham, Canadian Institutes of Health Research, Ottawa ON, Canada
Dr. J. Grimshaw, Ottawa Health Research Institute, Ottawa ON, Canada
Dr. S.E. Hanna, McMaster University, Hamilton ON, Canada
Ms. M.E. Kho, McMaster University, Hamilton ON, Canada
Dr. P. Littlejohns, National Institute for Health and Clinical Excellence, London UK
Ms. J. Makarski, McMaster University, Hamilton ON, Canada
Dr. L. Zitzelsberger, Canadian Partnership Against Cancer, Ottawa ON, Canada

AGREE Research Trust website: www.agreetrust.org
AGREE II Project Office email: agree@mcmaster.ca
Method of Development
The AGREE (Appraisal of Guidelines for Research and Evaluation) Next Steps Consortium was formed with several members from the original team, the AGREE Collaboration, to refine the original AGREE instrument. The goals of the AGREE Next Steps Consortium were to:
  • improve the measurement properties of the original AGREE instrument, including the tool's reliability and validity;
  • refine the instrument's items to better meet the needs of intended users; and
  • improve supporting documents (the user guide) to facilitate use of the instrument.
The Consortium's efforts resulted in the publication of the AGREE II instrument. The AGREE Research Trust (ART) is an independent body that manages the interests of the AGREE enterprise, and supports a research agenda regarding its development (http://www.agreetrust.org).
Release Date
2003
Contact Person/Source
Dr. Melissa C. Brouwers
Associate Professor, Department of Oncology
Associate Member, Department of Clinical Epidemiology & Biostatistics
Provincial Director, Program in Evidence-based Care, Cancer Care Ontario
Lead, Capacity Enhancement Program, Canadian Partnership Against Cancer

Department of Oncology
McMaster University
Juravinski Hospital Site
G Wing, 2nd Floor, Room 207
711 Concession Street
Hamilton ON Canada L8V 1C3
Tel: 905-527-4322 x42824
email: mbrouwer@mcmaster.ca

Resources

Title of Primary Resource
AGREE II: Advancing guideline development, reporting and evaluation in healthcare.
File Attachment
None
Web-link
Reference
Brouwers, M., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., Graham, I.D., Grimshaw, J., Hanna, S., Littlejohns, P., Makarski, J., Zitzelsberger, L. for the AGREE Next Steps Consortium. (2010). AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Canadian Medical Association Journal, 182(18), e839-e842. doi:10.1503/cmaj.090449
Type of Material
Journal article
Format
Periodical
Cost to Access
Journal article purchase 
Language
English
Conditions for Use
Copyright © 2010 Canadian Medical Association 

Title of Supplementary Resource
Development of the AGREE II, part 2: Assessment of validity of items and tools to support application
File Attachment
None
Web-link
Reference
Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., et al. (2010). Development of the AGREE II, part 2: Assessment of validity of items and tools to support application. Canadian Medical Association Journal, 182(10), e472-e478. doi: 10.1503/cmaj.091716
Type of Material
Journal article
Format
Periodical
Cost to Access
Journal article purchase 
Language
English
Conditions for Use
Copyright © 2010 Canadian Medical Association 

Title of Supplementary Resource
Development of the AGREE II, part 1: Performance, usefulness and areas for improvement
File Attachment
None
Web-link
Reference
Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., et al. (2010). Development of the AGREE II, part 1: Performance, usefulness and areas for improvement. Canadian Medical Association Journal, 182(10), 1045-1052. doi: 10.1503/cmaj.091714
Type of Material
Journal article
Format
Periodical
Cost to Access
Journal article purchase 
Language
English
Conditions for Use
Copyright © 2010 Canadian Medical Association 

Title of Supplementary Resource
Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: The AGREE project.
File Attachment
None
Web-link
Reference
The AGREE Collaboration. (2003). Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: The AGREE project. Quality and Safety in Health Care, 12, 18-23. doi: 10.1136/qhc.12.1.18
Type of Material
Journal article
Format
Periodical
Cost to Access
Journal article purchase 
Language
English
Conditions for Use
Copyright © 2003 by the BMJ Publishing Group Ltd. 


Have you used this resource? Share your story!