building capacity for
evidence-informed public health

user story webinar summary

Critically appraising practice guidelines: The AGREE II instrument

A summary of Brouwers, M., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., Graham, I.D., Grimshaw, J., Hanna, S., Littlejohns, P., Makarski, J., Zitzelsberger, L. for the AGREE Next Steps Consortium. (2010). AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Canadian Medical Association Journal, 182(18), e839-e842. doi:10.1503/cmaj.090449

How to cite this NCCMT summary:

National Collaborating Centre for Methods and Tools (2011). Critically appraising practice guidelines: The AGREE II instrument. Hamilton, ON: McMaster University. (Updated 01 November, 2013). Retrieved from http://www.nccmt.ca/registry/view/eng/100.html.

Categories: Appraise, Policy development

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Relevance for Public Health
This tool could be used by individual practitioners to critically appraise public health guidelines and by decision-makers to inform policy decisions.
Description
The AGREE II Instrument assesses the methodological rigour of how a clinical practice guideline was developed. A clinical practice guideline is a systematically developed statement to help people make decisions in clinical settings. Guidelines have also been developed for health policy formation at the system level. Since the quality of the development of these guidelines can vary considerably, a strategy is needed to choose which guidelines should be implemented. An international team of guideline developers and researchers, known as the AGREE Collaboration (Appraisal of Guidelines, Research and Evaluation), was established to develop such a tool. The original AGREE Instrument was published in 2003. Refinements to the original tool resulted in a revised version, the AGREE II Instrument. The AGREE Collaboration defines quality of guidelines as "the confidence that the potential biases of guideline development have been addressed adequately and that the recommendations are both internally and externally valid, and are feasible for practice."The purpose of the AGREE II Instrument is to provide a framework to: assess the quality of guidelines; provide a methodological strategy for the development of guidelines; and inform what information and how the information ought to be reported in guidelines. The AGREE II Instrument consists of three major sections: Introduction (overview, key resources and references) User's Manual: Instructions for using the AGREE II AGREE II Instrument
Implementing the Method/Tool
Steps for Using Method/Tool
The AGREE II Instrument is available as a PDF or in electronic form, which can be accessed from http://www.agreetrust.org/resource-centre/agree-ii/. Other resources can be accessed from the website, including: AGREE II Training Tools (two online tutorials) AGREE II Translations (translated versions of the instrument in six languages) AGREE-related publications Practice guidelines The Original AGREE Instrument The AGREE II Instrument consists of 23 items organized within six domains, followed by two global rating items for an overall assessment. Each domain captures a specific aspect of guideline quality. Domain 1: Scope and Purpose—overall aim of the guideline, target group Domain 2: Stakeholder Involvement—extent to which appropriate stakeholers were involved in developing the guideline and represents the views of its intended users Domain 3: Rigour of Development—process of gathering and summarizing the evidence, methods used to develop recommendations Domain 4: Clarity of Presentation—language, structure, format of guideline Domain 5: Applicability—potential barriers and facilitators to implementation, strategies to improve uptake, resources needed to implement the guideline Domain 6: Editorial Independence—biases due to competing interests Overall assessment includes rating the overall quality of the guideline and whether the guideline would be recommended for use in practice.Items are rated on a 7-point scale from 1 (Strongly Disagree) to 7 (Strongly Agree). A score of 1 is given when there is no information on that item or if it is poorly reported. A score of 7 is given if the quality of reporting is excellent and when full criteria have been met (in the User's Manual).A quality score is calculated for each of the six domains, which are independently scored. Domain scores are calculated by summing up all the scores of items in the domain and by scaling the total as a percentage of the maximum possible score for that specific domain.
Who is involved
The AGREE II Instrument is intended to be used by the following groups: frontline practitioners, to assess a guideline before adopting its recommendations in practice guideline developers, to follow a structured methodology to ensure that their guidelines are methodologically sound policy-makers, to inform decision making regarding policies and which guidelines could be implemented in practice educators, to enhance critical appraisal skills amongst practitioners.
Conditions for Use
Not specified
Evaluation and Measurement Characteristics
Evaluation
Has been evaluated. The AGREE Next Steps Consortium conducted two studies to evaluate the usefulness, reliability (Brouwers, et al., 2010 - Part 1) and construct validity (Brouwers et al., 2010 - Part 2) of the AGREE II instrument.
Validity
Validity properties meet accepted standards. Brouwers et al., 2010 (Part 2) conducted the first systematic  evaluation of the construct validity of the AGREE instrument. This study involved 30 participants: 25 people identified their primary role as a guideline developer or researcher, three described their primary role as a clinician and two individuals identified themselves as policy or decision-makers. The purpose of the tool is to distinguish between higher and lower quality guidelines. The study assessed the capacity of the AGREE items to distinguish among guideline content of known varying quality (guideline content was manipulated by the researchers to reflect either high or low quality content).The researchers considered the construct validity results as promising. For all items assessed (NB: item 16 was not assessed because it was not applicable to the guideline used in the study), mean ratings were in the intended direction (i.e., content manipulated to be high quality tended to be rated higher by participants than content manipulated to be lower quality). Three of the items included in the analysis (items 10, 11 and 12) did not yield statistically significant differences between the high and low quality content. The authors noted that future validity studies employing larger and more diverse samples of stakeholders are warranted to increase confidence in the validity of the tool.
Reliability
Reliability properties meet accepted standards. Brouwers et al., 2010 (Part 1) conducted an exploratory analysis of the internal consistency of the AGREE II instrument. The authors noted that further analysis of the measurement properties of the instrument is required. Chronbach alpha scores measuring internal consistency of the six domains ranged from 0.64 to 0.89. Only two of the domains achieved an alpha value that met conventionally accepted standards for internal consistency (alpha value above 0.8). It should be noted, however, that the internal consistency of the domains is consistent with ranges reported for the original AGREE instrument (The AGREE Collaboration, 2003).Inter-rater reliability was adequate. The number of appraisers required to reach a level of inter-rater reliability of 0.7 ranged from two to five across domains.
Methodological Rating
Moderate
Method/Tool Development
Developer(s)
AGREE Next Steps ConsortiumDr. Melissa C. BrouwersPrincipal Investigator, AGREE Next Steps ConsortiumMcMaster University, Hamilton ON, CanadaConsortium Members:Dr. G.P. Browman, British Columbia Cancer Agency, Vancouver Island, CanadaDr. J.S. Burgers, Dutch Institute for Healthcare Improvement CBO, The NetherlandsDr. F. Cluzeau, Chair of AGREE Research Trust; St. George's Hospital Medical School, London UKDr. D. Davis, Association of American Medical Colleges, Washington DC, USADr. G. Feder, University of Bristol, UKDr. B. Fervers, Cancer et Environement, Centre Leon Berard, FranceDr. I. Graham, Canadian Institutes of Health Research, Ottawa ON, CanadaDr. J. Grimshaw, Ottawa Health Research Institute, Ottawa ON, CanadaDr. S.E. Hanna, McMaster University, Hamilton ON, CanadaMs. M.E. Kho, McMaster University, Hamilton ON, CanadaDr. P. Littlejohns, National Institute for Health and Clinical Excellence, London UKMs. J. Makarski, McMaster University, Hamilton ON, CanadaDr. L. Zitzelsberger, Canadian Partnership Against Cancer, Ottawa ON, CanadaAGREE Research Trust website: www.agreetrust.orgAGREE II Project Office email: agree@mcmaster.ca
Method of Development
The AGREE (Appraisal of Guidelines for Research and Evaluation) Next Steps Consortium was formed with several members from the original team, the AGREE Collaboration, to refine the original AGREE instrument. The goals of the AGREE Next Steps Consortium were to: improve the measurement properties of the original AGREE instrument, including the tool's reliability and validity; refine the instrument's items to better meet the needs of intended users; and improve supporting documents (the user guide) to facilitate use of the instrument. The Consortium's efforts resulted in the publication of the AGREE II instrument. The AGREE Research Trust (ART) is an independent body that manages the interests of the AGREE enterprise, and supports a research agenda regarding its development (http://www.agreetrust.org).
Release Date
2003
Contact Person/Source
Dr. Melissa C. BrouwersAssociate Professor, Department of OncologyAssociate Member, Department of Clinical Epidemiology & BiostatisticsProvincial Director, Program in Evidence-based Care, Cancer Care OntarioLead, Capacity Enhancement Program, Canadian Partnership Against CancerDepartment of OncologyMcMaster UniversityJuravinski Hospital SiteG Wing, 2nd Floor, Room 207711 Concession StreetHamilton ON Canada L8V 1C3Tel: 905-527-4322 x42824email: mbrouwer@mcmaster.ca

Resources

Title of Primary Resource
AGREE II: Advancing guideline development, reporting and evaluation in healthcare.
File Attachment
None
Web-link
Reference
Brouwers, M., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., Graham, I.D., Grimshaw, J., Hanna, S., Littlejohns, P., Makarski, J., Zitzelsberger, L. for the AGREE Next Steps Consortium. (2010). AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Canadian Medical Association Journal, 182(18), e839-e842. doi:10.1503/cmaj.090449
Type of Material
Journal article
Format
Periodical
Cost to Access
Journal article purchase 
Language
English
Conditions for Use
Copyright © 2010 Canadian Medical Association 

Title of Supplementary Resource
File Attachment
None
Web-link
Reference
Type of Material
Format
Not specified
Cost to Access
 
Language
Not specified
Conditions for Use
 

Title of Supplementary Resource
Development of the AGREE II, part 2: Assessment of validity of items and tools to support application
File Attachment
None
Web-link
Reference
Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., et al. (2010). Development of the AGREE II, part 2: Assessment of validity of items and tools to support application. Canadian Medical Association Journal, 182(10), e472-e478. doi: 10.1503/cmaj.091716
Type of Material
Journal article
Format
Periodical
Cost to Access
Journal article purchase 
Language
English
Conditions for Use
Copyright © 2010 Canadian Medical Association 

Title of Supplementary Resource
Development of the AGREE II, part 1: Performance, usefulness and areas for improvement
File Attachment
None
Web-link
Reference
Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., et al. (2010). Development of the AGREE II, part 1: Performance, usefulness and areas for improvement. Canadian Medical Association Journal, 182(10), 1045-1052. doi: 10.1503/cmaj.091714
Type of Material
Journal article
Format
Periodical
Cost to Access
Journal article purchase 
Language
English
Conditions for Use
Copyright © 2010 Canadian Medical Association 

Title of Supplementary Resource
Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: The AGREE project.
File Attachment
None
Web-link
Reference
The AGREE Collaboration. (2003). Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: The AGREE project. Quality and Safety in Health Care, 12, 18-23. doi: 10.1136/qhc.12.1.18
Type of Material
Journal article
Format
Periodical
Cost to Access
Journal article purchase 
Language
English
Conditions for Use
Copyright © 2003 by the BMJ Publishing Group Ltd. 


Have you used this resource? Share your story!