Relevance for Public Health
This tool could be used by individual practitioners to critically appraise public health guidelines and by decision-makers to inform policy decisions.
The AGREE II Instrument assesses the methodological rigour of how a clinical practice guideline was developed. A clinical practice guideline is a systematically developed statement to help people make decisions in clinical settings. Guidelines have also been developed for health policy formation at the system level. Since the quality of the development of these guidelines can vary considerably, a strategy is needed to choose which guidelines should be implemented. An international team of guideline developers and researchers, known as the AGREE Collaboration (Appraisal of Guidelines, Research and Evaluation), was established to develop such a tool. The original AGREE Instrument was published in 2003. Refinements to the original tool resulted in a revised version, the AGREE II Instrument. The AGREE Collaboration defines quality of guidelines as "the confidence that the potential biases of guideline development have been addressed adequately and that the recommendations are both internally and externally valid, and are feasible for practice."The purpose of the AGREE II Instrument is to provide a framework to: assess the quality of guidelines; provide a methodological strategy for the development of guidelines; and inform what information and how the information ought to be reported in guidelines. The AGREE II Instrument consists of three major sections: Introduction (overview, key resources and references) User's Manual: Instructions for using the AGREE II AGREE II Instrument
Accessing the Method/Tool
Implementing the Method/Tool
Time for Participation/Completion
1 to 2 hours
Depending on the length of the guideline document, quality assessment of a guideline using AGREE II will take approximately 1.5 hours.
Additional Resources and/or Skills Needed for Implementation
Basic knowledge of the principles of evidence-informed decision making and health care methodology can assist users of the AGREE II intstrument. A user's manual is provided to help beginners with critically appraising guidelines.
Steps for Using Method/Tool
The AGREE II Instrument is available as a PDF or in electronic form, which can be accessed from http://www.agreetrust.org/resource-centre/agree-ii/. Other resources can be accessed from the website, including: AGREE II Training Tools (two online tutorials) AGREE II Translations (translated versions of the instrument in six languages) AGREE-related publications Practice guidelines The Original AGREE Instrument The AGREE II Instrument consists of 23 items organized within six domains, followed by two global rating items for an overall assessment. Each domain captures a specific aspect of guideline quality.
Domain 1: Scope and Purpose—overall aim of the guideline, target group
Domain 2: Stakeholder Involvement—extent to which appropriate stakeholers were involved in developing the guideline and represents the views of its intended users
Domain 3: Rigour of Development—process of gathering and summarizing the evidence, methods used to develop recommendations
Domain 4: Clarity of Presentation—language, structure, format of guideline
Domain 5: Applicability—potential barriers and facilitators to implementation, strategies to improve uptake, resources needed to implement the guideline
Domain 6: Editorial Independence—biases due to competing interests Overall assessment includes rating the overall quality of the guideline and whether the guideline would be recommended for use in practice.Items are rated on a 7-point scale from 1 (Strongly Disagree) to 7 (Strongly Agree). A score of 1 is given when there is no information on that item or if it is poorly reported. A score of 7 is given if the quality of reporting is excellent and when full criteria have been met (in the User's Manual).A quality score is calculated for each of the six domains, which are independently scored. Domain scores are calculated by summing up all the scores of items in the domain and by scaling the total as a percentage of the maximum possible score for that specific domain.
Who is involved
The AGREE II Instrument is intended to be used by the following groups: frontline practitioners, to assess a guideline before adopting its recommendations in practice guideline developers, to follow a structured methodology to ensure that their guidelines are methodologically sound policy-makers, to inform decision making regarding policies and which guidelines could be implemented in practice educators, to enhance critical appraisal skills amongst practitioners.
Conditions for Use
Evaluation and Measurement Characteristics
Has been evaluated.
The AGREE Next Steps Consortium conducted two studies to evaluate the usefulness, reliability (Brouwers, et al., 2010 - Part 1) and construct validity (Brouwers et al., 2010 - Part 2) of the AGREE II instrument.
Validity properties meet accepted standards.
Brouwers et al., 2010 (Part 2) conducted the first systematic evaluation of the construct validity of the AGREE instrument. This study involved 30 participants: 25 people identified their primary role as a guideline developer or researcher, three described their primary role as a clinician and two individuals identified themselves as policy or decision-makers. The purpose of the tool is to distinguish between higher and lower quality guidelines. The study assessed the capacity of the AGREE items to distinguish among guideline content of known varying quality (guideline content was manipulated by the researchers to reflect either high or low quality content). The researchers considered the construct validity results as promising. For all items assessed (NB: item 16 was not assessed because it was not applicable to the guideline used in the study), mean ratings were in the intended direction (i.e., content manipulated to be high quality tended to be rated higher by participants than content manipulated to be lower quality). Three of the items included in the analysis (items 10, 11 and 12) did not yield statistically significant differences between the high and low quality content. The authors noted that future validity studies employing larger and more diverse samples of stakeholders are warranted to increase confidence in the validity of the tool.
Reliability properties meet accepted standards.
Brouwers et al., 2010 (Part 1) conducted an exploratory analysis of the internal consistency of the AGREE II instrument. The authors noted that further analysis of the measurement properties of the instrument is required. Chronbach alpha scores measuring internal consistency of the six domains ranged from 0.64 to 0.89. Only two of the domains achieved an alpha value that met conventionally accepted standards for internal consistency (alpha value above 0.8). It should be noted, however, that the internal consistency of the domains is consistent with ranges reported for the original AGREE instrument (The AGREE Collaboration, 2003).Inter-rater reliability was adequate. The number of appraisers required to reach a level of inter-rater reliability of 0.7 ranged from two to five across domains.
Melissa C. Brouwers
AGREE Research Trust
Method of Development
The AGREE (Appraisal of Guidelines for Research and Evaluation) Next Steps Consortium was formed with several members from the original team, the AGREE Collaboration, to refine the original AGREE instrument. The goals of the AGREE Next Steps Consortium were to: improve the measurement properties of the original AGREE instrument, including the tool's reliability and validity; refine the instrument's items to better meet the needs of intended users; and improve supporting documents (the user guide) to facilitate use of the instrument. The Consortium's efforts resulted in the publication of the AGREE II instrument. The AGREE Research Trust (ART) is an independent body that manages the interests of the AGREE enterprise, and supports a research agenda regarding its development (http://www.agreetrust.org).
Melissa C. Brouwers
Cancer Care Ontario
11 Concession St
Hamilton, ON L8V 1C3
Phone: (905) 527-4322, ext. 42824
These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.