Tools to evaluate knowledge translation interventions: A systematic review

Van Eerd, D., Cole, D., Keown, K., Irvin, E., Kramer, D., Brenneman Gibson, J., Kohn, M.K. et al. (2011)
Toronto, ON

How to cite this summary:

National Collaborating Centre for Methods and Tools (2012). Tools to evaluate knowledge translation interventions: A systematic review. Hamilton, ON: McMaster University. Retrieved from http://www.nccmt.ca/registry/view/eng/114.html.

Download PDF Summary
Method Knowledge Knowledge Knowledge Evaluating

These summaries are written by the NCCMT to condense and to provide an overview of the resources listed in the Registry of Methods and Tools and to give suggestions for their use in a public health context. For more information on individual methods and tools included in the review, please consult the authors/developers of the original resources.

Relevance for Public Health
 The tools cited in this resource could be used to evaluate knowledge translation activities in a public health setting. For example, instruments could be used to measure changes in practitioners' work and capacity to use research evidence in their practice.
Description
 The Institute for Work and Health (IWH) conducted a systematic review to identify instruments to evaluate knowledge translation (KT) initiatives and their impact. With an increasing emphasis on KT in health and public health, attention is shifting to the effectiveness of KT interventions. However, the effectiveness of KT practices has not been consistently assessed; this may be due to a lack of valid and reliable tools.

The IWH team conducted a comprehensive and systematic review of peer-reviewed literature to find instruments to evaluate KT practices. Instruments included questionnaires, surveys and interviews that collected information from the end-users of research evidence (such as practitioners). 

Knowledge translation, or KTE (knowledge transfer and exchange), was defined as "an iterative and dynamic process by which relevant research information is created, synthesized, disseminated and exchanged through interactive engagement with decision-makers/knowledge-users to improve outcomes, provide more effective services and products, and strengthen the use of evidence in decision-making, practice, planning and policy-making."

The Institute for Work and Health's resources on tools for evaluating KT interventions can be accessed from their website (http://www.iwh.on.ca/sys-reviews/kte-evaluation-tools). The resources include:
  • a summary highlighting key evidence points;
  • a list of promising KT evaluation tools; and
  • the full report.
Evaluation and Measurement Characteristics
Evaluation
 Information not available
Validity
 Not applicable
Reliability
 Not applicable
Methodological Rating
 StrongStrong  (Click here to see rating sheet for this systemic review.)
Implementing the Method/Tool
Who is involved
 Anyone interested in assessing the effectiveness of a knowledge translation activity could use the tools listed in this systematic review.
Time
 Information not available
Additional Resources and/or Skills Needed for Implementation
 Not specified
Steps for Using Method/Tool
 The main findings of the Institute for Work & Health's systematic review are:
  • There are few well-developed instruments to evaluate the implementation and impact of KTE practices.
  • A total of 16 articles describe promising tools to evaluate KTE intitiatives. These instruments show some degree of validity and reliability, and are able to measure changes in knowledge, attitudes and behaviours among decision-makers and stakeholders.

The implications of this systematic review for practitioners and researchers include the following:
  • Consider evaluation instruments found in this review when planning an evaluation of a KTE initiative. When choosing an instrument, look for a clear presentation of the measurement properties.
  • Select or construct well-developed instruments for evaluation.
  • Consider developing instruments that can be used in various contexts. It is important to separate implementation, which is dependent on context, from instrument development, which can be based on theory and be independent of context.
  • Systematically develop and use evaluation instruments with known measurement properties to advance KTE evaluation.
Conditions for Use
 Not specified
Method/Tool Development
Developer(s)
 Dwayne Van Eerd
Institute for Work & Health
Department of Health Studies and Gerontology, University of Waterloo

Donald Cole
Institute for Work & Health
Dalla Lana School of Public Health, University of Toronto

Kiera Keown
Institute for Work & Health

Emma Irvin
Institute for Work & Health

Desre Kramer
Centre of Research Expertise for the Prevention of Musculoskeletal Disorders, University of Waterloo

Jane Brenneman Gibson
Institute for Work & Health

Melanie Kazman Kohn
St. Michael's Hospital
Faculty of Medicine, Department of Health Policy Management and Evaluation, University of Toronto

Quenby Mahood
Institute for Work & Health

Tesha Slack
Institute for Work & Health

Benjamin Amick III
Institute for Work & Health

David Phipps
Office of Research Services, York University

John Garcia
Department of Health Studies and Gerontology, University of Waterloo

Sara Morassaei
Institute for Work & Health
Method of Development
 The review was conducted by a multidisciplinary team who searched for articles on knowledge translation in nine databases from different disciplines (including medicine, psychology, education, agriculture and others), in addition to hand searching specific journals and consulting with content experts. The initial search yielded 9,998 articles after duplicates were removed.

The articles were reviewed in terms of their relevance to the research question: "Does the article describe a KTE outcome or a tool to measure a KTE outcome as a result of a KTE application?"

The review team identified 346 potential articles as relevant, and examined qualitative and quantitative articles. Qualitative articles described specific approaches to evaluating KTE, while quantitative articles included instruments used to evaluate KTE (such as questionnaires, surveys and interviews).

A total of 12 qualitative and 54 quantitative articles met the team's inclusion criteria. Information from all 66 articles was collated and details about the measurement properties of instruments from the quantitative articles were collected. The review team collected information on the validity, reliability and responsiveness of instruments. Of these 54 instruments, 16 were identified as promising based on their measurement properties.
Release Date
 2011
Contact Person/Source
 Dwayne Van Eerd
Institute for Work & Health
481 University Ave., Suite 800
Toronto ON M5G 2E9
email:dvaneerd@iwh.on.ca
website: www.iwh.on.ca

Resources

Title of Primary Resource
 Report on Knowledge Transfer and Exchange Practices: A systematic review of the quality and types of instruments used to assess KTE implementation and impact.
File Attachment
 None
Web-link
 http://www.iwh.on.ca/sys-reviews/kte-evaluation-tools
Reference
 Van Eerd, D., Cole, D., Keown, K., Irvin, E., Kramer, D., Brenneman Gibson, J., Kohn, M.K., Mahood, Q., Slack, T., Amick III, B.C., Phipps, D., Garcia, J., & Morassaei, S. (2011). A systematic review of the quality and types of instruments used to assess KTE implementation and impact. Toronto, ON: Institute for Work & Health.
Type of Material
 Report
Format
 On-line Access
Cost to Access
  
Language
 English
Conditions for Use
 Copyright © 2011 Institute for Work & Health 

Title of Supplementary Resource
 Sharing best evidence: Promising KTE evaluation tools
File Attachment
 None
Web-link
 http://www.iwh.on.ca/sys-reviews/kte-evaluation-tools
Reference
 Institute for Work & Health (2011). Sharing best evidence: Promising KTE evaluation tools. Knowledge Transfer & Exchange, Institute for Work & Health.
Type of Material
 Fact sheet
Format
 On-line Access
Cost to Access
 None 
Language
 English
Conditions for Use
 Not specified 

Title of Supplementary Resource
 Sharing best evidence: Finding tools to measure the impact of KTE activities
File Attachment
 None
Web-link
 http://www.iwh.on.ca/sys-reviews/kte-evaluation-tools
Reference
 Institute for Work & Health (2011). Sharing best evidence: Finding tools to measure the impact of KTE activities. Knowledge Transfer & Exchange, Institute for Work & Health.
Type of Material
 Summary statement
Format
 On-line Access
Cost to Access
 None 
Language
 English
Conditions for Use
 Not specified