National Collaborating Centre for Methods and Tools

Relevance for Public Health

Public health decision-makers, researchers and program evaluators will benefit from the guidelines in this tool to determine how evidence has been used in decision making. For instance, this tool can be used to assess the impact of evidence in policy briefs, media campaigns and program implementation.

Description

This tool provides an inventory of impact measures for health researchers to assess how research has been used to inform decision making.
Decision-makers and researchers can use the tool to measure the impact of health research in different contexts and for different audiences. It is helpful to target impact measures to specific audiences, rather than using a one-size-fits-all approach. These audiences include the general public, patients, clinicians, health managers, R&D officers (in private firms) and public policy-makers.

Four approaches linking research to action inform the different types of health impact measures discussed in this tool. These approaches can be used individually or in combination to increase the use of research in practice.

1. Producer-push efforts, where researchers promote the use of research to decision-makers. This approach may be especially useful if potential research users are unaware of particular research findings when considering an issue.
2. User-pull efforts involve decision-makers 'reaching in' to the research community to extract information relevant for making a decision. This approach may be used when research users have an identified information gap and are accessing researchers to address this gap in a timely manner.
3. Linkage and exchange efforts where research users and producers form partnerships to increase understanding, interactions and influence between these two groups. This effort could involve decision-makers being involved in setting research agendas and researchers acting as advisors to decision-makers in the decision-making process.
4. Integrated efforts, through large-scale knowledge translation platforms, include elements of the other three approaches.

This tool is based on five underlying assumptions. Impact measures of health research are based on:

1. An understanding of how best to transfer research to action for a specific action; not every approach to link research to action is appropriate for all situations.
2. The ability to assess across situations for similar types of decisions by similar decision-makers (comparative analysis).
3. The ability to assess the influence of research on decision making, and not the effect of research on health outcomes. The effect of research on health status is a separate endeavour.
4. Cultural shifts amongst researchers and decision-makers to facilitate ongoing research use by decision-makers is an appropriate measure of the future impact of research.
5. The recognition that a single tool cannot fully capture the impacts of all evidence types.

This resource recognizes that a number of factors influence the impact of health research on the decision-making process. Some of the factors influencing research use by decision-makers include: having access to information and research; the relevance of research findings to the decision-making context; the time required to read and interpret research; and trust in the research and research source.

Program directors, program managers, research and evaluation specialists, policy analysts, team leaders and other staff would be involved in identifying impact measures for specific policies and programs, collecting data, analyzing data and intepreting results.

Depending on the initiative and measures being observed, data collection and analysis may be short-term or ongoing. Gaining background knowledge of target audiences would also be helpful in implementing this tool and often requires a significant amount of time.

1. Identify the target audiences for research knowledge - Specify the different types of decisions made by particular target audiences. Recognize the organizational and political factors that influence the decision-making process for different audiences. 
2. Select the appropriate category of impact measures for health research - Refer to the table on page 167 regarding producer-push, user-pull and exchange measures. These measures are further divided into process measures (e.g., number of information requests by decision-makers for a user-pull measure), intermediate outcome measures (e.g., decision-makers' awareness of the research organization's expertise) and outcome measures (e.g., decision-makers' selection of a policy based on research supporting its effectiveness). 
3. Select impact measures based on the resources available to measure impact and other constraints - Examine process measures if resources are limited, or intermediate outcome measures if sufficient resources are available to conduct a survey. Case studies permit assessing outcome measures and how health research was used in a decision-making context with competing influences. Different ways that research can be used in decision making include: instrumental use (using research directly to solve a problem); conceptual use (using evidence indirectly to increase general understanding of an issue through concepts); and symbolic use (using research to justify a position or decision that has already been made).
4. Identify data sources, collect data and analyze data, and identify areas for growth and improvement - Through assessments of process, intermediate outcome and outcome measures, an organization is able to determine how research can be used more effectively in decision making, or how to better package and communicate research findings to facilitate their use by decision-makers. This process involves identifying the conditions in which research is used and not used in decision making. 

Title of Primary Resource

File Attachment

Web-link

Reference

Type of Material

Format

Cost to Access

Language

Conditions for Use

Title of Supplementary Resource

File Attachment

Web-link

Reference

Albert, M. A., Fretheim, A., & Maiga, D. (2007). Factors influencing the utilization of research findings by health policy-makers in a developing country: The selection of Mali's essential medicines. Health Research Policy and Systems, 5(2). doi:10.1186/1478-4505-5-2.

Type of Material

Format

Cost to Access

Language

Conditions for Use

Title of Supplementary Resource

File Attachment

Web-link

Reference

Lavis, J. N., Robertson, D., Woodside, J. M., McLeod, C. B., Abelson, J., & the Knowledge Transfer Study Group. (2003). How can research organizations more effectively transfer research knowledge to decision makers? Milbank Quarterly, 81(2), 221-248.

Type of Material

Format

Cost to Access

Language

Conditions for Use

Title of Supplementary Resource

File Attachment

Web-link

Reference

Lavis, J. N., et al. (2002). Examining the role of health services research in public policymaking. Milbank Quarterly, 80(1), 125-154.

Type of Material

Format

Cost to Access

Language

Conditions for Use

Title of Supplementary Resource

File Attachment

Web-link

Reference

Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review, 39(5), 426-431.

Type of Material

Format

Cost to Access

Language

Conditions for Use

Title of Supplementary Resource

File Attachment

Web-link

Reference

Type of Material

Format

Cost to Access

Language

Conditions for Use