National Collaborating Centre for Methods and Tools

One of the most frustrating experiences I have is that once I have found an article, even a systematic review, sometimes there is no conclusive statement at the end that gives me the definitive direction I need. The paper often ends with another question…what do I do with that kind of ending?

I sometimes feel like I’d rather rely on the experiences of my colleagues…at least their experiences are clear – what worked and what didn’t. The problem with that approach, is that it won’t go anywhere in my health department. They want us to be looking at research evidence when we are making recommendations about programs.

Hi
I am going to answer your questions in turn.

1. There is no conclusive statement at end of study

This is true – sometimes there is no “actionable message” to studies and reviews. That means that the research is not conclusive and it ends with more questions than it started.
The good news is - you looked at the evidence.
The bad news is - the state of the research for that question, so far does not provide an answer.

In the absence of any studies, we consult expert opinion.

2. Relying on experiences of colleagues

Yes, it might save you some time in the search step of evidence-informed decision making (finding the answer), but what if you get the wrong answer? Given the average age of public health practitioners in Canada is mid to late 40s, and they have been practicing based on their education (now 20 years old), could they be missing anything???? Experience is very valuable, and creates that expert practitioner level where the whole picture can be seen more readily, and intuition is better honed. However, they could also have been doing the wrong thing for 20 years.

We have an ethical responsibility to look for the best research evidence, but then using the venn diagram model (see www.nccmt.ca), expert opinion has to be brought to bear on any decision, including research evidence, evidence regarding the broader context, resources, and most importantly, client or population preferences.

Neither research evidence nor expert opinion is enough, on its own. And congratulations to your workplace for creating a culture that supports the expectation of research use.


Donna

In order to reply to the 'Anonymous observation of 2009-09-28' what do you think about the severeal approaches to synthesize evidence?
I'm working on the 'Realist Synthesis' (Pawson R, Edidence based policy, Dage, 2006) and I'd like to exchange my experience with other people on thi issue.

In thinking about the posting above, I believe that all the different types of evidence are needed to bring to light on an issue. So perhaps we could address this in light of a real life scenario for application.

In northern Manitoba, communities have issues about access to fresh fruits & veg in their local stores. If the stores have brought in fresh f&v, the costs are high and if you don't get to the store within the first day or two, the shelves are empty till the next shipment. To address access issues, some communities are beginning to implement community gardens to have fresh veggies. Some of the impacts are teaching skills to younger generations, physical activity, reducing social isolation.

I have done some preliminary searches in systematic reviews, and community gardening does not appear. In fact, there is no solid evidence about the effectiveness of community gardening. As I am writing this, I am thinking, "What is the question that I want to address - effectiveness of community gardening to increase access to fresh vegetables? Or improving physical activity? Do we know if community gardens are effective?" So asking the right question is a big part of finding evidence.

So what do we do if there really isn't a lot of evidence to support a program but the momentum is there from the community and it is addressing the issue the community has identified? Do we stop the program because of lack of evidence? The communities have begun a project to meet a need that perhaps is not supported by evidence but is providing an answer to access. How do we support that?

Perhaps one way of supporting is then to see how can we "research" it to become part of the world of acceptable evidence. Develop the tools, do the studies in various communities and parts of the country to bring this intervention into the world of evidence to provide it with credibility that organizational structures are beginning to require (kudos to your org!). We need some better dialogue between researchers and communities about the issues that communities identify as needs.

Evidence is not just a research experiment about what works/doesn't work. It should be relevant to the community, address an identified need, and involve the community. Just like finding evidence to support a decision needs to be done in light of meeting not only the organizational requirements but also in meeting the needs of the community. Balancing evidence from different sources becomes important because no one source has all the evidence either. Look at the evidence based on the community, not just what the research says. Use good community development principles.

As practitioners, we have a responsibility not only to our communities in finding good evidence, but also motivating good research based on good practices. We need innovative practice to move research along. Not all communities are created equal so knowing the community, finding the champions, using experience are important. No wonder health promotion is an art and not just a science!

re: post dated 2010-04-23

It would be great to hear about your experience with Pawson’s realist synthesis. Are you able to share with the group the topic, and a bit about how realist synthesis differs from Cochrane classic style syntheses?
Thanks in advance!
Donna

Yes, I would like to hear more about people's experiences with realist syntheses. I like the concept but I wonder about rigour and replicability of these reviews. I haven't done one myself.