Evaluating knowledge mobilisation: issues, challenges and works-in-progress

The Research Unit for Research Utilisation (RURU) held an informal event to explore issues around the evaluation of knowledge mobilisation work. Around 30 participants from a range of fields including health, education, social care, environmental management and technology development enjoyed the sea views from the Boardroom in the School of Management’s Gateway building and grappled with the issues and challenges of evaluating knowledge mobilisation as seen from the perspective of a range of projects.

Six short presentations from different fields were used to provoke reflection and open up discussions:

The presentations generated lively discussions which continued into the evening over an informal dinner.

Among the issues raised were the following

  • The need to distinguish between the mechanisms of impact and the actual outcomes or impacts;
  • The importance of the timing of an evaluation both in terms of what can be evaluated and how propitious the circumstances are for using the finding and the learning that come from the evaluation;
  • The value of different types of evaluation: for some agencies it may be appropriate to do large evaluations which also contribute to the literature while for others, smaller evaluations that enable the agency to move forward may be helpful and more realistic;
  • The importance of taking into account in any evaluation the different perspectives that different stakeholders have (e.g. the outcomes that matter to the research end-users may be very different from those of the researchers);
  • The importance of surfacing and acknowledging the underlying assumptions that stakeholders hold about knowledge, knowledge mobilisation/exchange and evaluation: these will affect both the knowledge mobilisation/exchange activities and the evaluation;
  • The value of capturing process outcomes as well as end outcomes (e.g. what effects does simply being involved in research have on practitioners over and above the intervention itself? Why is it that some practitioners are keen to be involved in any research study?)
  • The value of building on existing knowledge mobilisation activities and mechanisms when designing activities and their evaluations: what is already working well in this community?
  • The need to avoid ‘impost’: an unacceptable level of burden on the research participants (e.g. social care professionals) and to recognise the importance of reciprocity in qualitative interviews (e.g. the interviewer being open about the research project);
  • The role of emotion, irrationality and power dynamics in social worlds and their impact on knowledge mobilisation and evaluations of knowledge mobilisation activities;
  • The importance of framing questions in constructive ways so that ‘evaluation’ can be seen by those being evaluated more as a process of open reflection (e.g. “What have you learnt that you would like to pass on to others?”) than one of assessment ;
  • The importance of early framing of the research questions and parameters of the evaluation: What are we hoping to achieve? What will we value? How will we know when we have achieved it?