Evaluation & Dissemination

Martin Oliver & Grainne Conole

An overview of the afternoon

     A recap of EFFECTS & of the topics for today An introduction to evaluation and dissemination An emphasis on using these as strategic and political tools Asking for you to think about experiences and understanding, and using these to generate ideas, principles, etc. (A file with this information will be circulated afterwards.) Hand out some papers & these slides at the end

Setting the scene

 In 1997, professional development of academics was put on the national agenda   A group of people got together to think about how to develop academics to use new technologies… 1998: “EFFECTS (Effective Frameworks for Embedding C&IT with Targeted Support) ” project starts with TLTP funding

Setting the scene

   Committed to a number of values, including – – – Local implementation (not “one size fits all”) Scholarship (don’t just do it, think about it) Consultation and discussion (this isn’t just about us) Reflected in the project outcomes – – – Strong commitment to evaluation Strong commitment to research Track record of dissemination Also incorporated into learning outcomes

Setting the scene

   We wanted staff who were informed, not just trained We wanted to help people develop (not do things to them) A slogan emerged from one project meeting (and made it onto the cover of the final evaluation report)  “EFFECTS – not so much a framework as an agenda for change” This sense of political action was important – a theme for today’s seminar

Setting the scene

 Two EFFECTS outcomes for this afternoon: – –

Outcome 5: Evaluated impact of the interventions

This will include evidence that you have:

Evaluated the impact of the incorporation of technology on students and colleagues. Maintained an awareness of external changes and made adaptations as necessary.

– –

Outcome 6: Disseminated the findings of the evaluation

This will include evidence that you have:

Provided feedback for students and colleagues and disseminated experience and findings to department or more widely.

Before we give you information…

    We want to start by getting you thinking about evaluation and dissemination First, think about different times when you’ve been involved in evaluation (either doing it or having it done to you!) Then, write down: – – A brief description of the best experience you’ve had with evaluation A brief description of the worst experience you’ve had with evaluation We’ll spend about five minutes on this

But what is evaluation?

   It’s not one “thing” – so there’s no single definition However, a useful starting point is: “The process of making judgements about the worth (costs and values) of something” Also used to describe… – Descriptive studies – – – – Intervention studies (e.g. formative evaluation) Empirical research Monitoring Quality Assurance processes

But what is evaluation?

 What do these things have in common?

– Evaluators. “Evaluation is what evaluators do.” (A community of practice kind of definition) – – Empiricism. Evaluation involves judgements about data.

For the most part, judgement (the “value” in “evaluate”)

But what is evaluation?

 And what do these activities look like?

– – Data collection of various types  Interviews, surveys, finance spreadsheets, focus group transcripts, documents, tallies of promotions, emails… No inherent reason why it has to be done a particular way  Evaluation doesn’t have to involve interviews, or randomised control trials, or…  Choice of method depends on the person, the situation, the audience and so on

Utilization-focused evaluation

   Evaluation in EFFECTS followed a particular tradition: utilization-focused evaluation (Patton, 1997) A philosophy that arose from: – – The realisation that no-one read evaluation reports The feeling that the qualitative/quantitative paradigm war was never going to be ‘solved’ Argues that evaluations should be judged against how well they help people to do things …and not on whether they are ‘good’ technically (e.g. ‘valid’), of a particular type (e.g. ‘experimental’) or by particular types of people (e.g. external evaluators)

Utilization-focused evaluation

  Communication is key to this kind of evaluation – No matter how ‘good’ the study, it’s useless if the people who need the information don’t get it in time Closely related to ideas of democratic and emancipatory evaluation – – – We need to understand how other people understand this situation We need to get groups (e.g. policy makers, academics) talking to each other Evaluation can provide a framework in which this can happen

Now back to you!

   Spend a short while thinking: – – Has this made you think of any other experiences of evaluation?

Do these ideas help you to make sense of your experience of evaluation at all?

And spend a short while talking: – With a convenient group of people, compare your experiences –

Why

were these good or bad? (What feature or quality made it a good or bad experience?) We’ll gather some suggestions from you after a few minutes

But we want more…

 Think about any involvement you’ve had with project dissemination  As before, write down: – – A description of the best experience you’ve had A description of the worse experience you’ve had  Take a few minutes to do this on your own

And what’s dissemination?

  Sounds like a stupid question But… if we’ve rejected the idea we can ‘transmit’ learning to students, why do we persist in thinking we can just ‘transmit’ findings to peers?

 This is not about volume (how many people, how many papers, how loud you talk) it’s about making meaning

And what’s dissemination?

  So how can we make sure our messages are meaningful to people?

In EFFECTS, we tried: – – – – Giving out drafts to see what people thought, e.g. at workshop sessions in ALT C, so other people’s voices were represented too Running workshops so that the ideas could be discussed, not just presented Working with people (“partner sites”) so that experiences could be shared and jointly interpreted Building relationships with people so that they learnt how to interpret the kinds of things we offered them

And what’s dissemination?

  We also need to think about what ‘counts’ as dissemination – – – – – – Publishing a journal paper?

Giving a workshop?

Producing a leaflet?

A project team meeting?

Chatting to a colleague over coffee?

Moaning to your partner about work?

Formal project evaluation favours ‘obvious’ forms, but studies of change in organisation suggest that ‘invisible’ forms might be more effective – Whether they’re more powerful or not, they’re different

And what’s dissemination?

  A rhetorical question: what method of dissemination would you find most meaningful: – – – A paper about evaluating EFFECTS?

A presentation about evaluating EFFECTS?

Chatting with people who evaluated EFFECTS about their experience?

– Being asked to take ideas from EFFECTS, relate them to what you do and compare this with others?

We’re offering all of these because different people might respond differently to each

Back to you - again

   Spend a short while thinking: – – Has this made you think of any other experiences of dissemination?

Do these ideas help you to make sense of your experience of dissemination at all?

And spend a short while talking: – With a convenient group of people, compare your experiences –

Why

were these good or bad? (What feature or quality made it a good or bad experience?) We’ll gather some suggestions from you after a few minutes

Drawing this first part together

 Developing some principles – We’ve collected qualities of good and bad evaluation, and of good and bad dissemination – How can we use this understanding to guide what we do?

 You volunteer some principles for evaluation and then for dissemination and we’ll record them

A tool for planning

    The Evaluation Toolkit – an online, step-by-step guide to help people plan evaluation & dissemination activities Provides ‘layered’ guidance on the steps, process, associated resources & issues for each stage Consists of three stages: planning, advising and presenting Available online at http://www.ltss.bris.ac.uk/jcalt/

Evaluation Planner

 Five steps: – What are you evaluating?

– – Reasons Context – Who is it for – Devising the question

Working through this

    What are you evaluating?

– Different kinds of evaluation, e.g. of a web site, a project, a strategy, a teaching innovation Reasons – why are you evaluating this?

– Validation, monitoring, research, justification, improving, selecting, to provide evidence Context – Scope and constraints of your evaluation Who is it for?

– – Identifying key stakeholders, their needs and interests Students, managers, funders, colleagues

Working through this

  Devising the question – Using the previous steps, brainstorm different ways of formulating questions – Try to devise a range of question types (e.g. comparisons, contrasts, explorations, quantities, negatives) Keep it simple – – Focus on key stakeholders and key questions Easy to get out of hand – no more than 3 stakeholders recommended!

Evaluation Adviser

 Two (big) steps: – – Data capture   Choosing the methods to use Describing how you’re going to use this in practice (when, what with, under what constraints) Data analysis   Choosing the methods to use Describing how you’re going to use this in practice (when, what with, under what constraints)

Working through this

 Data capture methods – Mapping your evaluation questions to appropriate methods – – – Take account of your own level of expertise and available time Be aware of what each approach was designed to do (don’t use stats on a group of four people or try to interview 200…) Use a variety of methods to build a coherent picture (triangulation)

Working through this

 Five common methods – – Focus groups  A quick way of getting a range of views/ideas, good for exploration  Not necessarily representative; can go off-topic, and individuals can dominate them Interviews  A way to understand people’s experiences of things; provides in-depth picture of individual views  Time consuming

Working through this

– – – Surveys  Good broad overview of issues  Can be time consuming to analyse; need to be careful when devising questions Usage logs  Readily available  Need to be careful when interpreting what these mean Experiments  Good to compare two things  Difficult to do controlled studies in educational settings; can raise ethical issues

Working through this

 Data analysis – Need to map methods to types of data – – Be aware of your expertise and time As before, be aware of what each approach can and can’t do

Working through this

 Four common examples – – – – Grounded theory  Doesn’t pre-suppose particular outcomes  Takes ages to do well, requires iterative data collection Statistical analysis   Can use standard methods to analyse things quickly You need to know what you’re doing Narrative case study   Gives a rich, contextual picture Isn’t generalisable Pre-determined list of categories   Builds on previous research May not map to this particular situation

Evaluation Presenter

 Two steps: – Closing the loop (reflecting on the process) – Presentation tools  Selecting the tools to use  Describing how you’re going to use these in practice

Working through this

 Six common presentation tools: – – – Journal article  Academic credibility  Long lead-time, might only reach a narrow group of people Newsletters  Quick  Disposable Email lists  Quicker & targeted to particular groups  May not be read

Working through this

– – – Committee reports  Specific stakeholders; can be used for political mileage  Counter-politics Verbal presentation  Quick and easy to do, targeted to particular audiences (responsive)  Transient Workshops  Allows you to work through issues in detail  Time consuming, reaching only small groups

Planning a study

 Organise yourselves into small groups – – – – – – Decide whose study to focus on Look through the summary plan to see how it’s been described Work through the steps of the toolkit, making notes about how your own study might look (about 15 minutes) Choose whether to try and work through the whole plan or whether to spend most of the time discussing particular sections (You’ll need to come back to this at the end of the session) Two or three groups to volunteer to describe interesting features of their plans

Thinking strategically about evaluation

 Think through individually – Key barriers and enablers – individual, departmental, institutional, and external  – Who are the key people and committees to target Think of key people (internal and external), committees, etc., where power lies Share experiences – in pairs and with group

Thinking strategically about evaluation

  Example drivers: – Quality audit, institutional audit, learning & teaching strategies, operational plans, new appointments of key people, external drivers (e.g. funding) Current examples, – new academy (with Liz Beaty in place) – – – Learning and teaching strategies Using research initiatives as a Trojan horse Beware different strategies might work at different times; also some will work in some institutions and not others!

Thinking strategically about dissemination

    Think about different ways of disseminating – – What formats to use Who to target Now think about when it would be most effective to do dissemination – Draw up relevant lifecycles (e.g. academic & other internal lifecycles, relevant external events) – Consider how these can be targeted and used Make a personal list for the study you’ve got in mind Share your timetable in pairs, then with the group

Thinking strategically about dissemination

 Some things to consider: – – – – – – These might be the same stakeholders as for evaluation, but they might not be!

Critical times of the year: when to and when not to disseminate Start of the year, exams, as part of other development activities/events Think ahead of time, and work in things such as using external speakers Not just about presenting: e.g. input into strategic plans, operational plans, etc.

Indirect dissemination through others can be very effective!

Thinking personally about evaluation

    Leaving aside what evaluation can do for your project… …what can evaluation do for you?

At a personal level – – What do you hope to learn?

What might you gain?

– Who do you hope to persuade?

– What problems might you cause?

Spend five minutes writing down a list – this is one you

don’t

have to share!

Thinking personally about evaluation

      Evaluation is (should be) a learning experience It’s a chance for you to make connections with people It’s a chance to associate with (or criticise – constructively!) a project It can be a chance to build goodwill by giving good advice or helping solve problems for the project team Reporting findings gets your name in front of funders/policy people/managers/committees You might be able to publish something based on the study

Thinking personally about evaluation

   These aren’t things that are often talked about If you’re an evaluator, you have power and opportunity – so be honest about it!

The potential problem: being professional, and conflicts of interest – – – Would any of these personal aspirations prevent you from doing your role well?

Would any affect timeliness, usefulness, how informative the evaluation was, etc.?

Which can you pursue, and which might you have to give up in order to do the best job for your clients?

Thinking personally about evaluation

 So, back to your personal lists: – – – – Think creatively about what your evaluation might enable you to do. Are there other people this could help you meet or influence, for example?

Think about the tensions between your personal aspirations and what you might call your professional duty – where might conflicts arise?

Revise your list of personal aspirations in light of these exercises Are there any examples of things that haven’t been mentioned you’d be willing to share? (Call them out!)

Being timely

    You’ve established aims for your project and for yourself You’ve thought about how you’re going to gather the data You’ve thought about who wants to know what So when’s all this going to happen?

Being timely

 Evaluation is time consuming – If you’re lucky, you’ll have a useful plan by the end of today. Some plans take days of discussion time – particularly those that are politically sensitive.

– Gathering and analysing data takes longer than you think (e.g. 1 hr of interview taking 4 hrs transcription before analysis starts) – Writing can be time consuming, especially in teams

Being timely

     It’s not just the quantity of time, though… Do you need data from students?

– – When will they be able and willing to provide it?

Do they disappear just before exams, never to return?

Do you need data from staff?

– Are they busy all term and/or absent all summer?

When do you have free time to analyse all this?

– – Can you set aside time as part of your job?

Do you need to make time by giving up other things?

When does it have to be done by?

– Which committee will you report to, and when does it meet?

Being timely

 Are there opportunities or problems on the horizon that might influence what you do?

– – – – People are often more interested in evaluations just before a quality audit Documents might be useful in gaining ‘points’ for your department if you can provide evidence of furthering strategic priorities or fitting with the learning & teaching strategy You might find a controversial course proposal is helped if you can append an evaluation of potential students’ needs; so can you evaluate these in time?

Evidence of success might help in terms of gaining access to funding (internally or externally)

Being timely

  Think about your study in terms of time; for example: – – Is the volume of work you’ve planned realistic?

Is the timing of work you’ve planned practical?

– – – Are there going to be any enforced delays?

Are there any important or immovable deadlines?

Do you need to catch particular people before they get busy/go on leave/leave, or want to wait until someone new is in post?

Use this to sketch out a plan for your study

How do I know impact when I see it?

   Lots of talk about the process of judging, but so far not much talk about judgement itself What counts as evidence of ‘impact’? (What do we mean by ‘impact’ anyhow?) And when you come to it, what exactly is e-learning, or staff development, or teaching…?

We can’t make judgements without making assumptions – so let’s be honest about what we’re assuming

“I was proceeding across campus in an orderly fashion when…”

     Another task for you to do Imagine you’re a detective, and you have been dispatched to an institution where the awful crime of ‘staff development’ appears to have taken place Your task is to make a case to prove that a particular person or project is responsible for doing this to staff What evidence would you look for? How would you use this to argue guilt?

Spend five minutes

on your own

planning you investigation and case

“I was proceeding across campus in an orderly fashion when…”

 Now get together in convenient groups  Spend a few minutes – Each of you present your case – The job of the listeners is to point out weaknesses in the case (“Have you checked their alibi? What if they were covering up for someone else? Do they really know what they’re saying?”)

The problem with evidence

 It’s not always easy to be convincing; e.g.

– – Documenting that things have happened doesn’t tell you

why

they happened Documenting people’s reasons only gives you their (partial) perspective on a situation – Measuring things (e.g. exam performance) tells you nothing about things you might not have measured (e.g. learning) (An aside: think about the rhetoric of models – if they tell you that the world works a certain way, they stop you from looking at things that

don’t

work that way)

The problem with evidence

  The importance of triangulation – – – Any kind of evidence (interviews, surveys, etc.) only gives you part of the story Any source of evidence (learners, academics, managers, etc.) only gives you part of the story Comparing and synthesising across partial accounts gives you a fuller (but never full!) story The importance of modest claims – – “Staff perceived that the workshops changed their lives” “This study demonstrated that re-training staff improved retention. However, it may be that our model is too simplistic, and factors such as the cost of education also had a role to play, even though we could not consider this here.”

Judging things

 We’d like to hear: – Some examples of convincing cases (and why you thought they were convincing) – Some examples of unconvincing cases (and why you thought they were unconvincing)

Drawing it all together

  By way of a recap, we’ve covered: – Definitions – – – – – Principles Strategy (people and politics) Personal politics Timeliness Judgement Are there any other topics you want to raise for discussion at this point?

Drawing it all together

 Time to make use of all this – Go back to the study plans you worked on – Discuss how you would change this in light of this afternoon’s work

So, what have you learnt?

 We’d like you to pause, then share in groups:  – – – Any major changes (in terms of focus and approach) in the study you were thinking about Anything you’ve learnt that you didn’t expect Any revelations you’ve had about your personal situation, and how to develop it We’ll then ask groups to share some examples with everyone

The outputs from today

 What do we think you should have got from this?

– Formal stuff: papers, overheads – – – Generated stuff: experiences of evaluation, experiences of dissemination, principles for these, a list of personal aspirations Personal stuff: the plans (for projects and for personal aspirations) that you’ve produced Intangible stuff: the contacts over coffee or from group work, the discussions that you’ve had, the concepts you’ve acquired and will take away

The outputs from today

  What evidence do we have (at least in theory!) that you’ve learnt something from this?

– – The things you just told us you’d learnt!

Revisions of your lists: evidence you’ve changed your understanding and beliefs – Production of outputs: our co-construction of understanding (e.g. of principles of good evaluation) So could we claim that this session has developed staff…?

– …and on that note we’ll call a halt!