Transcript FAMILIES FIRST EDMONTON: THE PROCESS OF USING …
FAMILIES FIRST EDMONTON: THE PROCESS OF USING LOGIC MODELING TO DEVELOP A COMMUNITY-BASED RESEARCH INTERVENTION
ACSW Conference 2008 Solina Richter, D Cur, RN, University of Alberta Scott Smillie, B.S.W., R.S.W., Manager of Family Programs, YMCA of Edmonton Perdita Baier, B.S.W., R.S.W., Project Manager, FFE Jane Drummond, PhD, RN, University of Alberta 1
Introduction
Presenters Participants (Round Robin) -A bit about yourselves -Interest in this workshop 2
Objectives
You will: Be informed about a community-based research project focused on service integration to low-income families.
Consider the value and complexities of community based randomized control trials.
Explore the logic model and RFP process used to design and implement the practices of service integration in the research project.
3
Objectives
You will : Gain insight into the value of using logic models to strengthen intervention fidelity in complex community settings, and Examine the challenges and determinants of a successful service delivery-research partnership. 4
FFE Overview
Community-based randomized control trial Four groups with 1200 families in total Goal is to link families Families are followed for 2 years of service delivery and up to 3 years of follow-up Provides opportunity for evidence-based decision making 5
Historical Overview
How FFE was initiated?
Quality of Life Commission
“When the Bough Breaks” (Browne et al., 2001)
FFE steering committee Community-University Partnership for the Study of Children, Youth, and Families (CUP) 6
Development Phase
Recreation coordination pilot project Development of project documents Program logic model (PLM) Request for proposal (RFP) Dormancy phase Research Funding 7
The RFP Process
Request for proposal (RFP) process RFP based on PLM Families Matter Partnership Initiative (FMPI) YMCA Bent Arrow Kara Family Resource Centre Multicultural Health Brokers The launch of Families First Edmonton 8
FFE Is A Community-Based Research Initiative
What is Community Based Research: Community-based research in public health can be defined as a partnership approach to research that equitably involves community members, organizational representatives, and researchers in all aspects of the research process to enhance understanding of a given phenomenon and integrate the knowledge gained with action to improve the health and well-being of community members .
Source: Israel, B.A., Shulz, A.J., Parker, E.A., Becker, A.B. Review of Community-based research: assessing partnership approaches to improve public health. Ann Rev Public Health 19: 173-202 (1998).
9
Traditional and Mainstream
What do we mean by traditional or mainstream research?
Traditional or mainstream research is the type of research that is most often published in peer-reviewed journals. It is usually concerned with: Generating reliable, replicable knowledge Being ‘scientific’ and rigorous Maintaining objectivity Retaining ‘distance’ from its subject matter Being neutral, impartial and avoiding value-based judgements.
Randomized control trials have been seen as the gold standard for this type of research.
Source: Research as empowerment, Joseph Rowntree Foundation, The Toronto Group, 2005.
10
Traditional versus Mainstream Cont.
However, this description of traditional research fails to recognize: The social and political context of research The fallibility of researchers The range of knowledge being generated Issues of ethics Participatory research and emancipatory research appear to continue to be seen as the poor relations to traditional or mainstream research.
11
The values of a Community-based RCT
Although more complex, it is also more rich. In the FFE Context: The common work of data management, participant recruitment, identification and systems responses to service barriers, and planning and implementing intervention practices, all become more meaningful.
Non profit partners add social value. (Service delivery partners and Sounding Board partners).
12
The values of a Community-based RCT
Through collaboration, a broader range of community assets are mobilized.
A safer and more neutral forum can be nurtured to incorporate alternate views and innovative solutions to systems related problems/barriers.
Provides an opportunity for participants to be heard, engaged in the processes, and provides for a more direct link to policy makers.
13
The values of a Community-based RCT
Challenges more traditional notions of research and provides an opportunity to test other more participatory approaches in a balanced and meaningful manner.
Ensures for more checks and balances and a greater degree of accountability.
14
RCT- What is it?
Generally quantitative in nature Controlled experiment is considered to be the gold standard for yielding reliable evidence about causes and effects Commonly used in drug trials, but can also be developed for many other areas (in case a community-based RCT) 15
Components
Involves randomization and a control group Blinding Endpoints Sample size and power Participant recruitment Application to FFE 16
The complexities of a Community-based RCT
Effectiveness versus efficacy trial CBR is collaborative in nature. The larger the collaboration the more complex issues such as planning, implementation, monitoring, decision making, coordination and problem solving, become.
CBR takes longer and time becomes a variable to manage.
CBR requires navigating different agendas and outcome needs of various stakeholders.
17
The complexities of a Community-based RCT
The range of stakeholders can also mean a range of organizational and professional cultures. (different language, different color lenses, different needs/agendas, different understanding of authority, etc.) Necessitates the sharing and development of knowledge required to work together, i.e. service delivery decision makers learning about research concepts and research learning about service delivery management needs in order to meet research rigour, etc.
18
The complexities of a Community-based RCT
Randomized control trials are more complex to manage in community settings than in medical or pharmaceutical settings.
19
The Logic model
“Logic Models are typical diagrams, flow sheets, or some other type of visual schematic that conveys relationships between contextual factors and programmatic inputs, processes and outcomes.” “.. Link in a chain of reasoning about what causes what, in a relationship to the desired outcome or goal” (Schmitz, 1999).
20
People say that a logic model is a roadmap.
"If you don't know where you are going, how are you going to get their.“ Yogi Berra Where are you going?
How will you get there?
What will show that you have arrived?
21
The logic model describes logical linkages among program resources, activities, outputs, audiences and outcomes These linkages relate to a specific problem.
The tool have an accountability aspect: Who is responsible for what?
What resources are committed?
What are the deliverables?
22
23 Simplest form
INPUTS OUTPUTS OUTCOMES
Reference: University of Wisconsin Extension, Program Development and Evaluation 23
• • • •
What does a logic model look like?
Graphic display of boxes and arrows; vertical or horizontal – Relationships, linkages Any shape possible – Circular, dynamic Level of detail – – Simple Complex Multiple models Reference: University of Wisconsin-Extension, Program Development and Evaluation
What is the logic model for FFE
It provides a common approach for integrating planning, implementation, evaluation and reporting.
25
FFE Service delivery Logic model
Complex
Vision of Families First Edmonton Mission of Families First Edmonton Statement of need Strategy Rationale Goals 26
FFE Service delivery Logic model
Every goal has a : Long term Outcome Intermediate Outcome Short-term Outcome Activities Outputs Inputs Indicator of success Measurement tools 27
FFE goals
Families and children participating in Families First Edmonton will experience improved family health and well-being.
Families will be increasingly empowered to access services in a more preventative and proactive way resulting in cost savings.
Collaboration of partners, researchers, service providers and families will create more proactive and responsive service delivery systems.
28
FFE goals
The family healthy lifestyle and recreation program practices will be well-documented enabling transferability.
Research will result in knowledge that informs health and social policy; service delivery practices, systems collaborations, and community research - program development & evaluation practices." 29
The use of the logic model to capture the service integration practices
Grounded theory methodology (What happened during this process?) Data collection included: One-to-one interviews (original logic model developers, managers at FMPI) Focus groups (logic model re-working group, electronic database developers) 30
The use of the logic model to capture the service integration practices
The long and disjointed history in developing the project, resulted in disengagement of researcher input from the logic model/RFP /contracting process. This was due partly to naivety, inexperience and researcher willingness to trust the community-based process.
31
The use of the logic model to implement the service integration practices
Re-integrating of researcher input into the process had to take place. The logic model redevelopment process emphasizes the importance of re-engagement between the researchers and the contracted agency.
32
Research Question?
How did the process of engagement between the contracted agency and the researchers, using the logic model as a vehicle, contribute to the FFE service integration interventions?
33
Face-off Phase: Surrounded by uncertainty and captured by differences
“from the service delivery perspective you know we wrote a proposal and put all these things in it, we come to the table, we’re prepared to do what we wrote because it was effective, then to find out that there’s this other piece on the table that we don’t quite – doesn’t look familiar and doesn’t seem to be in alignment with what we’ve been thinking, and…” (Focus group logic model working group).
34
Face-off Phase: Surrounded by uncertainty and captured by differences
“Yeah I could add that I think what I’ve experienced is a lot of uncertainty in the beginning what the different roles of the researchers have been …” (Focus group logic model working group) “…because sometimes interdisciplinary isn’t as easy as it sounds and there was some barriers and challenges associated with having myself and another person coming from two different backgrounds and trying to figure out what we actually needed to document and capture..” (Database developer interview).
35
Phase of Bumping into each other: acknowledging that there are shared barriers
“…part of the challenge was that we didn’t resource for that time demand and we had other deliverables that we had to be responsive and also in terms of hiring” (Focus group database developers).
“I think technically the challenges were ….. I spend time figuring out terms what were meaningful to Families First and then having to choose from the [electronic database] list, I found this was a barrier and challenge…” (Database developer interview).
36
“I had to back off that and come to terms with oh my …. there’s a loss of autonomy and ownership of what we imagined together and when we were bringing it to the table and then having to formulate something new and I didn’t really have confidence initially that I’d be met halfway in that by the research right because it seemed like the research agenda was going to prevail at all costs and that there wasn’t necessarily the goodwill to recognize what we were bringing in terms of experience…” (Focus group logic model working group).
37
“I think there’s some competing needs there like because you were talking about a bureaucratic organization versus kind of a linear kind of process …” (Focus group logic model working group).
“…of the needs what we have competing needs sometimes or different needs” (Focus group logic model working group).
“ … there were lots of questions about the logic model and trying to have the development of the program fit into the logic model and that was frustrating for us because there wasn’t fitting” (Focus group logic model working group).
38
Starting-to-play Phase: Establishing relationships and getting situated
“Just in terms of having those conversations in the logic model was a real eye-opener for me in understanding the research side and the service delivery side. …., and it is very complex, there’s no doubt and it’s very clear in my mind that service delivery and research have to be at the table, them work together” (Focus group Logic model working group). 39
“… because we are really one big team but we have different jobs to do, and our relationship with the data collectors and the research assistants, the relationship building is really, really important to our ability to be as seamless as possible, and to meet each other’s needs ..” (Focus group database developers).
“…but I was needing researchers to have faith in what we brought and what we know as community agency experts….” (Focus group Logic model working group).
40
“ It was so important to have the dialogue that did happen and the differences of opinion and then I think sometimes we spent an hour talking about one word and what it meant to all of us” (Focus group Logic model working group).
“ ….and more reflection but I think probably that laid some common values or aims...” (Focus group Logic model working group).
41
Establishing a sense of being equal among members
“..because I think it was a real… in a way it was a very team-building kind of process and I think that’s because of the good will on table but I think we could mitigate it and then we’ll pass on some of those learnings for the next time, something like this is done…” (Focus group Logic model working group).
42
Becoming a member and developing a sense of ownership
“ … to me that was about that finding that unifying place because what we want collectively won’t happen unless we both… understand and work together to get it, right, so for me it works both ways. Your openness allowed me to also feel open…” (Focus group Logic model working group).
43
Winning the game Phase: Developing a strong engagement and becoming human
“…so there was a really like you say a human side and a kind of it was like an icebreaker sometimes and you know we sort of spend some time at the beginning just sort of hey how you doing you know checking in with each other and got through all of that and then away we went on the logic model” (Focus group Logic model working group).
44
What is Intervention Fidelity?
“ the adherence and competent delivery of an intervention by the interventionist as set forth in the research plan” Source: Santacroce, Maccarelli, & Grey, 2004,p. 63 45
What is Intervention Fidelity?
Intervention fidelity is about doing what you said you would do.
Intervention fidelity is about protecting your intervention from influences that can “water down’ or “change” the impact of the intervention 46
What is Intervention Fidelity?
The lack of intervention fidelity makes it hard to determine if the intended interventions can be attributed to causing the measured outcomes The strength of a RCT is “controlling” the amount and type of intervention, so that outcomes aren’t attributed to other variables (in-deliberate or unplanned or activities/influences) which create rival hypothesis 47
What is Intervention Fidelity?
What we want to accomplish is to create as much certainty as possible about which practices/interventions resulted in which outcomes. Without this certainty, it is difficult to replicate the intervention. 48
The shortcoming of some similar studies has been that even though positive outcomes were noted, the researchers couldn’t clearly demonstrate the practices that led to those outcomes, making it difficult to replicate. It also makes it difficult to confidently be able to generalize the findings to those outside of the study, who we wish to serve, weakening the impact form a public policy perspective. 49
Components of Intervention Fidelity
Adherence (or integrity)
The degree to which a given intervention is implemented in accordance with essential theoretical and procedural aspects of the model
Competence
The level of interventionist skill in utilizing core intervention techniques and responding to the unique needs of each participant 50 Source: Hogue, Liddle, Singer, & Leckrone, 2005
Components of Intervention Fidelity
Differentiation
The theoretical distinctiveness of the model’s main intervention principles in comparison to those of other models of interest or within efficacy trials, in comparison to those of competing study conditions Source: Hogue, Liddle, Singer, & Leckrone, 2005 51
Importance of intervention fidelity
Increases retention Increases the likelihood of an effect for FFE Increases the likelihood of transferability of FFE Increased the ability to attribute the effect of FFE to the FFE interventions Enhances statistical power 52
Importance of intervention fidelity
Improves FFE intervention through the discipline of consistent monitoring and feedback Is the foundation of evidence-based practice Promotes the avoidance of harm to all partners improves outcome increases the ethical component reduces unintended consequences enhances reputation 53
Replicability Is A Goal Of FFE
FFE Models, Approaches, and Practices,
is a program evaluation that includes intensive analyses of the implementation of the different service-delivery models. These analyses are critical for determining the fidelity and effectiveness of the service-delivery models, and also for documenting practices (including engagement and retention) so that the lessons learned can be maintained and transferred (Appendix 1E).Drummond, J.E., Family First Edmonton, Year 1 Research Proposal, p.12b
54
Researcher questions directed at establishing intervention fidelity for the FFE service integration practices
Adherence:
What are the practices associated with the awareness, knowledge, skills, and attitudes of FFE service integration?
Are practices implemented consistently?
How do the key elements of FFE appear in the service integration practices (family-centred, cultural sensitive, capacity building, reflective practice)?
55
Researcher questions directed at establishing intervention fidelity for the FFE service integration practices
Competence:
How do the interventionists implement the practices associated with the awareness, knowledge, skills and attitudes of FFE service integration?
How do the interventionists focus on the key elements of FFE service integration in their work?
How do the interventionists use their uniqueness in their implementation of the FFE service integration practices?
How does Families Matter support the interventionists in their service integration practices?
56
Researcher questions directed at establishing intervention fidelity for the FFE service integration practices
Differentiation:
Are each of the FFE service integration vehicles (Family Healthy Lifestyle and Recreation Coordination) distinct?
How does Families Matter keep the FFE service integration vehicles distinct?
57
Where are the researchers looking for intervention fidelity in FFE?
Service Integration Practice Key Elements
-practices associated with awareness, knowledge, skills, attitudes
Participant families
- Audit of links - family Service Inventory (FSI) -family-centred, cultural sensitivity, capacity building, reflective practice - Focus group - Family satisfaction survey
Family Workers
Audit of practices used Toolkit - Audit of referrals Focus groups Toolkit
Families Matter Collaboration
-
managers
-
supervisors
-Toolkit (Training, supervision, management, and hiring practices) - Intensity and duration of intervention (dosage) -HOMES Logic model Document analysis HOMES Toolkit Focus groups Individual interviews Source: research working plan 58
Learnings in the development
Research perspective: A need for strong leadership Service delivery input is essential early in the process of intervention design Long term engagement is needed between service providers and researchers 59
Learnings in the development
Universities and funders need to adequately resource and acknowledge trans-disciplinary and inter-sectoral involvement in community-based research. Research needs to be clarified within the community context.
The need to have a ‘vehicle’ to enhance communication between a service delivery and research team. 60
Logic modeling as a Process
The redevelopment of the logic model played an integral role in forming a collaborative approach between research and service delivery. It was very important to the successful implementation of a community-based randomized control trial and establishing a framework for strengthening intervention fidelity.
61
Thank you for your attention For More Information http://www.familiesfirstedmonton.ualberta.ca/ffe_in volved.html
To Contact Us: [email protected]
62