Transcript Powerpoint - SWEAP Official Website
S OCIAL W ORK E DUCATION A SSESSMENT P ROJECT (SWEAP):
Responding to the Challenges of Assessment
Pre-Conference Workshop BPD Annual Conference Louisville, KY March 19, 2014
• • • •
W
ORKSHOP
Introductions
o Introducing SWEAP /Goodbye BEAP o o The SWEAP Team Workshop Participants
Overview of Accreditation
o History of Accreditation o o o Direct vs. Indirect Measures Implicit Curriculum Multiple Measures
O
UTLINE
Organizational Framework for Program Assessment
o EPAS 2008 (&2015) o o o Competencies Characteristic Knowledge, Values & Skills Practice Behaviors
SWEAP Instruments
o Entrance o o o o o Exit Alumni/Graduate Employer Curriculum Instrument (FCAI) Field Instrument (FPPAI) •
BREAK
• • • • •
W
ORKSHOP
O
UTLINE ( CONTINUED )
Unique Benefits of SWEAP Tying SWEAP to Program Assessment
o o o o Linking SWEAP to EPAS Matrix Program Integration Example Using SWEAP beyond EPAS
Sharing Assessment Ideas
o Group Work
Navigating SWEAP
o Website o o o o Ordering Raw Data Policy Processing Reports
Questions
I
NTRODUCING
SWEAP
• • Was BEAP Now SWEAP o o o
Not just for undergraduate programs anymore
Can assess foundation year for graduate programs as well Can be modified for program specific advanced year assessment
T
HE
SWEAP T
EAM
Vicky Buchan
Colorado State University
Kathryn Krase
LIU Brooklyn
Brian Christenson
Lewis-Clark State College
Phil Ng [email protected]
Tobi DeLong Hamilton
Lewis-Clark State College, CDA
Patrick Panos University of Utah [email protected]
Ruth Gerritsen-McKane
University of Utah
Roy (Butch) Rodenhiser
Boise State University
Sarah Jackman [email protected]
I
NTRODUCING
W
ORKSHOP
P
ARTICIPANTS
• • • • Your Name Your Program Your Role Have you used SWEAP?
o If so, which instrument(s)?
O
VERVIEW OF
H
ISTORY OF
A
CCREDITATION
:
CSWE A
CCREDITATION o o o o What was the process like before BEAP?
How has it changed over time?
How do changes to EPAS impact it? Move to competency based educational standards in EPAS 2008
O
VERVIEW OF
A
CCREDITATION
:
D
IRECT VS
. I
NDIRECT
M
EASURES o o o
Direct Measures
• “Student products or performances that demonstrate that specific learning has taken place.” • Examples (SWEAP and Non-SWEAP)
Indirect Measures
• “May imply that learning has taken place (e.g., student perceptions of learning) but do not specifically demonstrate that learning or skill.” • Examples (SWEAP and Non-SWEAP)
Which do you need and why?
O
VERVIEW OF
I
MPLICIT
A
CCREDITATION
:
C
URRICULUM o What is Implicit Curriculum?
• • • “Educational environment in which the explicit curriculum is presented. It is composed of the following elements: the program’s commitment to diversity; admissions policies and procedures; advisement, retention, and termination policies; student participation in governance; faculty; administrative structure; and resources.” (EPAS, 2008) How do you measure it?
Why should you measure it?
O
VERVIEW OF
M
ULTIPLE
A
CCREDITATION
:
M
EASURES o The Importance of Multiple Measures o SWEAP alone is not enough
• •
O
RGANIZATIONAL
P
ROGRAM
A F
RAMEWORK FOR SSESSMENT
EPAS 2008 (& 2015) Working under EPAS 2008 EPAS 2015 underway… expected changes o o Fewer competencies Fewer practice behaviors
•
O
RGANIZATIONAL
P
ROGRAM
A F
RAMEWORK FOR SSESSMENT
C OMPETENCIES
EPAS 2.1—Core Competencies
o Competency-based education o o Outcome performance approach to curriculum design. Measurable practice behaviors comprised of knowledge, values, & skills. o o o Need to demonstrate integration & application competencies in practice with individuals, families, groups, organizations, and communities.
10 competencies listed along with description of characteristic knowledge, values, skills, & practice behaviors that may be used to operationalize the curriculum and assessment methods. Programs may add competencies consistent with their missions and goals.
• • • • • • • • • •
O
RGANIZATIONAL
P
ROGRAM
F
RAMEWORK FOR
A
SSESSMENT
C OMPETENCIES 2.1.1—Identify as a professional social worker and conduct oneself accordingly. 2.1.2—Apply social work ethical principles to guide professional practice. 2.1.3—Apply critical thinking to inform and communicate professional judgments. 2.1.4—Engage diversity and difference in practice. 2.1.5—Advance human rights and social and economic justice. 2.1.6—Engage in research-informed practice and practice informed research. 2.1.7—Apply knowledge of human behavior and the social environment. 2.1.8—Engage in policy practice to advance social and economic well-being and to deliver effective social work services. 2.1.9—Respond to contexts that shape practice. 2.1.10— Engage, assess, intervene, and evaluate with individuals, families, groups, organizations, and communities.
O
RGANIZATIONAL
P
ROGRAM
C HARACTERISTIC K
F
RAMEWORK FOR
A
SSESSMENT
:
NOWLEDGE , V ALUES & S KILLS • • Current focus on measuring practice behaviors.
Don’t forget about knowledge, values & skills.
• • •
O
RGANIZATIONAL
P
ROGRAM
P RACTICE
A F
RAMEWORK FOR SSESSMENT
B EHAVIORS Multiple practice behaviors per competency Each practice behavior MUST be measured for self study/reaccreditation TWO measures required for each practice behavior o At least one measure must be DIRECT
SWEAP I NSTRUMENTS
W
HAT ARE THEY AND HOW DO
I
USE THEM
?
• • • • • Entrance Exit Alumni/Graduate & Employer Curriculum Instrument (FCAI) Field Instrument (FPPAI)
• • • • • •
E
NTRANCE
P
URPOSE
Provides demographic profile of entering students. Completed at time of entrance into the program (Program Defined).
Provides overview of financial resources students are using or plan to utilize. Provides employment status & background information regarding both volunteer and paid human service experience. Helps track planned or unplanned changes in the profile of students in the program. Evaluate impact of policy changes, such as admissions procedures, over time.
o o o o o o o o o o o o o
E
NTRANCE
Q
UESTIONS
Student tracking: ID number & Survey completion date Gender Year in school Overall GPA & GPA in major & Highest possible GPA at school Length of current social work-related work experience (volunteer & paid) Citizenship/ length of residence in USA Employment plans during social work education Hours per week expected to work during education Sources of financial aid expected Language fluency Expected date of graduation Race/ Ethnicity Disabilities/Accommodation
E
XIT
P
URPOSE
• • • • Completed by students just prior to graduation. o Often administered in field seminar or capstone seminar. Feedback from students about their experiences while in the program. Addresses: o o Evaluation of curriculum objectives based on EPAS. Post-graduate plans, related to both employment and graduate education, are addressed.
Collects demographic information to compare with the entrance profile.
• • • • •
E
XIT
Q
UESTIONS
Educational Experience o Including implicit curriculum assessment Current Employment Employment-seeking activities Current & anticipated Social Work Employment o Primary function & major roles Post Graduate Educational Plans
E
XIT
Q
UESTIONS
• • • Students Evaluate how well program prepared them to perform practice behaviors Professional Activities o Use of research techniques to evaluate client progress & Use of program evaluation methodology Personal Demographic Information o Gender, Citizenship, Language fluency, & Disabilities
• • • •
A
LUMNI
/G
RADUATE
P
URPOSE
Intended for completion two years after graduation o
Standardized timing for administration is essential to create a reliable dataset for comparison over time.
Alumni evaluate how well program prepared them for professional practice. Alumni employed in social work and those not employed in social work are surveyed. Also gathers information on current employment, professional development activities, and plans/ accomplishments related to further education.
• • • • • •
A
LUMNI
/G
RADUATE
Q
UESTIONS
Current Employment Current Social Work Employment Evaluation of preparation by the Program in the 10 EPAS competency areas (using Likert-type scale) Educational Activities Professional Activities Demographics
E
MPLOYER
P
URPOSE
• • • • Intended for completion two years after graduation Addresses both accreditation and university concern for feedback from the practice community. Measures graduate’s preparation for practice based on supervisor’s assessment.
Alumni/ae request employer complete the survey: o o Addresses primary concern of confidentiality.
Use of student identified allows connection to other instruments.
E
MPLOYER
Q
UESTIONS
• • Educational background of supervisor/employer Twelve (12) items which evaluate alumni/ae proficiency in all EPAS competencies.
BREAK TIME
Foundation Curriculum Assessment Instrument
C URRICULUM I NSTRUMENT
P
URPOSE
(FCAI)
1. Provides Pre/Post test in seven major curricular areas of the foundation year.
2. Provides a direct measure to assist programs with evaluation of their curriculum.
3. Assists with identification of curricular areas that may need attention.
4. Provides national comparative data.
Curricular Components
Curriculum Area Number of Questions
Practice 13 Human Behavior & Social Environment Policy Research Ethics and values Diversity Social and Economic Justice 8 7 10 9 9 8
Sample HBSE Question
• The concept “person-in-environment” includes which of the following: a. Clients are influenced by their environment b. Clients influence their environment c. Behavior is understood in the context of one’s environment d. All of the above
Sample Practice Question
• Determining progress toward goal achievement is one facet of the _____ stage. o a. Engagement o b. Evaluation o c. Assessment o d. Planning
Overview of FCAI Respondents
Instrument 2011 2012 2013 Total
Entrance Exit 1,986
2,009
964
1,714 8,432 5,369 12,427 8,047
Totals 2,950
3,723 13,801 20,474
Reliability Testing
•
Version 9
–
Tested in two junior practice classes
–
Students tested twice, 2 weeks apart
–
Pearson’s correlation coefficient
•
r = .86
Item difficulty index
• Overall difficulty or average should be around .5 (Cohen & Swerdlik, 2005) • FCAI = .523 (n=415) • “This is a very good difficulty level for the test.
Not likely to misrepresent the knowledge level of test takers”.
Reliability & Effect Size
•
Cronbach’s alpha = .784
•
Effect Size d = 6.87
Current Data 1/13 to 9/13
• • Number of Schools using the FCAI in 2013 110 Number of Respondents in 2013: o Pre-test: 8432 o Post-test: 5369
School X / National Comparison
School X
PRE TEST
31.24 (48.81%)
POST TEST
39.35
(61.48%)
SD
6.19
SIG
.000
National 30.38
(48.10%) 37.54
(58.60%) 7.59
.000
Raw Score
Overall Scores Pre-Post
BSW Entering (N = 8432)
32.8
BSW Exiting (N =5369)
40.05
T-test
380.2
S.D.
7. 8
Sig
.000
Mean % correct 51.2% / 64 62.5%/64
BSW Student Scores by Curricular Area One Program FCAI Entrance & Exit
Curricular Area Question Suggested Competency Correct responses and % Entrance Correct responses and % Exit
Practice Question 1 2.1.1C
14/64 21.8% 39/64 61% Question 2 2.1.10A
Question 4 2.1.10H
46/64 71.8% 21/64 32.8% 60/64 93.7% 43/64 67.2%
Expansion beyond BSW
• Based upon CSWE assertions related to educational levels in social work education, we expanded testing to three additional groups: o MSW foundation students: o • •
entering exiting
Advanced standing students: •
entering
Points to keep in Mind about the FACI
1. Purpose of this instrument: to review and improve curriculum 2. Program will want to “monitor” scores over several years (or several cohorts) for trends.
3. The FCAI can be considered a measure of “value added” from program entry to exit.
4. Benchmarks: can be set two ways, a. by competency b. overall score
Field Practicum/Placement Assessment Instrument (FPPAI)
F IELD P RACTICUM /P LACEMENT A SSESSMENT I NSTRUMENT (FPPAI)
• Responds to need for a standardized field/practicum assessment instrument that measures student achievement of practice behaviors.
F IELD I NSTRUMENT
P ILOTING
(FPPAI)
P HASE • • • • • • • Initial Piloting for BSW in May 2008 Second Pilot in Fall 2008 /Spring 2009 Third Pilot in Fall 2009 Reliability Analysis o Chronbach’s Alpha of 0.91 or higher in each practice behavior Full implementation in BSW: Fall 2010 Piloting in MSW: Spring-Fall 2012 Full implementation in MSW (Foundation): Fall 2013
• • • • • •
F IELD I NSTRUMENT (FPPAI)
M ETHODOLOGY 58 Likert Scale questions measuring practice behaviors linked to the EPAS 2008 competencies.
Qualitative feedback form for each domain available for program use.
Available online and in print format.
Individual program outcomes report with national comparisons available.
Individual program outcomes report with national comparisons for EPAS 2008 Competencies & Practice Behaviors including CSWE benchmark reporting.
Can be used as a final field assessment and mid test/post test design.
F IELD I NSTRUMENT
CAL
(FPPAI)
F IELD I NSTRUMENT
C URRENT
(FPPAI)
S TATUS • 84 programs currently use this instrument • More than 3,132 administrations to date • National data comparisons are available • Cronbach’s Alpha reliability test of internal consistency at midpoint (Pilot and first two years Testing): 0.969
• Cronbach’s Alpha reliability test of internal consistency at final (Pilot and first two years testing): 0.975
BREAK TIME
SWEAP
U NIQUE B ENEFITS • • • • • • Student demographics Numerous data points for comparison Explicit and Implicit curriculum assessment Doesn’t end at graduation Peer comparison by region, program type, auspice & nationally …and more
TO P T YING ROGRAM
L INKING
SWEAP A SSESSMENT
SWEAP TO EPAS
• • • All instruments updated to reflect 2008 EPAS All instruments will be updated and available as soon as 2015 EPAS goes into effect Competency Matrix for 2008 EPAS(Handout)
TO T YING SWEAP P ROGRAM A SSESSMENT
Educational Policy 2.1.1—Identify as a professional social worker and conduct oneself accordingly Practice Behavior use supervision and consultation Measures SWEAP-Exit SWEAP-FPPAI SWEAP-Employer SWEAP-Alumni
T YING SWEAP TO P ROGRAM A SSESSMENT
: M ATRIX
Employer Competency Alumni Exit FCAI FPPAI
2.1.7—Apply knowledge of
human behavior and the
social environment. A. utilize conceptual frameworks to guide the processes of assessment, intervention, and evaluation; B. critique and apply knowledge to understand person and environment.
TO T YING SWEAP P ROGRAM A SSESSMENT
P ROGRAM I NTEGRATION E XAMPLE • Program Integration Example (Handout)
TO
U
SING
T
YING
SWEAP P
ROGRAM
A
SSESSMENT
SWEAP B
EYOND
EPAS
• • • Curricular Development Addressing Implicit Curriculum Concerns Understanding student body o Changes in student body over time
S HARING A SSESSMENT
Break into small groups, tackle the assigned competency discussing the following:
I DEAS
How can/do you measure competencies in your programs?
Offer SWEAP & Non SWEAP options
S HARING A SSESSMENT I DEAS
Break into small groups, tackle the assigned competency discussing the following: How can/do we measure Implicit Curriculum?
Offer SWEAP & Non-SWEAP Measures
N AVIGATING
T
HE
SWEAP
W
EBSITE
http://sweap.utah.edu
o Navigating the site o Locating the results of your instruments
N AVIGATING SWEAP
O
RDERING o Online through website.
o Pay by credit-card, or send check.
o Receive email with .pdfs to be printed, or links to online instruments.
o Printing .pdfs
• Can be double-sided, BUT DO NOT STAPLE For questions about your order or status:
• • • • •
Online Instruments
All SWEAP instruments are available in electronic format.
eTickets (electronic tickets) are simple to use. You order them via the SWEAP order page on our website, just like you would order instruments to print on paper. Each eTicket is an individual instrument and cannot be copied.
The simplest way to administer instruments with eTickets is to send an email to each student, field educator or alumnus/a who is expected to complete the instrument. In that email, you provide an individual eTicket link.
The respondent enters the letters and numbers that follow the equal sign (=) on the eTicket link. For instance, to access the email example, the code to enter in the pink box is RITNFN26197 (see next slide).
• • •
N AVIGATING
R
AW
D
ATA
SWEAP
P
OLICY We offer return of de-identified data to programs with the following data removed: Year of Birth o o o o o Day of Birth Parts of Social Security Numbers ZIP code of Employment or Residence.
Sex / Gender Ethnic Identity o In addition, specific answers to the FCAI instruments will not be returned, nor specific answers to the Values at Exit or Entrance. These will be reported in an aggregate only.
FPPAI instruments, due to their nature, will return all identifying information.
•
N AVIGATING
R
AW
D
ATA
SWEAP
P
OLICY There will be a surcharge to utilize this service, it will be a one time charge of $25.00 to have all permissible data returned to the program.
• •
N AVIGATING SWEAP
P
ROCESSING Paper instruments need to be mailed to Utah.
o o Processing involves scanning forms into system and then calculating data.
Can take up 3-6 weeks, depending on time of year.
Online instruments o Processing is very quick.
o Can be days, instead of weeks.
o No need to be scanned at our end.
• • •
N AVIGATING SWEAP
R
EPORTS Receive email when reports are ready.
Retrieve reports through website.
Can request reports with tailored comparisons.
Coming in 2015!
• • • Bridging the gap for direct measures: New direct measures in-line with 2015 EPAS Outcome Reports on your webpage as required by CSWE All current instruments updated for EPAS 2015