Transcript Evaluating Educational Technology Planning and Implementation

Evaluating Technology’s
Impact on Teaching and
Learning
Educational Technology Integration
Contact Information
[email protected]
978-453-3070
www.sun-associates.com/eval/sample
This presentation is linked to that page
Where Do We Stand?
Hot topics
Your District
State
National
Your expectations for today
Workshop Goals
To overview a scheme for creating a
formative evaluation of technology’s
impact
To review related issues and
developments visa vis our experience in a
particular school district (Fayette County,
KY)
To create an action plan for conducting a
similar assessment in your own district
A Framework for Review
From Designing P rofession al
Development for T eachers of Science
and Mathem atics, Loucks-Horsley,
Hewson , Lo ve, an d St iles. Corwin
Press Inc. 19 98
Why Evaluate?
To realize your investment in technology
What sort of “difference” has all of this
technology made?
To continue your school or district
technology planning effort
Evaluation is a key component of ongoing
technology planning!
Evaluation Starts with
Technology Goals
Evaluation should be rooted in a strong
educational technology plan which...
Is more than an infrastructure plan
Focuses on technology’s impact on teachers
and students
Has clear goals and objectives for what you
want to see happen
Skeletal Plan Format
VISION
Curricular Goals
Goal 1
Acitivity 1
Activity 2
Staff Development Goals
Goal 2
Goal 1
Activity 1
Activity 1
Goal 2
Activity 1
Activity 2
Community Goals
Goal 1
Goal 2
Activity 1
Activity 1
Your Goals?
Using the Plan Mapping Worksheet, map
your plan in terms of vision, goals, and
actions.
Goals Lead to Questions
What do you want to see happen?
These are your goals
Achieving these goals requires a process
Are you performing the process steps?
All of this can be measured through a
formative evaluation
The Evaluation Process
Create meaningful evaluation questions
Design relevant indicators
Organize those indicators in a
performance rubric which can be used to
measure progress and achievement
Collect data to score the rubrics
Create a report which applies the
evaluation in a formative manner
Take a break...
FCPS Example Questions
How have students been impacted by
technology integration?
Are teachers using technology in ways
that match district goals for technology
and use the potential for instructional
technology?
Have we adequately allocated district
technology resources?
Processes of Technology
Evaluation
What are you looking for?
Ways to make the implementation of
technology better
Teaching and learning impacts — this is not
about counting machines!
Developing Indicators
What is it that you want to measure?
Impact on teaching and learning
Change
Progress towards desired outcomes
You need indicators for progress to be
measured
Indicators should reflect your school’s
unique goals and aspirations
Arise from your plan
Rooted in your vision and goals
Indicators must be indicative of your unique
environment...what constitutes success for
you might not for someone else.
Indicators of Engaged Learning
Engaged learners are
Responsible for their own learning
Strategic problem-solvers
Energized by learning
Collaborative learners
These are all observable characteristics
These indicators can help inform your
student impact rubric
Levels of Teacher Appropriation
Stage
Entry
Adoption
Adaptation
Appropriation
In vention
Teacher Behavior
Teachers learn the "basics" of using new
technology.
Teachers use new technology to support
traditional instructional methods.
Teachers integrate new technology into
traditional classroom practice, f ocusing on
increased student productivity and
engagement through the use of tools such as
word processors, spreadsheets, and graphics
tools.
Teachers f ocus on cooperative, project-based,
and interdisciplinary work which incorporates
technology as needed and as one of many
tools.
Teachers discover new uses for technology
tools of ten by designing projects that combine
multiple technologies.
Adapted from "A Report on 10 Years of ACOT Research" (Apple Computer, 1996)
Point is... There are things we can look for
in terms of student and teacher behavior.
The goal in creating indicators is to link
aspects of behavior to levels of
achievement
What’s Realistic?
When developing questions and
indicators, what makes sense visa vis the
research on technology’s potential
impacts?
www.sun-associates.com/eval/resources.html
What the Research Says
CBI generates measurable increases in
mastery of core content
Kulik’s meta-analysis
Fletcher’s evaluation of military training
Pace, repetition, and standardization
promote memorization of “facts”
Kulik’s work shows that CBI did virtually
nothing in terms of changing student
“attitude” toward subject matter
Productivity tools -- particularly tools for
writing -- encourage students to produce
more and pay more attention to the quality
of their work.
This more free-form use of technology cuts
directly to the heart of the connection
between attitude toward learning and student
achievement
Students will do more of what they like to do
Technology improves the learning
environment
Motivates students
Makes learning more exciting
Makes learning more relevant to real-world
tasks
None of these benefits come solely
through technology!
It gets more gray when considering
“cognitive uses”
Cooperative and collaborative tasks
Communication
Problem solving
Using technology as a key element of a rich,
supportive, educational environment
Student Performance in Rich
Educational Environments
Higher rates of attendance and graduation
(ETS report)
Higher qualification rates for honors
programs
Greater development of critical thinking and
problem-solving skills
Generally higher scores on traditional
assessments
Lower numbers of disciplinary incidents
These benefits have been
observed/documented specifically in those
classrooms where technology has been a
key intervention...but never the only
intervention!
What Other Interventions?
More teacher training
Usually in relation to using the technology
tools as part of an overall reformed
instructional environment
More individualized, student-centered,
learning
Usually as a result of problem-based,
cooperative learning environments where
technology is the catalyst for change
So, What Changes?
The student outcomes from participation in a
technology-enriched environment are indeed
positive
Measures are usually in terms of new things
that can be done versus increases on
traditional assessments
Nevertheless, traditional assessments do
show improvement...it just occurs over time
and is part of a broader, system-wide, picture
of change.
Try a Sample Indicator
Using the Developing Indicators
worksheet, let’s develop a few sample
indicators
Work in several groups
Select someone to report out
Try to develop some consensus on indicators
which would work across districts
FCPS’s Indicators
Use of technology positively impacts and
fosters the student’s motivation to engage
in learning practices that lead to new ways
of thinking, understanding, constructing
knowledge, communicating results, and
acquiring basic skills
The faculty and staff are proficient,
knowledgeable, and current with available
technology and translate that knowledge into
relevant learning opportunities for students
Teachers create learning opportunities and
physical environments that allow students to
assume more independent roles in their own
learning through their use of technology.
All teachers and learners throughout the
district have sufficient access to
technology-based productivity tools, online services, media-based instructional
materials, primary sources of data, and
adequate support for using these
resources so as to enrich and extend their
learning goals.
Rubrics
Que s tion
Bas ic
Indicator
Le ve l 4
Stude nt Achie ve m ent
How has technology positively impacted student achievement?
Use of technology positively impacts and fosters the studentÕs motivation to engage in learning practices that
lead to new w ays of thinking, understanding, constructing know ledge, communicating results, and acquiring
basic skills
Technology is used routinely and seamlessly as a tool f or learning. Technology positively impacts and f osters
the studentÕs motivation to engage in learning practices that lead to new w ays of thinking, understanding,
constructing know ledge, communicating results, and acquiring basic skills. Technology is used to inspire
critical thinking and the solving of problems relevant to real-life skills w ith recognition of the tradeof fs inherent in
the application of technology in society. Student w ork parallels the w ay in w hich professionals in the w ork force
use technology.
Students exercise a high degree of personal judgement in the choice and application of technology to their
learning. Teachers support this choice by designing and f acilitating a student-centered learning environment
w hich makes use of a w ide variety to technology tools.
Level 4 Technology is used in all of the w ays documented in the previous levels.
Evidence
In addition, the choice of technology tools used in learning is primarily student directed. Students make
appropriate choices w ith regard to w hen and w here to use technology. For exampleÉ
1. The student exercises personal judgement in the maximum and most appropriate use of technology to
assist problem solving, reasoning, and thinking.
2. Students demonstrate critical thinking and media literacy skills in the use of technology as a research tool.
www.fayette.k12.ky.us/central/edtech/suneval.asp
Le ve l 3
Students regularly use technology w ithin their learning activities. Fluent use of technology positively
impacts and f osters the studentÕs motivation to engage in learning practices that lead to new w ays
of thinking, understanding, constructing know ledge, communicating results, and acquiring basic
skills.
Use of technology is equally student directed as it is teacher directed. Students exercises om e
degree of personal choice and judgement in their use and application of technology to learning.
Level 3 As f or Level 2, but considerably more advanced uses of technology are demonstrated. For
Evidence example...
1. Students use technology tools to solve problems w hich require the organization and analysis of
data (i.e., a graphing calculator, spreadsheet, database, etc.)
2. Students use softw are to create presentations to communicate ef fectively (i.e. Pow erPoint slide
show s, etc.)
3. Students use email to contact experts and communicate w ith peers about a specific area of
interest in order to expand know ledge.
Le ve l 2
Students use s om etechnology nearly every day in relation to learning activities. The primary
emphasis of technology use is still in the acquisition of basic skills, but there is evidence that the
student is beginning to use technology to engage in learning practices that lead to new w ays of
thinking, understanding, constructing know ledge and communicating results.
The vast majority of student technology use is teacher-directed, w ith students assigned to use
particular technology tools, applications, or resources as part of their academic w ork.
Level 2 As w ith Level 1, except technology use becomes more routine and regular. More advanced uses
Evidence begin. For exampleÉ
1. Students use a w ord processor or appropriate sof tw are to enhance the organization of
products, and to improve quality (i.e., cut-paste, spell check, grammar checking, etc)
2. Students use technology to solve problems (i.e., a graphing calculator, spreadsheet, database,
etc.)
3. Students use technology to improve problem solving, reasoning, and thinking.
4. Students use application programs to discover concepts and relationships, especially in
science, math, and social studies.
5. Students use the Internet (dow nload and copy, know usef ul w eb sites, and use search
engines) and appropriate technologies, such as CD-ROM encyclopedias, as research tools.
Le ve l 1
The studentÕsinitial us eof technology supports the acquisition of basic skills and increased
productivity.
All uses of technology are teacher-directed. Teachers w eave the use of some technology tools
and devices into student lessons and activities.
Level 1 1. Students use calculators, games, tutorial, and application programs to practice and acquire
Evidence
basic skills and to increase the depth of understanding of particular subject area curriculum.
2. Students use w ord processors to w rite (i.e., cut-paste, spell check, grammar checking, etc) and
produce products.
3. Students demonstrate the know ledge of basic application/productivity softw are such as
spreadsheets, databases, and presentation tools.
4. Students begin to explore the Internet and become acquainted w ith email.
Rubric Tips
Even number of levels
Create from ideal to lesser
Yes, Yes But, No But, No
Be as descriptive as possible
Do not get excessively quantitative
Take a break...
FCPS Rubric Scores
4
Student Impact
3.5
Teacher Fluency
3
2.5
2
1.5
1
0.5
0
Indicators
Teacher Use with
Students
Teacher
Productivity
Administrator
Support
Infrastructure
Allocation
Student Impact, 2
Technology is being used daily as a basic
tool within a range of traditional learning
activities
Some evidence that technology is
supporting new ways of thinking and
learning
Most student technology use is very
teacher-directed and occurs in settings
outside of the classroom
Teacher Fluency
Teacher Fluency = 2.5
Teachers are reasonably fluent, but need to
do more to connect technology to the core
curriculum
Use of Technology with Students = 2.25
Teachers direct students to use technology,
but need to use technology to empower
students as independent, engaged, learners
Teacher Productivity = 4
Teachers are very proficient in using
technology as a tool for personal and
professional productivity
Administrator Support = 2
Administrators use technology, but need to
advocate and support more integrated uses
among their teaching staff
Infrastructure Allocation, 2.75
The basic infrastructure -- hardware, network,
software -- is complete and adequate
Individuals in Technology Support functions
(schools and district) are doing an exemplary
job and should receive much of the credit for the
advances we have seen thus far
Schools need even more technology resource
teacher support
Efforts must continue to map technology on to
the core curriculum
Sum It All Up?
Fayette County is doing very well!
Based on our research nationwide, FCPS is
clearly above average in most areas of its
technology implementation
Teachers are using technology in ways
that are consistent with their overall
instructional approach
Directed instruction
Technology as a “skill” versus an integrated
element of the learning process
Technology can do more, and teachers
can do more
Professional development and increased
levels of resource teacher support will make
this happen
The full report contains a number of
recommendations for professional
development and policy designed to positively
impact performance on the indicators
To Summarize...
Start with your plan
From your goals, develop indicators and a
performance rubric
Accumulate data and/or develop your data
collection tools
How will you collect the data necessary to
document your stage of fulfillment?
Data Collection
Review Existing Data
Current technology plan
Curriculum
District/school improvement plans
www.sun-associates.com/eval/sample
Create a checklist for data collection
Collect New Data
Focus Groups/Interviews
Teachers
Parents
Students
Administrators
Other stakeholders
Classroom Observations
Surveys
Creating good surveys — what do you want to
know
•
•
•
•
•
length
differentiation (teachers, staff, parents, community, etc..)
quantitative data
attitudinal data
timing/response rates (getting returns!)
www.sun-associates.com/eval/samples/samplesurv.html
Solicitation of teacher/parent/student
stories
This is a way to gather truly qualitative data
What does the community say about the use
and impact of technology?
Compile the report
Determine how to share the report
School committee presentation
Press releases
Community meetings
Conclusion
Build evaluation into your technology
planning effort
Remember, not all evaluation is
quantitative
You cannot evaluate what you are not
looking for, so it’s important to —
Develop expectations of what constitutes
good technology integration
Technology and Traditional
Assessment Methods
Does technology use result in higher test
scores?
Probably, but there is not a direct link
Technology alone is not helpful in
improving student performance
A high degree of technology integration is
more than likely to be an indicator itself of
high performing schools and students
Is There a Connection?
High scores on technology integration
rubrics generally equate to successful
teaching and learning practices -regardless of whether these practices
have any specific relation to technology
e.g., Constructivism, Engaged Learning, and
Student-Centered Classrooms
Technology supports these practices and
these practices are shown to improve
student learning
Schools where these practices are in
place and technology is used
appropriately are high performing schools
It’s not the technology, it’s the teaching!
We measure how well the technology is
used to support successful teaching
practices.
Our formative evaluations highlight how to
improve teaching, technology integration,
and ultimately student success
Next Steps?
Awareness building
Establish the need for evaluation
Establish a process
Division leadership
As with technology implementation in
general, this is very important
The evaluation will only be as good as it is
taken seriously
Evaluation committee membership
Stakeholders!
Are tech plans in shape?
Ideally, the evaluations will be based in
sound visions and goals and will take into
account existing data on completed actions
Do you divisions have this capacity?
Evaluation work can be used to
rejuvenate the local revision process, but
this will take time and organization
Training
It takes about a day to train people in this
approach/process
Regional workshops?
District teams?
Standardizing and cross-division review
People should share and review their
questions, indicators, and rubrics before
setting out to conduct an evaluation
Think about data collection
Integrating a meaningful technology impact
survey into existing survey efforts
Integrating observation data into existing
teacher/class observations
More Information
[email protected]
978-453-3070
www.sun-associates.com/eval/sample
This presentation is linked to that page