Transcript Performance Analysis T/TAC Online Project

Design Treatment Presentation
Fall 2005 T/TAC Online Project
VA Assessments
December 15, 2005
Matt Humphrey
Aala’a Mashaal
Pamela Sharpe
Tim Smith
Maria Washington
Graduate School of Education
George Mason University
1
Presentation Overview
•
•
•
•
•
Part 1
Immersion Overview
Instructional Design Process
Part 2
Background & Introduction
Mission and Goal of Project
Part 3
Analyses
•
•
•
•
•
Needs Analysis
Task Analysis
Findings and Results
Design Approach
Next Steps
Part 4
2
Immersion
 Overview
 Constructivist Learning Environment
 Action Learning
 The Learning Process
 Teamwork
 Portfolios and Reflections
 Challenges
 Benefits
3
Instructional Design Process
ADDIE MODEL
Analysis
Design
Develop
Implement
Evaluate
4
Integrated Learning Design Framework
Exploration
Enactment
Online Learning Developer
Evaluation
5
Instructional Design Process
Performance Centered Design Process
Process
Persona/
User
Performance Zone
Content/
Knowledge
6
Background & Introduction



No Child Left Behind Act (NCLB)
Individuals with Disabilities
Education Improvement Act of 2004
(IDEA)
VA Assessments section for T/TAC
Online
7
T/TAC Online
 T/TAC Online is a web-based
community linking people and resources
to help children and youth with
disabilities.
 http://ttaconline.com
8
Problem
 Transition to Cognitive Standards Based
Assessments
 Communication Skills
 State Mandated Assessments
 New Process for Teachers
9
Analysis
Design
Develop
Implement
Evaluate
Analysis
Needs
Analysis
Performance Analysis
Learner Analysis
Task Analysis
10
Needs Analysis
Overview
The process of determining, analyzing,
and prioritizing needs and, in turn,
identifying and implementing solution
strategies to resolve high-priority
needs.
11
Needs Analysis
Objectives
The objectives of the needs
analysis are to:
 Determine the best solution
 Generate data to support
recommendations
 Provide specific recommendations
 Decide priorities for action
12
Needs Analysis
Six Steps
The Six Steps involved in our
Needs Analysis:
1. Determine Current Status
 Performance Analysis
2.
3.
4.
5.
6.
Analysis Results
Teacher Expectations
Identify Discrepancies
Determine the Goals
Set Priorities for Action
13
Performance Analysis
Overview
 Is used to determine what needs to
be done in order to accomplish the
client’s stated goals and objectives.
 Focuses on determining the elements
that support the project
 Investigates the current situation
(actuals) and desired performance
(optimals) (Rossett, 1999).
14
Performance Analysis
Clients & Information Sources
Clients
Information Sources
•Dr. Patricia Abrams
•Dr. Michael Behrmann
•John Eisenberg
•Nathan M. Sparks
Human:
•All clients
•Jocelyn DelSignore
•Anya Evmenova
•Teachers
Content:
•T/TAC & VA DOE websites
•VAAP Manual
•Mary Wilds
15
Performance Analysis
Current Situation/Actuals
 Special Education teachers have not been doing assessments for
their students in a standardized format
 Special Education teachers collaboration outside the classroom is
limited
 The assessment documents are difficult to locate on the Virginia
Department of Education website
 The challenge of assessing special education students creates a
barrier
 Special education teachers of students with severe disabilities have
not been required to conduct SOL based assessments
 Teachers spend an inordinate amount of time developing portfolios
which were not aligned with SOL expectations
16
Performance Analysis
Desired Performance/Optimals
 Special Education Teachers should assess their
students using the different assessment standards
including the VAAP, VGLA, and the VSEP.
 The documents should be in one location as a “one
stop shop” for teachers
 The documents should be searchable for easy access
and easy downloading
 The creativity of the special education teachers should
be shared with other teachers by creating and
submitting lesson plans
 Teachers can submit standardized assessment
portfolios with streamlined evidences
17
Needs Analysis
Data Collection
 Review the manuals (VAAP, VGLA, VSEP)
 Attend conferences to observe teachers
reactions
 Administer Surveys
 Visit schools to interact with Special
Education Teachers
 Informal talk with Jocelyn
 Interview with Dr. Behrmann
 Informal talk with Anya
 Interview with Mary Wilds
 Interview with John Eisenberg
18
Needs Analysis
Survey Questions
T/TAC ONLINE | VA ASSESSMENTS PROJECT SURVEY
Dear teacher:
We are conducting research in order to help teachers transition to
using the new VAAP manual in the most effective and efficient way
possible. Please assist by completing this survey. Thank you.
1.
2.
3.
4.
5.
Name:
School name:
Age (Circle One): 20-30, 31-40, 41-50, 51+
Years of Experience in Special Education field:
Type of License:
On a scale of 1-5 (5 being the strongest) rate the following:
6.
I am proficient using the internet.
7.
I have sufficient access to a computer (in the class/at home).
8.
I use the internet to assist me in planning lessons and/or
activities.
Surveys conducted 10/25/05 and 11/3/05, 89 SPED teachers
19
Needs Analysis
Survey Questions
Regarding the new VAAP (Virginia Alternate Assessment
Program)
7. I agree with the need to assess cognitive as well as life skills for
VAAP students.
8. I currently have a good method to assess my VAAP students.
9. I spend a lot of time assessing my VAAP students.
10. I use/have many (5 or more) tools for assessments.
11. My lesson plans are shaped by the current VAAP assessments.
Assessments
12. There are many hindrances (time, resources, lack of appropriate
information, etc) that affect my assessment process.
13. I think the new assessments will change the way I prepare lesson
plans.
14. I would be interested in interacting with other special education
teachers online who are involved in the same VAAP transition.
15. I would like to share lesson plans and activities.
16. I would like to get help aligning the cognitive skills and life skills into
the curriculum.
Surveys conducted 10/25/05 and 11/3/05, 89 SPED teachers
20
Needs Analysis
Findings/Results – New VAAP
New VAAP
Figure 1
Lesson Plans to VAAP
Tools for access
Time Assessing
Method of Assessment
Agree w /assessment
0.0
Agree w /assessment
0.5
1.0
Method of Assessment
1.5
2.0
Time Assessing
2.5
3.0
Tools for access
3.5
4.0
4.5
Lesson Plans to VAAP
•85.6% of the teachers indicated that they agree with the assessments.
•67% of teachers surveyed indicated that they spent a lot of time assessing the
students.
•61% of teachers surveyed agreed that they have five or more tools for assessments
Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers
21
Needs Analysis
Findings/Results - Assessment
Figure 2
Assessments
Hindrances
Prepare Lessons
Interacting-Transition
Share Lesson Plans
Help Aligning
Know ledge of T/TAC
Use T/TAC
5.0
4.0
3.0
2.0
1.0
0.0
Hindrances
Prepare
Lessons
InteractingTransition
74%
72%
77%
Share Help Aligning Know ledge Use T/TAC
Lesson
of T/TAC
Plans
75%
79%
62%
50%
Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers
22
Needs Analysis
Findings/Result - Age Ranges
AGE
50+
40-50
30-40
20-30
16
16.5
17
20-30
17.5
30-40
40-50
18
18.5
19
50+
Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers
23
Needs Analysis
Findings/Results - Years of
Teaching Experience
Years Teaching
1 - 5
4%
21 +
38%
6 - 10
12%
11 - 15
19%
16 - 20
27%
1 - 5
6 - 10
11 - 15
16 - 20
21 +
Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers
24
Needs Analysis
Findings/Results - Licensure
Professional
License
Provisional
Provisional
15%
Professional
85%
Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers
25
Needs Analysis
Teacher Performance Expectations
1. For students designated to be assessed under the
VAAP teachers must begin aligning lesson plans to new
aligned standards of learning (ASOL) guidelines, which
include cognitive skills.
2. Skills will be measured in four categories: Reading,
Math, History & Social Science and Science.
3. Teachers must develop lesson plans that align to SOL
guidelines and can be rated 3 or above on the
established scoring rubric by locally identified scorers
in the areas of 1) student performance, 2)
communication context, 3) communication settings, 4)
communication socialization, and 5) communication
support.
26
Needs Analysis
Teacher Performance Expectations
Teachers must be able to demonstrate the following skills and knowledge:

Have knowledge of ASOL guidelines and how to incorporate communication
skills

Have access to VAAP Manual

Know how to complete an Individual Education Plan (IEP)

Have access to students IEP (work with IEP team)

Align lesson plans to SOL guidelines for SPED students

Develop lesson plans based on cognitive skills

Align lessons to SOL enhanced

Use computer:







Download blank forms
Internet access
Be able to use word-processing software
Be familiar with state and district-wide assessment
Have knowledge of learner’s skill level, needs, and characteristics
Have knowledge/access to proven- research based instruction techniques
Be able to develop lesson plans that fit the student’s current stage of
learning for a particular skill.
27
Needs Analysis
Discrepancies
 Teachers do not have experience in aligning
their lesson plans to the new standards.
 Some of the teachers do not have
experience assessing using the former
VAAP standards.
 The greatest discrepancy lies in the issue
that this process is completely new to all
teachers in the state of Virginia.
 Teachers have limited resources to help
them with the assessment process.
 Teachers have limited time outside the
classroom to dedicate to this process
28
Needs Analysis
Goals
 1. Provide scaffolding for teachers to
develop lesson plans aligned with the
SOLs.
 2. Produce a searchable VAAP manual
 3. Provide additional resources for
teachers to utilize throughout the
assessment process
29
Needs Analysis
Priorities for Action
 Analyze lesson plan process
 Based on the needs, determine
potential solution
 Create prototype of the solution
system
30
Task Analysis
Overview
 A task analysis is a process of
analyzing and articulating the kind of
learning that you expect the learners to
know how to perform (Jonassen,
Tessmer, & Hannum, 1999).
 A thorough task analysis is essentially
a blueprint of the instructional sequence
and it assures the instructional designer
that there are no gaps in the
instruction.
31
Task Analysis
Information Processing Analysis
 The task analysis method selected was
information-processing analysis (IPA).
This type of analysis is used most often for
procedural and cognitive tasks.
 According to Smith and Ragan (1999),
conducting an information-processing analysis
is the first step in “decomposing” or breaking
down a goal into its constituent parts,
identifying what the students need to learn to
attain the goal.
32
Task Analysis
Process
 A task analysis was conducted on the lesson plan to
compile all components in an organized manner. The
following task analysis is based on extant data research,
observed teacher training of VAAP instruction process, as
well as an expert interview with T/TAC Online Asssistant
Director at GMU.
 When a teacher completes a lesson plan that aligns with
the task analysis all necessary components will be
included to target the ASOL, CS, and IEP goals.
 The completed task analysis will support the design and
development of the solution system.
33
Task Analysis
Challenges
 Is this the only lesson plan process?
 Problem types for each task
 Problem solving, rule using, etc
 Appropriate supports for each task
 What does the final result of each
task look like?
34
Task Analysis
Lesson Plan Document
35
Task Analysis
Lesson Plan Process
Prerequisite Tasks
1. Review present level of performance,
assessment data, and IEP team
recommendations
2. Review student profile document
3. Identify communication skills
4. Identify student’s entry points
5. Plot entry-level points and communication
skills on the student and classroom matrix
36
Task Analysis
Lesson Plan Process
1.
Identify a main theme or event of the lesson plan
 Rule Using
2. Record the date of creation of the lesson plan
 Rule Using
3. Identify the student tasks and activities
 Main tasks, sub tasks, prerequisites, instructional
strategies
 Rule Using
4. Choose an ASOL subject matter
 Organizing topic, standard used, support in activity
context
 Problem Solving
37
Task Analysis
Lesson Plan Process
5. Choose a communication skill that can be incorporated
into the lesson
 Can be used multiple times
 Problem Solving
6. Identify the materials needed
 Rule Using
7. Identify and list assistive technology needs
 Ensure that all students have appropriate assistive
technology throughout the lesson
 Rule Using
8. Identify evidence collections materials
 Problem Solving
38
Task Analysis
Further Analysis
Field test the lesson plan process
 Check for completeness
Task analyze observed processes to
determine adequate supports for each
task
Task Analysis used to support design
and development of the solution
system
39
Analysis  Design
Develop
Analysis
Implement
Evaluate
Design
Needs Analysis
Flowcharts
Performance Analysis
Schemas
Learner Analysis
Prototype
Task Analysis
40
a. Flowchart 1: Searching the VAAP Manual
41
Database Schema 1: Searching the VAAP Manual
42
Flowchart 2: Lesson Plan Document
43
Design Approach
Users
Using the data collected from the surveys and
interviews with SME’s, teachers and
administrators we found that our users were:
 Novice Special Education Teacher
 Intermediate Special Education
Teacher
 Expert Special Education
Teacher
44
Design Approach
EPSS
 EPSS was selected as the solution system
for the users
 Rationale:
 Teacher are familiar with the lesson plan process
but not with aligning lesson plans with standards
(supporting an existing process)
 Teachers are very dispersed, a web based
solution can reach more teachers.
 Teachers time is limited, an EPSS provides just
in time support for their needs.
45
Design Approach
Characteristics of an EPSS
Key Characteristics of EPSS
Deborah Alpert Sleight (1993) states that an EPSS should
contain these key characteristics:
 Computer-based
 Provide access to the discrete, specific information
needed to perform a task at the time the task is to be
performed
 Used on the job, or in simulations or other practice of
the job
 Controlled by the user
 Reduce the need for prior training in order to
accomplish the task
46
Design Approach
Attributes of an EPSS
Attributes
In her classic article “Attributes and Behaviors of Performance
Centered Systems”, Gloria Gery delineated numerous elements of
effective performance support systems including:










Establishes and maintain a work context
Aids in establishing the goal
Structures the work process
Reflects the natural flow of work
Provides alternative views data, information, and knowledge
Provides contextual feedback
Provides support resources without breaking the task context
Embeds knowledge into the interface
Provides access to underlying logic
Automates tasks
47
Design Approach
Prototype
http://immersion.gmu.edu/ttac/fall2005/epss
48
Next Steps
Analysis
Implement
Design
Evaluate
Develop
Prototype
Formative
Evaluation
49
Next Steps
 Identify Supports to be integrated
into the EPSS
 Continue Development of Prototype
 Implement Prototype
 Usability Testing
 Formative Evaluation
50
Reference
 Gery, G. (1991). Electronic Performance
Support Systems. Tolland, MA: Gery
Associates.
 Gery, G. (1995). Attributes and Behaviors
of Performance-Centered Systems.
Performance Improvement Quarterly, 8(1),
47-93.
 Smith, Patricia L. & Ragan, Tillman J.
(2005). Instructional Design. Hoboken, NJ:
John Wiley & Sons, Inc.
51
Thank You for Listening
• Any Questions?
52