Design Treatment Presentation Fall 2005 T/TAC Online Project VA Assessments December 15, 2005

Matt Humphrey Aala’a Mashaal Pamela Sharpe Tim Smith Maria Washington

Graduate School of Education George Mason University

1

Our Faculty and Facilitators

      Dr. Nada Dabbagh Dr. Kevin Clark Dr. Mike Behrmann Gary Dickelman Jocelyn DelSignore Anya Evmenova 2

Clients & Information Sources

Clients • Dr. Patricia Abrams • Dr. Michael Behrmann • John Eisenberg • Nathan M. Sparks • Mary Wilds Information Sources Human: • All clients • Jocelyn DelSignore • Anya Evmenova • Teachers Content: • T/TAC & VA DOE websites • Assessment Manuals 3

Presentation Overview

Part One

• •

Immersion Team Process Instructional Design Process Part Two

• •

Background & Introduction Mission and Goal of Project Part Three

•

Part Four

• • • • •

Analyses Needs Analysis Performance Analysis Learners Analysis Task Analysis Findings and Results Part Five

• •

Design Approach Development

• •

Next Steps Questions and Answers

4

Immersion Team Process

   The Learning Process  Teamwork  Portfolios and Reflections Challenges Benefits 5

The Instructional Design Process

   Who  Learner What  Learner needs  Instructional Strategies How  Evaluation 6

Instructional Design Process

Integrated Learning Design Framework

Exploration Enactment

Online Learning Developer

Evaluation

7

Instructional Design Process

The ADDIE Model

Analysis Design Develop Implement Evaluate

8

Project Background

      No Child Left Behind Act (NCLB) Individuals with Disabilities Education Improvement Act of 2004 (IDEA) Federal Mandates and VA Assessments Standard of Learning (SOL) Assessment Alternative SOL (VGLA and VSEP) Alternate SOL (VAAP) 9

Training/Technical Assistance Centers

(T/TAC)  T/TAC Online is a web-based community linking people and resources to help children and youth with disabilities http://ttaconline.org

10

Project Problem Statement

VA Assessments     Transition to Cognitive Standards Based Assessments Communication Skills State Mandated Assessments New Process for Teachers 11

Analysis

Analysis

Needs Analysis Performance Analysis Learner Analysis Task Analysis Design Develop Implement Evaluate

12

Needs Analysis

Overview

The process of determining, analyzing, and prioritizing needs and, in turn, identifying and implementing solution strategies to resolve high-priority needs.

13

Needs Analysis

Purpose

    Determine the best solution for T/TAC Online  Actuals and Optimals Generate data to support in making a recommendation Provide specific recommendations Decide priorities for action 14

Needs Analysis

Steps Taken

      Determine Current Status  Performance Analysis Analysis of Results Teacher Expectations Identify Discrepancies Determine the Goals Set Priorities for Action 15

Performance Analysis

Overview

   Used to determine what needs to be done in order to accomplish the client’s stated goals and objectives Focuses on determining the elements that support the project Investigates the current situation (actuals) and desired performance (optimals) 16

Performance Analysis

Results Initial Actuals  Teachers were not using standardized procedures for assessments   Teachers did not have sufficient access to resources on the VA DOE website Teachers were not targeting the Aligned Standards of Learning 17

Performance Analysis

Results Initial Optimals  Teachers will use standardized procedures for assessment   Teachers will have easy access to resources Teachers will target the ASOLs for assessment 18

Learner Analysis

A learner analysis is performed to determine who the users will be, what skills and knowledge are needed by the learners, the environment in which they work, and their motivations and goals for performing. 19

Learner Analysis

Why Create Personas?

   Personas identify the user motivations, expectations and goals responsible for driving behavior. Although personas are fictitious, they are based on knowledge of real users. Some form of user research is conducted before they are written to ensure they represent end users rather than the opinion of the person writing the personas. There are many varying personas that represent our users. 20

Learner Analysis

How We Developed Our Personas

Using the data collected from the surveys and interviews with SMEs, teachers, and administrators we found that our users were:      

Novice Special Education Teacher Intermediate Special Education Teacher Expert Special Education Teacher Reluctant Special Education Teacher T/TAC Online Administrator Parent/Guardian

21

    

Needs Analysis

Data Collection

Administer surveys Visit schools to interact with special education teachers  Interview: Dr. Behrmann     Reviewed the VAAP Manual Attend conferences to observe teacher’s reactions Mary Wilds John Eisenberg Jocelyn DelSignore Anya Evmenova 22

Needs Analysis

Survey

 Gathered information in the following topic areas:  Demographics  Technology skills   Attitudes toward the VAAP Attitudes toward new assessments 23

Needs Analysis

Findings – New VAAP

New VAAP Figure 1

Lesson Plans to VAAP Tools for access Time Assessing Method of Assessment Agree w /assessment 0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Agree w /assessment Method of Assessment Time Assessing Tools for access Lesson Plans to VAAP •85.6% of the teachers indicated that they agree with the assessments. •67% of teachers surveyed indicated that they spent a lot of time assessing the

students.

•61% of teachers surveyed agreed that they have five or more tools for assessments

Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

24

Needs Analysis

Findings - Assessment

Figure 2 Assessments

Hindrances Prepare Lessons Interacting-Transition Share Lesson Plans Help Aligning Know ledge of T/TAC Use T/TAC 5.0

4.0

3.0

2.0

1.0

0.0

Hindrances

74%

Prepare Lessons

72%

Interacting Transition

77%

Share Lesson Plans

75%

Help Aligning Know ledge of T/TAC Use T/TAC

79% 62% 50% Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

25

Needs Analysis

Findings - Age Ranges

50+ 40-50 30-40 20-30 16 16.5

17 17.5

18 20-30

Number of Teachers

30-40 40-50 50+ 18.5

19

Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

26

Needs Analysis

Findings - Years of Teaching Experience

21 + 38% Years Teaching 1 - 5 4% 6 - 10 12% 11 - 15 19% 16 - 20 27%

1 - 5 6 - 10 11 - 15 16 - 20 21 +

Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

27

Needs Analysis

Findings - Licensure

Professional Provisional

License Provisional 15% Professional 85% Surveys conducted 10/25/05, Fairfax County and 11/3/05, Fredericksburg, VA 89 SPED teachers

28

Needs Analysis

Current Situation/Actuals

 Special Education teachers have not been doing assessments for  their students in a standardized format Special Education teachers’ collaboration outside the classroom is limited  The assessment documents are difficult to locate on the Virginia Department of Education website  The challenge of assessing special education students creates a barrier  Special education teachers of students with severe disabilities have not been required to conduct SOL based assessments  Teachers spend an inordinate amount of time developing portfolios which are not aligned with SOL expectations 29

Needs Analysis

Desired Performance/Optimals

 For students designated to be assessed under the VAAP, teachers must begin aligning lesson plans to new aligned standards of learning (ASOL) guidelines, which include cognitive skills.  Skills will be measured in the four subject areas: Reading, Math, History & Social Science and Science.  Teachers must develop lesson plans that align to SOL guidelines and will receive a rating of three 3 or above on a scale of 1-4.  Differentiate the SOL enhanced lesson plans that are currently available on the SOL Enhanced area of T/TAC Online for their students assessed under VAAP guidelines. 30

Needs Analysis

Desired Performance/Optimals

 It is important to note that if a student does not perform well in the four subject areas they may still garner a passing score by performing well in the following communication areas:      Student performance Context Settings Socialization Support 31

Needs Analysis

Discrepancies in Teacher Performance      Teachers do not have experience in aligning their lesson plans to the new standards.

Some of the teachers do not have experience assessing using the former VAAP standards. The greatest discrepancy lies in the issue that the cognitive skill assessment process is new to all teachers in Virginia.

Teachers have limited resources to help them with the assessment process.

Teachers have limited time outside the classroom to dedicate to this process.

32

Needs Analysis

Desired Performance/Optimals

     Use computer:    Download blank forms Internet access Be able to use word-processing software Be familiar with state and district-wide assessment Have knowledge of learner’s skill level, needs, and characteristics Have knowledge/access to proven research based instruction techniques Be able to develop lesson plans that fit the student’s current stage of learning for a particular skill.

33

Needs Analysis

Desired Performance/Optimals

  The solution system should:  Assist teachers with assessing their students using the different assessment procedures outlined in the VAAP, VGLA, and VSEP manuals    Allow teachers to access the documents in one location or a “one stop shop” Facilitate the sharing of lesson plans with other teachers also involved in the process of creating lesson plans Facilitate the submission of standardized assessment portfolios with streamlined evidences thereby reducing the amount of time teachers spend compiling evidence of student performance The documents should be searchable for easy access and downloading 34

Needs Analysis

Priorities for Action

   Analyze lesson plan process Determine potential solution based on the needs Create prototype of the solution system 35

Task Analysis

Overview

  The process of analyzing and articulating the kind of learning that you expect the learners to know how to perform A blueprint of the instructional sequence and it assures the instructional designer that there are no gaps in the instruction 36

Task Analysis

Lesson Plan Process   Instruction is focused on the process of creating lesson plans  Support the teachers in aligning them to the new standards Task analyze the lesson plan process  Understand the process used by a special education teacher  Support those processes in the solution system 37

Task Analysis

Information Processing Analysis (IPA)



Used most often for procedural and cognitive tasks

 First step in decomposing or breaking down a goal into its constituent parts  Identifies what instruction is needed to attain the goal 38

Task Analysis

IPA/Lesson Plan Process    Creating a lesson plan is a procedural task Break down each step of the lesson plan creation process into its constituent parts Identify what we need to provide to the teachers to support them throughout the process  Supports will be utilized in the solution system 39

Task Analysis

What the Teacher Needs To Do

Complete prerequisite tasks 1.

Review present level of performance, assessment data, and IEP team recommendations 2.

3.

4.

Review student profile document Identify communication skills Identify the student’s entry points 5.

Plot entry level points and communication skills 40

Task Analysis

What the Teacher Needs To Do

Complete the lesson plan process 1.

Identify a main theme or event of the lesson plan 2.

Record the date of creation 3.

Choose an ASOL subject matter  Organizing topic, standard used, support in activity context 4.

 Problem-solving cognitive task Choose a communication skill that can be incorporated into the lesson  Can be used multiple times throughout the lesson  Problem-solving cognitive task 41

Task Analysis

What the Teacher Needs To Do

5.

6.

7.

8.

 Identify the student tasks and activities Main tasks, sub tasks, prerequisites, instructional strategies  Rule-using cognitive task Identify the materials needed  Rule-using cognitive task Identify the assistive technology needs  Ensure that all students have appropriate assistive technology throughout the lesson  Rule-using cognitive task Identify the evidence collections materials  Problem-solving cognitive task 42

Task Analysis

Challenges

     Is this the only lesson plan process?

Identifying types of cognitive tasks Identifying appropriate supports for each task Identifying instructional strategies What does the final result of each task look like?

43

Task Analysis

Lesson Plan Document

44

Solution System

Based upon the results of the needs and task analysis phases, our goals for the solution system are to: 1.

Provide scaffolding for teachers to develop lesson plans aligned with the new SOLs.

2.

Produce a searchable VAAP manual.

3.

Provide additional resources for teachers to utilize throughout the assessment process.

45

Analysis



Design

Analysis

Design

Develop Implement Evaluate Performance Analysis Needs Analysis Personas Task Analysis Design Approach Storyboards

46

Design Approach

EPSS Characteristics Electronic Performance Support System (EPSS)  Computer software program that improves performance by:   Reducing complexity Providing the performance information to perform a task  Providing a decision support system that enables the user to identify appropriate actions 47

Design Approach

EPSS Characteristics     Used on the job Controlled by the user Reduce the need for prior training in order to accomplish the task Supports new constraints on an existing process 48

Design Approach

Rationale     Supporting an existing process  Teachers have created lesson plans  New constraints on that process New standard format for lesson plans is a system requirement Teacher experience levels vary Just-in-time support in the classroom 49

Design



Development

Analysis Design Implement Evaluate

Develop

Flowcharts Schemas Prototype

50

Development of Prototype

 Users will be able to use the VA Assessments section on T/TAC Online to:  Search the VAAP Manual  Create Lesson Plans  Search for Created Lesson Plans 51

Searchable Online VAAP Manual

 Users should be able to search the VAAP Manual by:  Subject  Subject and ASOL code   Subject and Skill Group Instructional Strategy 52

Flowchart 1:

Searching the VAAP Manual

53

54

Creating a Lesson Plan

  Users will use the system to create lesson plans using a wizard with built in just-in-time supports Users can also receive assistance if they need help with a particular part of the lesson plan creation process 55

Flowchart 2: Lesson Plan Document

56

Lesson Plan Database Schema

57

Searching for Created Lesson Plans

 Users will be able to search for previously created lesson plans by:  Subject  Subject & Skill Group  Subject & ASOL Code 58

Design Approach

Prototype

http://immersion.gmu.edu/ttac/fall2005/epss 59

Next Steps

     Identify supports to be integrated into the EPSS     Interview expert teachers Surveys Resources Collaboration Continue Development of Prototype Implement Prototype Usability Testing Formative Evaluation 60

Thank You for Listening

Are there any questions?

61