Transcript Program Review Process - Council for Exceptional Children

4 - 2014
Program Review Update Webinar
CEC
Standards
Review
Language
Reviews
Conditions
Reviews
Discussion
• Preparation standards Revised
• Clarity and Quality
• Well-Written
• Stay focused
• Challenges, Suggestions, & Questions
7/7/2015
© 2013 Council for Exceptional Children. All
rights reserved.
2
CEC Preparation Standards
Revised Preparation Standards
CEC Preparation (Content) Standards
Existing CEC Initial
Content Standards
7/7/2015
New CEC Initial
Preparation Standards
2. Development & Characteristics
3. Individual Learning Differences
1. Learner Development & Individual
Learning Differences
5. Learning Environments & Social
Interactions
2. Learning Environments
1. Foundations
3. Curricular Content Knowledge
8. Assessment
4. Assessment
4. Instructional Strategies
6. Language - Embedded Within All
Preparation (Content) Standards
7. Instructional Planning
5. Instructional Planning & Strategies
9. Professional & Ethical Practice
6. Professional Learning & Ethical
Practice
10. Collaboration
7. Collaboration
© 2013 CEC. All rights reserved.
4
CEC Advanced Preparation (Content) Standards
Learners and Learning
1. Assessment
Content Knowledge and Professional Foundations
2. Curricular Content Knowledge
Instructional Pedagogy
3. Improving Supports & Services
4. Research and Inquiry
Professionalism and Collaboration
5. Leadership and Policy
6. Professional and Ethical Practice
7. Collaboration
7/7/2015
© 2013 CEC. All rights reserved.
5
CEC Preparation (Content) Standards:
Revised and Reorganized
November
2012
SASB approves CEC Preparation Standards Report
and triggers the start of the two year transition
period
During the transition period program faculty may
align their program assessments to either the CEC
2012 – 2015
Content Standards or the CEC Preparation
Standards
At the end of the transition period program faculty
Spring, 2015 must align their program assessments to the CEC
Preparation Standards
7/7/2015
© 2013 CEC. All rights reserved.
6
Consistency to Enhance
Reliability and Quality
CEC Program Recognition FAQ
Program Report Developer Resources
In addition to an array of technical support options, CEC provides the following resources:

Initial Level CEC Content Standards & Assessments Alignment Example

Language Analysis Project

Language Analysis Scoring Rubric

Section II: Program-based Performance Assessments

CEC Program Recognition FAQ

CEC Program Reviewer Application
Evidence for Meeting Standards: Assessment 2 - Assistive Technology Project
Section III Assessment to Standards Alignment
Guidelines for Preparing Recognition with Conditions Reports
Assessment II: Content Knowledge - Comprehensive Examination
Evidence for Meeting Standards: Assessment 5 - Behavior Change Project
Assessment 6 -Portfolio Artifact - IEP/IFSP
Assessment VIII Special Education Assessment Work Sample Folio
Developing Performance-based Special Education Preparation Program Reports (PowerPoint)
Guidelines for Course Grades as an Assessment of Candidate Knowledge
CEC Content Standards and Program Assessments Alignment Table











Unit Standards
7/7/2015
© 2013 CEC. All rights reserved.
8
Field Experiences
Appropriate to the license and roles for
which they are preparing, candidates
progress through a
 series of developmentally sequenced
field experiences for the
 full range of:




7/7/2015
ages,
types and levels of abilities, and
collaborative opportunities
that are supervised by qualified
professionals.
© 2013 CEC. All rights reserved.
9
Field Experiences
PART B - STATUS OF MEETING SPA STANDARDS
Field Experiences and Clinical Practice Standard
Met
The evidence in the program report establishes that special education
candidates progress through a series of developmentally sequenced field
experiences for the full range of ages, types and levels of abilities, and
collaborative opportunities that are appropriate to the license or roles for which
they are preparing, and that these field and clinical experiences are supervised
by qualified professionals.
PART B - STATUS OF MEETING SPA STANDARDS
Field Experiences and Clinical Practice Standard
NOT Met
The evidence in the program report establishes that special education
candidates progress through a series of developmentally sequenced field
experiences for the full range of ages, types and levels of abilities, and
collaborative opportunities that are appropriate to the license or roles for which
they are preparing.
THERE IS INSUFFICIENT EVIDENCE that these field and clinical experiences
are supervised by qualified professionals.
7/7/2015
© 2013 CEC. All rights reserved.
10
Program Assessment Components
7/7/2015
© 2013 CEC. All rights reserved.
11
CEC Preparation Standards
CEC does NOT
expect Specialty Set
Knowledge and Skills
ITEMS
to be explicitly referenced
to assessment items
or rubrics!!!!
7/7/2015
© 2013 CEC. All rights reserved.
12
Major Elements of CEC Preparation (Content) Standards
The program assessments, scoring
guides/rubrics, and data tables are
to be aligned to
the major elements (bolded
phrases) of each CEC Preparation
(Content) Standard as informed by
the appropriate specialty set(s)
7/7/2015
© 2013 CEC. All rights reserved.
13
Specialty Sets Inform the Preparation (Content) Standards
Question
What does “informed by the appropriate specialty knowledge and skills set mean?
Should the program report explicitly refer to specific CEC Knowledge and Skills Specialty
sets?
The major elements of the CEC Preparation (Content) Standards as informed
by the appropriate specialty set are to be reflected in the program’s
assessments, rubrics, and data. This means that every program must
demonstrate alignment to the major elements of the CEC Preparation
(Content) Standard as informed by the appropriate specialty set(s) whether
the program uses the Initial or Advanced Preparation (Content) Standards.
Without being informed by the appropriate specialty knowledge and skills set,
every special education preparation program would look the same. The content
of the Specialty sets differentiate program assessments.
Response
Programs can assure that the assessments, rubrics, and data are informed by
the appropriate specialty area in a variety of ways, but the most meaningful
way is to assure that performance levels within rubrics use of the content from
the appropriate specialty set(s).
There is no requirement or expectation for explicit or complete
correspondence between the items in a specialty set with assessment
items, and reviewers do not look for this level of correspondence.
Likewise, reviewers do not expect that programs use the exact wording of the
knowledge and skills within the rubrics. However, it is expected that the
content from the appropriate specialty set(s) is used in designing assessments
and rubrics.
7/7/2015
© 2013 CEC. All rights reserved.
14
Question
Do program reviewer references to “CEC Standards” pertain to the
Initial or advanced CEC Preparation (Content) Standards or to the
CEC Knowledge and Skill Specialty sets?
Response
CEC Program Reviewers use the CEC Preparation (Content)
Standards with the key elements as the organizing focus for their
review, and their references to “CEC Standards “are to the CEC
Preparation (Content) Standards.
7/7/2015
© 2013 CEC. All rights reserved.
15
Question
Response
Where does the program report provide evidence that the specialty set
informs the major elements of the CEC Preparation (Content) Standards?
CEC does not require or promote that program reports cite specific specialty
set items. However, CEC expects program reviewers to assure clear and
convincing evidence that the content of the appropriate specialty set(s) is
used in the assessments, rubrics and scoring guides, and Section 1
narratives.
Program faculty should assure that the content, populations,
vocabulary, concepts, settings, and issues from the specialty
Assessment set are used throughout the assessment items and
components.
Program faculty should assure that the content, populations,
Rubrics &
vocabulary, concepts, settings, and issues from the specialty
Scoring Guides set are used throughout.
Section I
Narrative
7/7/2015
Program faculty should describe how the assessment
addresses the specialty set specific content, populations,
vocabulary, concepts, settings, and issues.
© 2013 CEC. All rights reserved.
16
Aligning Program Assessments and Major Elements of the
CEC Preparation (Content) Standards
Question
Must program reports provide evidence that program
candidates master the major elements of the CEC
Preparation (Content) Standards?
CEC requires that a preponderance of the evidence
establish that the assessments align with the major
elements of the CEC Preparation (Content) Standards
as informed by the content of the appropriate specialty
set(s) and that program candidates master the major
Response
elements in CEC Preparation (Content) Standards as
informed by the appropriate specialty set(s).
7/7/2015
© 2013 CEC. All rights reserved.
17
Preponderance of Evidence
Question
What does CEC mean by “a preponderance of the evidence” for the
major elements of the CEC Preparation (Content) Standards?
“Preponderance of evidence” is a standard of proof indicating that the
evidence is clear and convincing, as opposed to the more rigorous “beyond a
reasonable doubt” standard. CEC program reviewers use the
“preponderance of the evidence” standard with each of the CEC Preparation
(Content) Standards. Using a preponderance of the evidence standard, the
reviewer judges whether the evidence in the report is clear and convincing. A
preponderance of evidence cannot be reduced to a simple quantity, i.e. 75%.
It is a reasoned judgment by a set of collegial reviewers and auditors based
on the evidence presented.
Response
7/7/2015
In order to determine that a program meets a CEC Preparation (Content)
Standard, the reviewers judge whether the pieces of evidence presented in
the program report are clear and convincing that the program assessment
aligns with the major elements of the respective CEC Preparation (Content)
Standard and that the program data demonstrate that the program
candidates are mastering the major elements of the CEC Preparation
(Content) Standard.
© 2013 CEC. All rights reserved.
18
Rubrics
Question
Our program faculty use a 3-tier rank i.e. "Unacceptable",
"Acceptable", or "Proficient". With a range of points assigned
to each of these categories.
Response
Routinely, reviewers look for whether the
performances at "Unacceptable", "Acceptable", or
"Proficient“ are clearly described on each scale.
Assigning each of the three tiers with a range of
scores is only acceptable as long as the ranges
are sufficiently described and differentiated to
make a reasonable level of inter-rater reliability
possible.
7/7/2015
© 2013 CEC. All rights reserved.
19
Data
Question
How many years of performance data are required in the
Program Report?
Initial submission program reports must include data for at least
2 administration cycles of the assessments.
In the case of state or national examinations that are given
multiple times throughout the year, data from two university
terms must be included
Response
Recognition with Conditions reports must include data from at
least 1 administration cycle of the assessments beyond the
data in the initial report.
Under no conditions does CAEP require program reports to
include data from more than three administration cycles of the
assessments.
A program is eligible for “Recognition with Conditions” with no
data
7/7/2015
© 2013 CEC. All rights reserved.
20
State - Assessments
Faculty may use State Assessments as
supplementary evidence for meeting CEC
Preparation Standards,
 But faculty may not use State
Assessments as a sole source of evidence
for meeting any CEC Preparation
Standard.

7/7/2015
© 2013 CEC. All rights reserved.
21
Well-Written Reviews
Writing Style
Write in professional language
DO NOT
 Use “I” or “me” or any other first person language
 State opinions or tell program faculty how to solve
problems
 Make side comments or direct questions to program
faculty or CEC Audit Team
AVOID
 Prescriptions
 Overstatements
 Personal observations
Proof, proof, and proof again!
7/7/2015
© 2013 CEC. All rights reserved.
23
Mechanics
 Write
in simple complete
active-voice sentences
 Be sure cuts and pastes fit the
program you are reviewing
 Check spelling
7/7/2015
© 2013 CEC. All rights reserved.
24
Thorough Report


Write comments that support the rating for each CEC Preparation
(Content) Standard
Write narrative for each section and each part excluding only
“Directions to the BOE” for which reviewers may or may not choose
to respond
The
Evidence
Your
Findings
Your
Conclusions
7/7/2015
The program report identifies the following program assessments as having elements
that align with CEC Preparation (Content) Standard 3
Assessment 3 Explicit Instruction Lesson Plan
Assessment 5 Curriculum-Based measurement Project
Assessment 6 Behavior Change Project
In regards to the cited Assessments, the description, scoring rubric, and program
candidate data for each are present and aligned to each other.
The Explicit Instruction Lesson Plan and the Curriculum-based Measurement Project
provides evidence that the assessments and scoring guides are aligned with the major
elements of the CEC Preparation (Content) Standard as informed by the appropriate
specialty set.
The report provides data for these assessments that the candidates are mastering the
major elements of the cited CEC Preparation Standard.
The preponderance of the evidence establishes that the program assessments align
with the CEC Preparation (Content) Standard, and that the program candidate data
indicate candidates are mastering the respective knowledge and skills.
© 2013 CEC. All rights reserved.
25
Non-Alignment Problem
The
Evidence
The program report identifies the following program assessments as having
elements that align with CEC Preparation (Content) Standard 3
Assessment 3
Explicit Instruction Lesson Plan
Assessment 5
Curriculum-Based measurement Project
Assessment 6
Behavior Change Project
In regards to the cited Assessments, the description, scoring rubric, and
program candidate data for each are present and aligned to each other.
Your
Findings
The Explicit Instruction Lesson Plan and the Curriculum-based
Measurement Project NOT provide evidence that the assessments and
scoring guides are aligned with the major elements of the CEC Preparation
(Content) Standard as informed by the appropriate specialty set.
Since the program assessment elements are NOT aligned to the major
elements of the CEC Preparation (Content) Standard , the report cannot
provide data that the candidates are mastering the major elements of the
cited CEC Preparation Standard.
Your
Conclusions
7/7/2015
The preponderance of the evidence does not establish that the program
assessments align with the CEC Preparation (Content) Standard, and that
the program candidate data indicate candidates are mastering the
respective knowledge and skills.
© 2013 CEC. All rights reserved.
26
Decision Support


The Program Review Report must support the decision.
Areas for considerations should tell program faculty what
needs to be addressed, not what is wrong or how to fix it
C.1. Candidates’ Knowledge of Content
The preponderance of the evidence presented for the program assessments and
the extent of their alignment to the major elements of the CEC Preparation
(Content) Standards (or does not) establishes that the program candidates have
satisfactory mastery of special education professional content knowledge.
C.2. Candidates’ Ability to Understand and Apply Pedagogical and
Professional Content knowledge, skills, and dispositions
The preponderance of the evidence presented for the program assessments and
the extent of their alignment to the major elements of the CEC Preparation
(Content) Standards (or does not) establishes candidates’ satisfactory ability to
understand and apply pedagogical and professional content knowledge, skills, and
dispositions.
7/7/2015
© 2013 CEC. All rights reserved.
27
Can’t make a decision
Refer to the Audit Committee, as
soon as possible, and please
include why you can not make a
decision
7/7/2015
© 2013 CEC. All rights reserved.
28
Recognized with Conditions
Decisions
•Reviewing
•Writing
RECOGNIZED WITH CONDITIONS
Well-written “Recognized with Conditions” decisions are:

BASED ON THE EVIDENCE

CLEAR

PRECISE

OBJECTIVE (Unbiased)

CONSISTENT with all narrative in the Review Report

COMPLETE, i.e. state everything to be included in the
next report
7/7/2015
© 2013 CEC. All rights reserved.
30
RECOGNIZED WITH CONDITIONS
CEC program reviewers need sufficient
information to review “Recognition with conditions”
reports.
So program report developers know specifically
what parts of the report to address and resubmit,
use the following italicized language in Part A.
Recognition Decisions of program review reports.
7/7/2015
© 2013 CEC. All rights reserved.
31
Question
Response
7/7/2015
Is there specific language to include in writing a “Recognition with conditions
program report”,
The following language should be included in Part A. Recognition Decisions of
program review reports.
CEC Preparation (Content) Standards x, y, z were found to be either “not met”
or “met with conditions”. For each CEC Preparation (Content) Standard or
CEC Field Experience Standard judged either “not met” or “met with
conditions”. the program resubmission report must provide:
1. The Section II and Section III tables that document the alignment of each
program assessment to the major elements of the CEC Preparation
(Content) Standard as informed by the appropriate specialty set(s);
2. The assessment descriptions, scoring guide/rubric, and data for each of
the program assessments that provide the evidence that they are aligned
to the major elements of each of CEC Preparation (Content) Standard as
informed by the specialty area knowledge and skills set(s); and
3. Sufficient performance data for reviewers to determine that the
preponderance of the performance data for each of the CEC Preparation
(Content) Standard as informed by the appropriate specialty set(s)
demonstrate that the program candidates master the major elements of
the CEC Preparation (Content) Standards as informed by the appropriate
CEC knowledge and skill set(s).
© 2013 CEC. All rights reserved.
32
RECOGNIZED WITH CONDITIONS


Whenever a “conditions” program report has additional
conditions, add these specific conditions to the review
report.
The assessment description, scoring guide/rubric, and
data form a vital chain and as the metaphor points out,
the chain is only as strong as the weakest link. It is
helpful to program faculty if the program review report
specifically identifies weak link(s). For example, “While
all the materials described above are required in the
resubmission, the scoring rubrics were particularly
problematic and will require extensive modifications.”
7/7/2015
© 2013 CEC. All rights reserved.
33
Reviewing Resubmitted Conditions Reports
All
previously “not met” conditions need
to be “met”.
 DO
NOT RE-REVIEW CEC PREPARATION
STANDARDS THAT WERE MET PREVIOUSLY.
No
new conditions may be cited.
If third review and clear progress toward
meeting conditions has not been made,
bite the bullet and give a “not
recognized.”
7/7/2015
© 2013 CEC. All rights reserved.
34
For the Programs You are Reviewing

Do:




assessments,
scoring guides/rubrics and
data
align in clear and convincing ways to the major elements
of the CEC Preparation (Content) Standards informed by
the appropriate knowledge and skill set?
Do they meet the data requirements?
7/7/2015
© 2013 CEC. All rights reserved.
35
Questions So Far and Topics For the Future
7/7/2015
© 2013 CEC. All rights reserved.
36
7/7/2015
© 2013 CEC. All rights reserved.
37