Presented by Denise Sibley Laura Jean Kerr Mississippi Assessment Center Research and Curriculum Unit

   Requirement established by Perkins IV Assessments have been developed Standard setting needs to be established

The proper following of a prescribed, rational system of rules and procedures that result in determining:    cut scores and performance levels what’s tested proper statistical analyses

   Ensures that decisions are based on high quality data Ensures that data are combined in a systematic, reproducible, objective, and defensible manner Increases understanding and trust on the part of the stakeholders

4 5 6 7 8 2 3

Step

1

Description

Choose standard-setting method Prepare descriptions of the performance categories Select subject matter experts (SMEs) from each program area to conduct standard-setting Prepare materials for standard-setting meeting Train SMEs to use standard-setting method Conduct meeting, compile results, and provide feedback Conduct an evaluation of the standard-setting process Document the standard-setting process and evidence on the validity of the resulting performance standards

     Researched standard setting methods Limited literature for CTE assessments – lots of academic literature Method used had to present a strong validity argument for the results Method used had to be suitable for the CTE content Method used had to be straight-forward

    Performance level descriptors (PLDs) need to developed  Descriptions of the performance levels into which examinees will classified (e.g., Minimal, Basic, Proficient, or Advanced) Participants rate items according to PLDS Items with agreement less than .67 are re-rated Use results to distinguish between the four performance level categories.

   PLDs should be specific to each standard in the curriculum Challenging because there were over 100 curricula Developed Career and Technical Education PLDs

    Advanced: An advanced student evaluates and integrates transferable academic/workplace knowledge and skills in multiple situations as a productive contributor in the workplace.

Proficient: A proficient student acquires and applies academic/workplace knowledge and skills to become a productive contributor in the workplace.

Basic: A basic student has limited acquisition and comprehension of academic/ workplace knowledge and skills that are necessary to become a productive contributor in the workplace.

Minimal: A minimal student has not developed academic/workplace knowledge and skills that are necessary to become a productive contributor in the workplace.

Included program descriptions and standards from the curricula as a reference for participants to ensure they aligned their ratings with the curriculum

    Used stratified random sample method for selecting participants–geographic location based on 15 community colleges Randomly selected teachers from each geographic location by program area Had to be able to attend standard setting meeting Approximately 400 subject matter experts participated

      PLDs Standards from each curriculum framework Program description from each curriculum framework Test item booklet with only field-tested items  Item difficulty is used to determine the range for each performance level Sign-in sheets with logins and passwords Bubble sheets for backup if site networks went down

        Facilitators were trained prior to the meetings to insure that meetings were conducted in a consistent manner Meetings were held across the state over the course of 3 days A Web interface was developed for participants to enter their responses Scheduled participants from each program area to meet on the same day Participants received instructions on using the materials and the Web interface Participants went through test item booklets and matched items to PLDs and also were able to write comments about the items When all participants submitted their responses, they were calculated and any items that didn’t have a .67 agreement were re-rated.

Participants completed a survey after they completed the meeting

    Conduct Standard Setting over long distances Coordinate Standard Setting for the best use of Subject Matter Experts time Keep up with ever-changing needs of Career and Technical Education Organize and compile data

  Consistency is the justification for subjective limitations  Training of Subject Matter Experts   Standard Setting environment Policies and procedures Consistency contributes to quality data

  Communication  “The PowerPoint would have been helpful if I had seen it before today.”   “I did not fully understand the descriptors. I probably over estimated the students' ability.” “The performance level indicators are open to many interpretations.” Lesson learned:  Communicate with SMEs early and often.

  Meeting Environment    “It was the room and atmosphere where the training was held. It was in a room off the cafeteria. Which means...guess what...too much traffic. We could hear the phone conversations next door from an office, when the phone was ringing and etc.” “Could use a break for the 200 questions and refreshments.” “People who arrive late should not be able to enter.” Lesson Learned:  Every aspect (even the smallest detail) of the testing environment should be taken into consideration.

  Technology    “I would like to see the questions online with the performance descriptors. I believe the process would go a lot faster and would allow for easier comments.” “Website does not allow you to go back and alter responses on previous pages in event of mistakes.” “Love doing this on computer!” Lesson learned:  Incorporated these comments into development of new interface

  Trust and Understanding    “I appreciate being able to give feedback on the questions. This is the first time I have been invited to do so.” “I think that this is a step in the right direction for standardized testing.” “Good program...would like to know more about how the process works with our peers, etc.” Keeping stakeholders in the “loop” fosters trust

     Compile data Provide feedback to stakeholders Document process Improve process Publish results

Denise Sibley Interim Coordinator of Assessment and Accountability [email protected]

Laura Jean Kerr Assessment Specialist [email protected]