Transcript Research and Theory, Introduction to the Scale or Rubric, Writing
Research and Theory, Introduction to the Scale or Rubric, Writing Measurement Topics, Score 3.0 and Sample Tasks
Ravalli County Curriculum Consortium Marzano Research Laboratory
RESEARCH AND THEORY BEHIND THE STANDARDS RUBRIC OR SCALE
Why are we doing this?
Typical State and National Standards Documents simply have too much content to cover effectively and with any depth Standards lack unidimensionality The use of formative assessment data allows for the greatest gain in student achievement Scales or rubrics provide the best feedback to students and are the most accurate way to assess student knowledge and growth
Typical Standards documents cover too much content!
If you wanted to teach all of the standards in the national documents, you would have to change school from K-12 to K-22!
255 standards across 14 subject areas 3,500 benchmarks 15,500 hours of instruction needed to cover the 3,500 benchmarks (Kendall and Marzano, 2000) 9,000 hours of instruction available (Marzano, Kendall and Gaddy, 1999)
Typical Standards documents lack Unidimensionality
Standards documents are not written with assessment in mind Benchmarks typically cover more than one skill or piece of knowledge In order to assess each individual dimension, standards and benchmarks must be “unpacked” so that they measure only one thing at a time
The use of formative assessment data can dramatically improve student achievement
Black and Williams (1998) in a synthesis of 250 studies, concluded that the use of formative assessment data, done properly, “…does improve learning. The gains in achievement appear to be quite considerable, and as noted earlier, amongst the largest ever reported for educational interventions.”
Types of Feedback from Formative Assessments
Bangert-Drowns, Kulik, Kulik, & Morgan, 1991 # of studies Characteristic of Feedback from Classroom Assessment Percentile Gain/Loss
6 Right/wrong -3 39 Provide correct answers 8.5
30 9 Criteria understood by student vs. not understood Explain 16 20 4 Student reassessed until correct 20
Fuchs & Fuchs 1988 # of studies Characteristic of Feedback from Classroom Assessment Percentile Gain/Loss
89 Displaying results graphically 26 49 Evaluation by rule [uniform way of interpreting results of classroom assessments using a tight logic) 32
Why use a rubric or scale for formative assessment and to assess student progress over time?
Formative assessment using rubrics provides the best form of feedback to students Well written and specific scales or rubrics are more accurate than a set of tests or formal assessments graded on a 100 point scale
In a study conducted by Marzano in 2002, a group of science teachers independently developed a 100 point scoring scheme for a test and scored it. The reliability coefficient was only .294 (out of 1.0).
The resulting scores were so variable that the test could not reliably be used to assess student knowledge and/or skills
The same teachers scored the same test using a 4 point rubric or scale.
The reliability coefficient was .719
This means that the resulting scores were significantly more consistent, and as a result the rubric came much closer to measuring what the student actually knows or is able to do than the test scored using the 100 point scale.
Using Standards Rubrics or Scales
Will… Create greater reliability in assessment of standards Create more consistency in teaching, assessment and learning throughout a district or state Allow for more consistent and better feedback throughout the course of learning Allow for more focused instruction in appropriate depth Make true assessment possible due to the creation of unidimensionality IMPROVE LEARNING!
Questions?
Comments?
Discussion?
Writing Standards Scales or Rubrics
The Standards Scale or Rubric
Strand: Algebraic Relationships Score 4.0 Score 3.0 Score 2.0 Score 1.0 Score 0.0 Topic: Modeling Rates of Change Grade: 5 In addition to Score 3.0, in-depth inferences and applications that go beyond what was taught.
Sample Tasks
Generate a hypothesis about changes in the projected gas mileage of General Motors vehicles based on their weight. Research and graph your findings. Generate a hypothesis about changes in the projected gas mileage of General Motors vehicles based on their engine size.
3.5
In addition to score 3.0 performance, in-depth inferences and applications with partial success.
The student:
identifies, models and describes situations with constant or varying rates of change
The student exhibits no major errors or omissions.
2.5
No major errors or omissions regarding 2.0 content and partial knowledge of the 3.0 content
There are no major errors or omissions regarding the simpler details and processes as the student:
recognizes or recalls specific terminology such as: o constant rate of change, varying rate of change identifies line graphs showing constant or varying rates of change
However, the student exhibits major errors or omissions regarding the more complex ideas and processes. 1.5
Partial knowledge of the 2.0 content but major errors or omissions regarding the 3.0 content
With help, a partial understanding of some of the simpler details and processes and some of the more complex ideas and processes.
Create a line graph for Perimeter of 3-Tile and 4-Tile Rectangles showing the perimeter (cm) on the y axis and number of rows on the x axis. (As rows are added, there will be a constant rate of change indicated by the line graph.) Create a line graph for a Penny Jar situation that starts with one jar with 2 pennies and accumulates 2 pennies for each round and a second jar with 2 pennies that doubles the number of pennies for each round. (As pennies are added in the second situation, the change itself changes, resulting in a graph that is not straight.) Call the graph Comparing Penny Jars with the y axis labeled Total Number of Pennies and the x axis labeled Number of Rounds. Include a key for one line showing adding 2 pennies per round and the second line doubling the number of pennies per round. The teacher will have 6 line graphs prepared on 12” x 18” paper. Students will identify which graphs indicate a constant rate of change and which identify a varying rate of change.
0.5
With help, a partial understanding of the 2.0 content but not the 3.0 content
Even with help, no understanding or skill demonstrated.
4
3 2 1 0
The Scale
In addition to exhibiting level-3 performance, in-depth inferences and applications that go BEYOND what was taught in class No major errors or omissions regarding any of the information and/or processes (SIMPLE OR COMPLEX) that were explicitly taught The Learning Goal: what you expect the student to know and be able to do No major errors or omissions regarding the SIMPLER details and processes BUT major errors or omissions regarding the more complex ideas and processes The simpler or foundational knowledge that is necessary as a step to mastery of the Score 3.0
With HELP, a partial knowledge of some of the simpler and complex details and processes Even with help, no understanding or skill demonstrated
1.
2.
3.
4.
4 Easy(?) Steps
Unpack/Cut Down Standards/Organize /Write Measurement Topics Specify expected learning by writing Score 3.0, and write sample tasks to further specify content Identify steps towards mastery of the content by writing Score 2.0, write sample tasks to further specify content Pilot and Use! Design Assessments, Assess student learning, Report student progress
Strands
Broad categories within each standard Sample strands Math: Number Sense Geometry Algebraic Concepts Social Studies Geography History Economics Civics Science Life Science Earth Science Scientific Inquiry Language Arts Reading Writing
Measurement Topics
Categories of learning within each strand Organize your standards into more precise categories of study (Measurement Topic) within each strand The Measurement Topic represent the critical or essential content that you want to measure and record student progress on You may need to leave some things out!
You will write scales or rubrics for each of the MT’s Each MT will assess only one specific piece of information or skill for each grade level
Writing Measurement Topics
MRL highly recommends limiting the number of MT’s to no more than 30-35 per grade level Write each measurement topic specific enough that it can be described by only 1 or 2 elements Combine similar or covarying elements Covarying: as the ability in one increase, so does ability in the second Split up statements that are not unidimensional Unidimensional: measuring only one concept or skill Delete content that is not essential Delete content that is not measureable Reduce! You can’t assess it all!
Other Options
Consider including “Life Skills” to assess school behaviors Examples: Participation, Work completion, Behavior, Working in Groups Reorganize at the High School level Organize by class and write measurement topics for each class Consider including a Teacher Choice rubric allowing teachers to assess something other than what was included
Measurement topics do not have to be assessed at every grade level!
They can disappear and reappear at various levels
10 9 8 7 6 5 4 3 2 1 K
Topic Topic Span of Topics Topic Topic Topic Topic
MT’s for Ravalli County
MRL took the Montana Standards for Mathematics and wrote 33 DRAFT MT’s as a starting point As the local content experts, the final decision on the MT’s is up to you!
Add, delete, modify as you wish, BUT… Keep in mind the total goal of no more than 30-35 per grade level!
1) 2) 3) 4) 5)
Measurement Topic Review
Divide into 6 teams. Each team will take one strand Review the list of topics. Does this list represent the areas that you want to measure student progress in? Do any MT’s need to be renamed or deleted? Are there any topics that need to be added?
Take a first cut at the grade levels for each topic. When will each topic be taught?
Complete chart with topic and grade levels Review the work of each of the other teams and make suggestions REMEMBER: This list is not final! Topics can and will change once you begin defining the learning goals or elements!
WRITING SCORE 3.0
Writing Score 3.0 elements
What is a Score 3.0 element?
3.0’s represent all of content that is explicitly taught in class under a measurement topic This is what you expect a student to know and be able to do This represents the parts of the curriculum that are being assessed
General Guidelines for Writing 3.0’s
3.0 elements: have a procedural or declarative focus Are unpacked – measure only one piece of knowledge or skill Are learning goals, not tasks Are measureable!
Are specific!
Are differentiated by grade level
Learning Goals vs. Learning Tasks
Score 3.0 elements are written in the form of learning goals, not learning tasks or activities Learning goals specify the content or skills that the student is expected to know or do Learning tasks or activities are the classroom activities that a student does in order to accomplish or be assessed on the learning goal.
Examples
Goal The student describes the division of the country during the Civil War Task or activity The student creates a map showing the states that formed the Confederacy and those that stayed in the United States
Measurability
3.0 elements must be measureable Measureable: Observable or Quantifiable Makes predictions about reading The teacher can observe a student doing this Performs 10 push-ups with proper form The teacher can count the number of pushup and observe proper form Not Measureable The student enjoys independent reading
Specificity of Elements
Elements must be written so that a teacher who is unfamiliar with the process and unfamiliar with the curriculum, knows what the student is expected to know or be able to do and can accurately assess a student!
Avoid terms that can’t be easily defined or allow for subjective interpretation Terms can be defined within the element Examples can also be used to provide specificity Non-Specific Monitors own reading for comprehension
Examples
Specific Employs self-monitoring strategies while reading (
e.g., predicting, confirming, rereading
)
Differentiation of Elements
Elements must be written to show a increasing level of complexity as the performance level increases If the element remains the same through several performance levels, examples must be provided to differentiate each level For Example – ELA will frequently assess the same reading strategies (e.g., inference) at multiple performance levels, but… the complexity of the expectation or text will change. Give examples of the type of inference, or level appropriate text to differentiate
Unidimensional Rubrics
For each measurement topic, we are going to write one (maybe 2 if you beg!) Score 3.0 element Each element should measure only one individual piece of knowledge or skill If the topic requires more than one (or 2) elements, then break the topic into two or more new topics to maintain unidimensionality Remember, you can’t measure it all! You will have to make decisions about what to include and measure and what not to include and measure.
WRITING SAMPLE TASKS
What are sample tasks?
Sample Tasks are examples of assessment items or tasks that teachers can use to construct formative assessments Sample Tasks are written to present a model of an assessment task to help clarify the intent of the element, not as a definitive list of assessment tasks Assessment items that are written to address the appropriate taxonomic level for each element and score level
The Purpose of Writing Sample Tasks
Sample tasks: Help to clarify the expectations of each element Provide instructional guidance to the teacher Suggest ways to assess an element Can be used to design formative assessment Can be used to construct a common assessment item bank
Writing Sample Tasks
Ask yourself some basic questions: How can a student demonstrate success for this element?
What kinds of learning activities or tasks would assess success in this element?
What taxonomic operation would be appropriate for this task at this grade level?
General Rules
When explaining or other higher level processes are involved in sample tasks, ensure that the students are explaining
how
or
why
something is happening and not just
that
elements are worded correctly this should not be a problem.
it happens. If the If an element refers to memorization, a sample task must say something like, produce from memory. Each bulleted element will generally have a corresponding sample task.
Be specific!!!!
Organizing the Work
We suggest that you split the work up into several teams, each with representatives from lower elementary, upper elementary and middle school. High school reps are helpful, if possible.
Each team would be responsible for writing the scales or rubrics K-8 within a measurement topic. This helps to ensure vertical alignment. A high school rep is helpful to make sure that the student ends up where they need to be!
High school teams can work to create rubrics by course.
Our Job today:
Review Draft MT’s and take a first cut at revising them. You can and will continue to revise as you develop the rubrics Develop the Learning Goal (3.0) for each of the MT’s at each grade level YOU WILL NOT FINISH THIS TODAY!