A. Effect Size Can Tell a Lot! B. Future Ready Core NC Accountability Conference February 2009

Mike Gallagher Accountability Services Dept.

NCDPI 1

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Purposes – Two Parts

• Part A. Show how effect size can give important information to compare test results across EOG, EOC and SAT and other tests.

• Part B. Review the background and give sources of information about the Future Ready Core.

2

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Part A: Effect Size Lens

• Usefulness: Enables one to compare across tests and test forms.

• Usefulness: Provides solid information about performance.

• Concept: What effect size means.

• Calculation: How to figure it.

3

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

How does North Carolina improvement on SAT math compare with its improvement on NAEP math?

4

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

SAT Math Score Trend for N.C.

• Between 1990 and 2008, the Math gap between NC and the Nation narrowed from 31 pts. to 4 pts.

• The annual SAT reports on the DPI Accountability web site have lots more information: www.ncpublicschools.org/accountability 5

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

SAT Math Score Trend

530 520 510 500 490 480 470 460 450 440 501 501 504 508 512 514 516 518 518 515 513 511 505 507 486 492 496 479 482 470 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 Nation North Carolina

6

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

NC progress on NAEP math

NAEP results from 1992 to 2005 Compared with the U.S. average and with other states, North Carolina went from back of the pack to ahead of the pack. NC was somewhat sluggish 2003 to 2007.

For the NAEP data in the following slides, go to: http://nces.ed.gov/nationsreportcard/mathematics/ 7

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

NAEP Grade 4 Math

Trend in Average Scale Score 250 245 240 235 230 225 220 215 210 1990

Cut for Basic = 214

1995 STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Cut for Proficient = 249

2000 2005 2010

8

NC US

NAEP Grade 8 Math

Trend in Average Scale Score 290 285 280 275 270 265 260 255 250

Cut for Proficient = 299 Cut for Basic = 262

1990 1992 1996 STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION 2000 2003 2005 2007

9

NC US

Finding the Effect Size aka, “Standardized Difference”

• Effect size compares two groups starting with a scale score.

• For calculation of effect size, the average scale score of each group is needed, as well as the standard deviation (SD) of the scores.

• Find the difference of the averages and divide by the SD.

10

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Comparing trends in tests: NAEP and SAT

• SAT math, comparing NC in 1990 and 2008.

…..Eff.Size = (511-470)/100 = 0.41

• NAEP Grade 4 math, comparing NC in 1992 and 2007.

…..Eff.Size = (242-213)/30 = 0.97

11

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Is that a “Big-un?”

General use Effect Size Indexes • 0.20 … Small • 0.50 … Medium • 0.80 … Large From A Power Primer, Jacob Cohen,

Psychological Bulletin,

1992, Vol 112, No. 1, 155-159.

12

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

While we’re talking SAT,

SAT puts an “official” question of the day on its web site.

The math questions are great thinking questions and they are mostly within our 8 th grade territory. Why not have 8 th graders use them as formative assessment for problem solving?

www.collegeboard.com

13

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Effect Size and NC Tests

• Effect size can provide a comparison across test editions or subjects.

• See the

attached table

examples.

for some …. The 3 rd Edition Math EOG had much more rigor than the 2 nd .

…. Effect size enables continuous comparison.

14

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Obtaining Av. Scale Scores

• The “disag” report is on-line.

• Go to the Accountability web page.

www.ncpublicschools.org/accountability then scroll way down to Data and Reports, chose “State, School System (LEA) and school performance disaggregated data report.” • Now choose location, test, grade, subj.

15

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Green Book

Find out about getting data from the Green Book and its online brother: • both in its standard form … • and its interactive, electronic form.

http://www.ncpublicschools.org/accounta bility/testing/ Scroll down to “State Testing Results” 16

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Questions to pose ..

• In your LEA (or school) which grades, courses, SAT results are stronger?

• In your LEA how do schools compare on average EOG, EOC, SAT?

• How does your LEA (school) compare with similar LEAs (schools)?

• What is the effect of what you’ve been trying to improve?

17

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Current Research

The Problem With “Proficiency”: Limitations of Statistics and Policy Under No Child Left Behind. By Andrew Dean Ho,

Educational Researcher

, 2008; 37, 351 • “The limitations [of the Percentage of Proficient Students (PPS) statistic] are unpredictable, dramatic, and difficult to correct.” …. From abstract.

18

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Part B Future Ready Core

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

19

ACT Study

… found that “whether planning to enter college or workforce training programs after graduation, high school students need to be educated in a comparable level in reading and mathematics.”

Ready for College and Ready for Work: Same or Different?

2006. Available on the web.

20

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Future Ready Core - Goals

On September 7, 2006, the NC Board of Ed approved goals: Future Ready Students for the 21 st Century.

Note the first goal and the 1 st item in that goal.

“

Goal: NC public schools will produce globally competitive students.

•

Every student excels in rigorous and relevant core curriculum that reflects what students need to know and demonstrate in a global 21st Century environment … ”

21

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Future-Ready Core Course of Study

The State Board of Education in June 2007 approved the Future-Ready Core Course of Study with the goal that all students would graduate prepared for post-secondary education and careers.

22

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

Future Ready Core Timeline

Beginning with the 2009-2010 ninth grade class, all students will be expected to meet the requirements outlined in the Future Ready Core Course of Study.

23

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

FRC: Primary Differences.

• An additional math course for those students who previously did not pursue the College/University Prep Course of Study.

• An increase from 20 to 21 for the total number of credits required for graduation.

• A recommended 4 elective credits in a concentrated area.

24

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

FRC: Mathematics

4 Credits

(Algebra I, Geometry, Algebra II) OR (Integrated Math I, II, III) 4th Math Course to be aligned with the student’s post high school plans

A student, in rare instances, may be able to take an alternative math course sequence as outlined under State Board of Education policy. Please see your school counselor for more details.

25

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

FRC - Information

• Information resources are linked from: www.ncpublicschools.org/gradrequire ments click “Resources.” 26

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION

RESOURCES (on dpi web site)

• Course & Credit Requirements Chart • Course and Credit Requirements Checklists for Parents • Graduating Future Ready • Making the Grade – Future Ready Graduates • Math Sequence • Questions and Answers • Student Accountability Standards Brochure 27

STATE BOARD OF EDUCATION DEPARTMENT OF PUBLIC INSTRUCTION