Transcript Measurement Workshop

Measuring important learning outcomes in the context of
two linked UK (mathematics education) projects: From
instrument development, to measure validation and
statistical modeling
Maria Pampaka (The University of Manchester)
Trondheim, February 2011
TLRP: “Keeping open the door to mathematically demanding
F&HE programmes” (2006 – 2008)
TransMaths: “Mathematics learning, identity and educational
practice: the transition into Higher Education” (2008-2010)
Outline of Presentation
•Some background to the projects
•Instrument Development
•Measure construction and validation
A measure of pedagogy
A measure of Mathematics Self Efficacy
•A modeling approach to respond to the
research questions
TLRP: “Keeping open the door to mathematically demanding
F&HE programmes” (2006 – 2008)
TransMaths: “Mathematics learning, identity and educational
practice: the transition into Higher Education” (2008-2010)
• Funded by ESRC (Economic and Social Research Council)
• The University of Manchester, School of Education
• The team:
Lead Principal Investigator: Prof Julian Williams
Other PIs:
– Laura Black
– Pauline Davis
– Graeme Hutcheson
– Brigit Pepin
– Geoff Wake
Researchers: Paul Hernandez-Martinez
Maria Pampaka
Educational System in UK (England)
Aims of the Projects
TLRP: To understand how cultures of learning and teaching can
support learners in ways that help them widen and extend
participation in mathematically demanding courses in Further
and Higher Education (F&HE)
•
AS Mathematics Vs AS Use of Mathematics
TransMaths: To understand how 6th Form and Further
Education (pre-university) students can acquire a mathematical
disposition and identity that supports their engagement with
mathematics in 6fFE and in Higher Education (HE)
•
Focus on Mathematically demanding courses in HE (‘control’ : non
mathematically demanding, e.g. Medicine and Education)
•Mixed methodology: longitudinal case studies, interviews
and surveys
TLRP Research Design:
The Survey in the general framework
March 06
Programme
effectiveness
Questionnaire
design
Classroom
practices
Learner
identities
DP1
Sept-Nov 2006
Pilot case studies
Sept 06
(i) initial
questionnaire
June 07
(ii) post test
Sept 07
(iii) delayed
post test
Dec 07
Case studies
in UoM and
traditional AS
Follow up
case studies
(i) initial
interviews
(ii) interviews
round 2
(iii) follow-up
interviews
DP2
Apr-June 2007
Teachers’
Survey
DP3
Sept-Dec 2007
The Surveys within the TransMaths Project and
the Relevant Research Questions
Sept-Oct
2009
April
2008
Instruments
Survey
End
of A2
Early
1st year
HE
Start
of 2nd
year HE
Case Studies
RQ1: How do different mathematics educational practices found in preuniversity/university transition interact with social, cultural and historical
factors to influence students’ (a) learning outcomes, and (c) decisions in
Interviews
relation to learning and using mathematics?
RQ2: How are these practices mediated by different educational systems (their
pedagogies, policies, technologies, assessment frameworks, institutional
conditions and initiatives)?
Analytical Framework
Instrument Development
Measures’ Construction
and Validation
(Rasch Model)
Model Building
(Multiple Regression, GLM)
The teacher’s Survey [TLRP]
• Section A: Background and class information
The TLRP Instrument
(Student Questionnaire)
• Section A: Background Information
• Section B: [Disposition to enter HE and study
mathematically demanding subjects]
• Section C: Mathematics Self-Efficacy
Instrument
Section A: Background Information
• Name, college, date, date of birth
• Address and telephone number (for follow up
survey/interview)
• Gender
• Course (UoM or AS Maths)
• Previous math qualification (GCSE grade and tier)
• University attended by close family
• Language of first choice
• Education Maintenance Allowance (EMA)
Section B: Dispositions…
Disposition to go to HE Intention to go to University?
• Expectations: family, friends, teachers
Disposition to continue with mathematically demanding
courses in HE
• Intention to study more maths after this course?
• Amount of mathematics in preferred option
• Importance of amount of mathematics of course in decision
• Feelings about future study involving maths
• Preferred type of maths (familiar, new)
University: what course? 
Using Mathematics: Self Efficacy
A “pure” item:
You are asked to rate how confident you are that you will be able to
solve each problem, without actually doing the problem, using a
scale from 1(=not confident at all) to 4(= very confident)
Using Mathematics: Self Efficacy
An “applied” item:
The TransMaths Student
Questionnaire
• Section A: Background Information
–
–
–
–
–
University, Course/Programme
Previous math qualifications
Ethnicity, gender, country of origin, language
Proxies of socio economic background
Special educational needs
• A series of instruments about different
aspects of the transition to HE…
The TransMaths ‘instruments’
Items / Measures
DP4
DP5
DP6

Reasons for choosing University and course

Experiences that influence choice of Uni Programme

Disposition to complete chosen course


Preparedness and Usefulness of ways of studying



Transitional Experiences
Mathematics Dispositions

Perceived Pedagogic Practices at Pre-uni (maths) experience




Perceived Pedagogic Practices at Uni (maths)
Mathematics Self Efficacy

Confidence with Mathematics



Usefulness of Mathematics



Perceived Mathematical support at Uni


Relevance of mathematics

Analytical Framework
Instrument Development
Measures’ Construction
and Validation
(Rasch Model)
Model Building
(Multiple Regression, GLM)
Constructing the measures:
Measurement methodology
• ‘Theoretically’: Rasch Analysis
– Partial Credit Model
– Rating Scale Model
• ‘In practice’ – the tools:
– FACETS and Quest Software [Winsteps more user
friendly]
• Interpreting Results:
– Fit Statistics (to ensure unidimensional measures)
– Differential Item Functioning for ‘subject’ groups
– Person-Item maps for hierarchy
Example 1:
Measuring Mathematics Self
Efficacy…some background
• Self-efficacy (SE) beliefs “involve peoples’ capabilities
to organise and execute courses of action required to
produce given attainments” and perceived self-efficacy
“is a judgment of one’s ability to organise and execute
given types of performances…” (Bandura 1997, p. 3)
• "a situational or problem-specific assessment of an
individual's confidence in her or his ability to
successfully perform or accomplish a particular maths
task or problem" (Hackett & Betz, 1989, p. 262)
Background – Why Mathematics Self
Efficacy?
• ‘Important in students’ decision making (sometimes
more than actual test scores)
• Positive influence on students’ academic choices,
effort and persistence, and choices in careers related
to maths and science.
• How to measure?
Contextualised questions
• TLRP project: a 30 item instrument for pre-university
students
Example 1: Measuring Mathematics Self
Efficacy [at the transition to university]
 Instrument measuring students’ confidence in different
mathematical areas ( 10 items):
•
•
•
•
•
•
•
•
Calculating/estimating
Using ration and proportion
Manipulating algebraic expressions
Proofs/proving
Problem solving
Modelling real situations
Using basic calculus (differentiation/integration)
Using complex calculus (differential equations / multiple
integrals)
• Using statistics
• Using complex numbers
Measuring
Mathematics
Self…
Efficacy
An example
Item
An example “applied” item
1.
Problem solving:
The table below shows how the circumference of a glass varies with height.
Find, using mathematics, the height to which you would need to fill the glass so that you have half a glass of
beer.




Not confident at all
Not very confident
Fairly confident
Very confident
Methods and Sample
•10 items
• 4 point Likert Scale (for frequency)
• Sample:1630 students
•Rasch Rating Scale Model
Results [1] – Checking Validity
One measure?
Table 2: Measures and fit statistics for the items of the scale
-------------------------------------------------------------------------------------------------|
|
Outfit |
Model | Infit
| Obsvd Obsvd Obsvd Fair-M|
|
| Score Count Average Avrage|Measure S.E. |MnSq ZStd MnSq ZStd | PtBis | Nu Items
-------------------------------------------------------------------------------------------------|
1.0 0 | .53 | 1 modeling
2.4 2.41| 1.28 .04 | 1.0 0
1328
| 3182
|
0.9 -3 | .56 | 2 calculating
3.2 3.25| -.56 .05 | 0.9 -3
1331
| 4225
1.0 -1 | .54 | 3 ratio_proportion |
3.1 3.13| -.26 .04 | 1.0 -1
1323
| 4055
|
0.9 -2 | .69 | 4 algebra
3.4 3.54| -1.39 .05 | 1.0 0
1326
| 4568
|
1.0 0 | .65 | 5 proof
.25 .04 | 1.0 0
2.9 2.91|
1320
| 3770
0.9 -2 | .55 | 6 problem_solving |
.38 .04 | 0.9 -2
2.8 2.85|
1318
| 3691
0.9 -1 | .71 | 7 Basic_calculus |
3.2 3.31| -.74 .05 | 1.0 1
1309
| 4239
|
0.9 -2 | .65 | 8 complex_calc
.46 .04 | 0.9 -2
2.8 2.82|
1313
| 3633
|
1.4 8 | .39 | 9 statistics
.36 .04 | 1.3 7
2.8 2.86|
1309
| 3683
|
1.1 3 | .63 | 10 complex_num
.20 .04 | 1.2 4
2.9 2.93|
1308
| 3764
-------------------------------------------------------------------------------------------------|
.00 .04 | 1.0 0.3 1.0 -0.1| .59 | Mean (Count: 10)
| 3881.0 1318.5 2.9 3.00|
|
.71 .00 | 0.1 3.2 0.1 3.3| .09 | S.D.
8.0 0.3 0.30|
| 374.2
-------------------------------------------------------------------------------------------------RMSE (Model) .04 Adj S.D. .71 Separation 16.15 Reliability 1.00
Fixed (all same) chi-square: 2476.3 d.f.: 9 significance: .00
Random (normal) chi-square: 9.0 d.f.: 8 significance: .34
--------------------------------------------------------------------------------------------------
Item Fit Statistics t check for the assumption of unidimensionality
Results [2] – Checking validity
Differences among student groups
1.5
Non-Maths Students
1.0
Items more relevant to
AS/A2 Maths context
More difficult for non maths
students
SE1: Modelling
SE10:Complex Numbers
0.5
0.0
SE6: Problem Solving
SE7: Calculus
SE9: Statistics
SE3: Ratio/Proportion
-0.5
-1.0
-1.5
-2.0
SE4: Algebra
-1.5
SE2: Caculating/Estimating
-1.0
-0.5
0.0
0.5
Students of Math-Demanding Courses
Differential Item Functioning
1.0
1.5
Results [3]
1.5
2.0
1.0
1.5
0.5
1.0
Non-Maths Students
Non-Maths Students
Two separate measures
Proof
0.0
Calculus
-0.5
SE1: Modelling
0.5
0.0
SE9: Statistics
-1.0
-0.5 SE3: Ratio/Proportion
-1.5
-1.0
-2.0
-2.0
-1.5
-1.0
-0.5
0.0
0.5
Students of Math-Demanding Courses
2a. AS-related MSE@Uni
Multidimensional Scaling?
1.0
1.5
-1.5
-1.5
-1.0
-0.5
0.0
0.5
Students of Math-Demanding Courses
2b. Applied MSE@Uni
1.0
1.5
Constructing the measures
•
Example 2: The Teacher Survey (TLRP)
‘28 item survey to teachers
 5 point Likert Scale (for frequency)
 Sample:110 cases from current project
 Rasch Rating Scale Model
Constructing the measures – Validity
[Unidimensionality - Fit]
B6: I encourage students to work more slowly
B24: I cover only the important ideas in a topic
Constructing the measures:
A measure of ‘pedagogical style’
Constructing the measures:
A measure of ‘pedagogical style’
I tend to follow the textbook closely
Students (don’t) discuss their ideas
I encourage students to work more quickly
I teach each topic separately
I tell students which questions to tackle
I know exactly what maths the lesson will
contain
Students (don’t) invent their own methods
Validation supported by qualitative data
“…I do tend to teach to the syllabus
now…If it’s not on I don’t teach it. …
but I do tend to say this is going to be
on the exam…”
“It’s old fashion methods,
“….
froma the
teachers
that I’ve
there’s
bit of
input from
metme
and
to…
it seems
attalked
the front
and
then I to me
thattry
onetoofget
the
big working,
differences is, I
them
mean I don’t
sort of as
use
practicing
questions
textbooks…
]…I want to get
quickly as [possible…”
students to think about the math, I
want students to understand, I
want students
to aconnect
ideas
“… there’s
sense that
I’ve
together,
to seethe
all those
things
achieved
purpose…I’ve
that gofound
together
I don’t
think
a
outand
what
they’ve
come
text and
bookwhat
did they
that…[
]. come
with
haven’t
with so…we can work with that
now”
The most challenging measure…
Transitional
Experiences
Analytical Framework
Instrument Development
Measures’ Construction
and Validation
(Rasch Model)
Model Building
(Multiple Regression, GLM)
From measures to GLM Modeling - TLRP
Outcome
Measure [A]
DP(n)
=
Outcome
Measure [A]
DP(n-1)
+
Related
Outcome
Measures
[B,C,..]
+
Process
variables
[course,
pedagogy,..]
+
Background
variables
Variables
• Outcome of AS Maths (Grade, or Dropout)
• Background Variables
• Disposition Measures at each DP
– Disposition to go into HE (HEdisp)
– Disposition to study mathematically demanding subjects in HE
(MHEdisp)
– Maths Self Efficacy (MSE: overall, pure, applied)
• A score of ‘pedagogy’ based on teacher’s survey
The TLRP Sample
Gender
Male
Female
Programme
AS Trad
AS UoM
781 (59.9 %)
351(68.8%)
523 (40.1%)
159 (31.2%)
1304
510
Total
1132 (62.4%)
682 (37.6%)
1814
Longitudinal design
–DP1: 1792
–DP2: 1082
–DP3: 608
Resolution for some outcome variables (e.g. AS outcome)
–Phone survey, School’s databases, Other databases
Results [1]: Math Dropouts
• Percentages of dropouts by course and previous attainment
GCSE tier-grade
Programme A* and A BHigher
BIntermediate CHigher
CIntermediate
AS Trad
9% (16%) 18% (50%) 31% (61%) 26% (65%) 46% (80%)
UoM
12% (13%) 18% (26%) 9% (24%) 27% (54%) 18% (45%)
•Effect Plots for a logistic regression model of
dropout
Our modeling framework (TransMaths)
VARIABLES COLLECTED
Learning
Outcomes
A
AS etc
Grades
Dispositions
B
LearningPreparedB4
LearningImportB4
PercistenceFinish
MathDisp-4
MSE-4
MathConfidencence
MathImportance
Intentions /
Decisions
C
UniInt
STEMint
Process / Conditions
D
Background
E
ChoiceReasons
Gender
ChoiceInfluence
UK/International
PerceptionPed@Colege
DP 4
January Exams @ some places
LearningPreparedUNI
LearningImportUNI
PercistenceFinish
MathDisp-5
MathConfidencence
MathImportance
MathRelevance
UniInt
STEMint
TransitionExperience
PerceptionPed@UNI
MathSupport
Language
Ethnicity
Family-in-HE
DP 5
SEN
DP 6
End of Year 1 Exams
End year 1
outcome
PercistenceFinish
MathDisp-6
MathConfidencence
MathRelevance
UniInt
STEMint
MathSupport
An example Model (TransMaths)
We hypothesized that:
Outcome of Year 1 (at University)=
Entry Qualification + Dispositions + Transitional experiences +
Background Variables
The resulting Linear Regression Model
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)
60.5085
2.4853 24.347 < 2e-16 ***
CombTransFeel
1.0621
0.5850
1.815
0.0708 .
maths[T.Yes]
-3.7296
1.9013 -1.962
0.0510 .
LPN
-1.0616
0.5650 -1.879
0.0615 .
MathCompCat
1.9893
0.3728
5.337 2.28e-07 ***
MSE_DP4
-1.2712
0.6849 -1.856
0.0647 .
--Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 11.72 on 228 degrees of freedom
(1544 observations deleted due to missingness)
Multiple R-squared: 0.1233,
Adjusted R-squared: 0.1041
F-statistic: 6.414 on 5 and 228 DF, p-value: 1.345e-05
Positive effect: Positive transitional experience, previous maths
qualification
Negative effect: Course, Low Participation Neighbourhood,
Mathematics Self Efficacy
CombTransFeel effect plot
maths effect plot
LPN effect plot
68
75
65
Year1Result
66
70
Year1Result
Year1Result
68
64
66
64
62
62
60
60
60
-2
0
2
4
6
No
Yes
CombTransFeel
maths
MathCompCat effect plot
MSE_DP4 effect plot
75
70
Year1Result
Year1Result
70
60
65
60
55
55
0
2
4
6
MathCompCat
8
-4
-2
0
2
3
LPN
75
65
1
2
MSE_DP4
4
6
4
5
Just a short summary
This is the quantitative aspect of mixed method project which
also includes case studies and interviews
The multi-step methodology described helps us to create and
validate our measures and then…
Use them to model (GLM- regression modeling) in order to
respond to the research questions we originally set…
For instance
What influences students successful transitions between various
stages of education (e.g. to University)?
How can we predict students’ progress at university) ?
Thank you
Q&A