A MULTIDIMENSIONAL APPROACH TO THE IDENTIFICATION OF TEST FAIRNESS

EXPLORATION OF THREE MULTIPLE-CHOICE SSC PAPERS IN PAKISTAN

Syed Muhammad Fahad Latifi [email protected]

Dr. Thomas Christie [email protected]

-Perspective of Test Fairness -Substantive / judgmental analysis - Statistical - Dimensionality of content.

Primary dimension Secondary dimension ( impact or bias) - Test item measuring dimension other than primary dimension is produce differential Item functioning (DIF).

- Bundle of Test items measuring dimension other than primary dimension produces differential bundle functioning (DBF).

 Multidimensionality of test item / bundle of items.

 DIF/DBF can be uniform and non-uniform.

 DIF / DBF can produce interesting explanations . e.g., it may be due to item format characteristics, subject matter related factors and cognitive skills measured on the test.

 Males are considered as reference group (majority group)  Females are considered as focal group (minority group)

RESULTS

Phase –one DIF

0.15

Favors Males 0.10

0.05

Beta-Uni

0.00

0 -0.05

Favors Females -0.10

-0.15

1 2 3 English Mathematics Physics One item in each subject were found with severe DIF. i.e. only 3.7% of the total item pool.

4

Phase –one DIF (cont.):

Three Severe DIF Items

Phase One-DIF (cont.):

Content and proficiency expectation for C-level DIF.

Item

14 11 4

Beta-Uni

0.100* 0.105* -0.150* * is Significant at p < 0.05.

Subject

English

Content Area

Reading Skills

Required proficiency Favors

Explore implicit meaning of a text.

Male Mathamatics Physics Equations and Graphs Plot the pairs of points to obtain the graph of a given linear Wave, Sound and Optics ultrasound.

Male Female

Three Severe DIF actual item text

Appendix A-2

Subject

English

Question

14. According to the passage, the construction of the Pisa tower began in A. the middle of the Field of Miracles.

B. white marble.

C. the twelfth century.

D. the period of Mussolini.

Content Area

Reading Skills

Mathematics

11. The pair of points which lie on the straight line (AB) ⃡ is (0,0) and A. (2,0) B. (3,-2) C. (-3,-3) D. (-2,2) Linear & Quadratic Equations, Inequalities and Graphs Physics 4. Ultrasound is used for different purposes, which of the following is NOT a current use of ultrasound?

A. Detection of fault in engine B. Measure the depth of an ocean C. Diagnosis of different diseases D. Ranging and detection of aeroplanes Waves, Sound and Optics

Phase Two-DBF

 Analogous to DIF, DBF is conceptualized as several DIF items acting in concert to produce an item bundle favoring matched examinees from one group over another, as judge by bundle score.  The term bundle indicate a set of items organized/ grouped together because they share a common content dimension, cognitive similarity or share a common item structure.  Four organizing principles are suggested in literature.

 1)

Test Specification

, 2) Content Analysis, 3) Psychological Analysis, and 4) Empirical Analysis.

 Test Specification as organizing principle is used in this study.

Phase Two-DBF (cont.) :

Phase Two-DBF (cont.) :

Results from Differential Bundle Functioning for SSC English, Mathematics and Physic, SSC May-2011 examinations.

Subjec t

English

Suspected bundle

Content area1:

Reading Skills.

Content area 2:

Listening Skills.

Mathematics

Content area 5:

Linear & Quadratics Equation, Inequalities and Graphs.

Content area 3 & 4:

Coordinate Geometry, Trignometry & Theorem; Fraction, Functions and Algebraic manipulation.

Content area 4:

Fraction, Functions and Algebraic manipulation.

Physics

Content area 6 & 7:

Electronics, ICT and Radioactivity; Electrostatics, Current and Magnetism.

Content area 8:

Wave, Sound and Optics

No. of items

12 12 9 20 4 17 7

Beta-Uni

-0.033

-0.181* -0.141

0.301

0.086* -0.249

0.132*

Favors

Females Females Females Males Males Females Males * is Significant at p < 0.05.

For each bundle analysis matching subtest was created by combining remaining items in the subject; items displaying C-level or larger DIF is ignored in all DBF analysis.

For the ease of reporting, each subtest within each subject is sequenced order in which they are presented in Figure 2. For example Listening Skills is called content area 1, Reading skills is called content area 2 and so on.

Phase Two-DBF (cont.) :

However, DBF is controversial due to amplification and cancellation effect.

The small item-level differences, which may go unnoticed, can be magnified when the same difference is evaluated with a bundle, also called DIF-amplification .

DIF cancellation is caused when the bundle of items exhibiting DIF against one group while another bundle of items exhibits DIF against the alternate group and therefore each is canceled out.

Authors of the

1999 Standards for Educational and Psychological Testing

state: “Although DIF procedures may hold some promise for improving test quality, there has been little progress in identifying the causes or substantive themes that characterize items exhibiting DIF.

That is, once items on a test have been statistically identified as functioning differently from one examinee group to another, it has been difficult to specify the reasons for the differential performance…

" (p. 78).

CONCLUSION

 The results of this study indicates that there were only three items with Level-C DIF in the AKU-EB`s SSC May 2011 English, Mathematics and Physics examinations.  DBF is controversial and has limited significance from practitioners’ perspective. Further, to-date , no guidelines exist to interpret the effect size measure for DBF and thus, research is needed to identify and evaluate effect size guidelines for interpreting differential bundle functioning.

 Taken together, the present study suggests that the small amount of DIF found does not confound the validity of the interpretation of the examinees’ test scores on SSC examination and likewise test development practices are fair for both male and females.