Assessing Tools that Measure Information Literacy (IL) Skills: Could Project SAILS Prove Useful for Law Libraries? AALL/Wolters Kluwer (Aspen) Grant –Funded Research Co-P.I.s: Molly Brownfield.
Download ReportTranscript Assessing Tools that Measure Information Literacy (IL) Skills: Could Project SAILS Prove Useful for Law Libraries? AALL/Wolters Kluwer (Aspen) Grant –Funded Research Co-P.I.s: Molly Brownfield.
Assessing Tools that Measure Information Literacy (IL) Skills: Could Project SAILS Prove Useful for Law Libraries? AALL/Wolters Kluwer (Aspen) Grant –Funded Research Co-P.I.s: Molly Brownfield & Dennis Kim-Prieto Rutgers Law Library – Newark Could SAILS Prove Useful? Assessing Assessment Tools? Assumption: Information Literacy (IL) is measurable. Therefore: 1. Instructional Law Librarians will want to measure IL. 2. Current state-of-the-art assessment is Project SAILS instrument: can it work for law libraries? 3. We think not: here’s why. Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? What is Project SAILS? It is an: • Item Response Theory (IRT)-based assessment tool, • targeting information literacy skills which are • based on ACRL Information Literacy Competency Standards for Higher Education. It has been funded and supported by: • • • • • Institute of Museum and Library Services, Association of Research Libraries, Ohio Board of Regents, Academic Library Association of Ohio, and Kent State University. Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? Why did we choose Project SAILS? • Ease of use: – SAILS uses graphical user interface (GUI) – accessible through standard internet browsers – SAILS performs all analyses • External validity: SAILS results positively correlate with subject SAT/ACT scores. • Price: $3.00 for each subject with a cap of $2,000.00 per total administration. • SAILS is explicitly keyed to measure ACRL’s IL standards. Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? What did we do with Project SAILS? • We administered the SAILS instrument to 28 1Ls and 29 3Ls (n=57). • Using law students in Human Subjects Research requires two things: – Institutional Review Board (IRB) approval & – CA$H!!! • IRB approval generally requires certification. • Your institution will have its own IRB, but rules tend to be standardized per federal research funding regulations. Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? What is IRT, and how does it work? Sample Question One You need to get information on an event that took place two days ago. Where are you most likely to find information about the event? CHOOSE ONE ANSWER • Book • Dissertation • Journal article • Magazine • Newspaper Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? What is IRT, and how does it work? Sample Question Two You need a case that addresses the following issues: employment discrimination and intentional infliction of emotional distress. Which boolean operator should you use? CHOOSE ONE ANSWER • And (&) • Near (within /n number of words) • Same Sentence ( /s) • Not (NOT) • Or (OR) Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? IRT against Classical Test Theory (CTT)? • CTT weighs both items the same, despite item differences in their level of difficulty. • Correct answers on Question 1 do not reveal much about subject IL, since most should get it right. • Correct answers on Question 2 are likely to reveal more, since not as many subjects will get it right. • CTT does not distinguish the value of each item, and weighs all items on a test the same. • IRT distinguishes item weights in analyzing responses, which reveals more about subjects’ IL. Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? Sample Questions from Project SAILS Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? Radcliff, Carolyn J., Joseph A Salem, Jr., Julie A. Gedeon, and Lisa G. O'Connor. 2007. “Project SAILS Sample Test Questions.” <https://www.projectsails.org/abouttest/samples.php>. Last Accessed 3 May 2008 Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? What did we find with Project SAILS? R-N Law Students outperformed institution-type benchmark on the following skill sets: • • • • • • • Developing a Research Strategy Selecting Finding Tools Searching Using Finding Tool Features Evaluating Sources Documenting Sources Understanding Economic, Legal, and Social Issues Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? What else did we find? Skill results for R-N Law Students in performance order: • • • • • • • • Using Finding Tool Features (Best Performance!) Retrieving Sources Documenting Sources Evaluating Sources Developing a Research Strategy Selecting Finding Tools Searching Understanding Economic, Legal, and Social Issues (Worst Performance?!) Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? Data from SAILS administration found … RU Law Students Students from Doctoral-granting Institutions Students from All Institutions Standard 1: Nature & Extent 655 ±26 584 ±1 576 ±1 Standard 2: Efficient Access 640 ±18 577 ±1 571 ±1 Standard 3: Evaluation & Incorporation 639 ±21 576 ±1 570 ±1 Standard 5: Socio-legal Issues 623 ±18 566 ±1 558 ±1 Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? No significant difference btw. 1L & 3L IL! SAILS Skill Sets RU 1L Scores RU 3L Scores Research Strategy Finding Tools Searching Finding Tool Feat. 643 ±31 629 ±56 628 ±36 683 ±57 643 ±35 643 ±45 631 ±35 715 ±63 Retrieving Sources Eval. Sources Doc’ting. Sources 667 ±67 641 ±29 662 ±37 674 ±61 667 ±35 674 ±39 Econ., Leg., Soc. Is. 625 ±32 624 ±29 Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? Our Observations • Our administration did not show any significant distinction between 1L and 3L SAILS performance; yet, • We know that 3Ls have had more training or exposure to legal research practices than 1Ls have. • As the current standard (SAILS) does not adequately reflect law student performance, • Accurate Law Student Information Literacy (LSIL) assessment requires a subject-specific tool. Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? Our Recommendations • As a cohort measure, and not an individual measure, SAILS data only assesses skill sets between and within institutions, and does not assess skill sets of particular students. • SAILS provides a useful model and valuable experience in assessing law students, but does not test the subject-specific research abilities that instructional law librarians teach. • We need a subject-specific IL assessment instrument if we want to measure LSIL. Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? What can academic law libraries do next? • Law libraries have a vested interest in measuring Law Student IL (LSIL). • Law schools and their libraries can contribute the following toward creating an adequate measure: – Subject pools, – Research support, and – A consensus on IL: what subject-specific skills make a law student more or less literate when evaluating information and performing research? Brownfield & Kim-Prieto Rutgers – Newark Could SAILS Prove Useful? Thank you for your interest! Questions? Principal Investigators: Molly Brownfield, Duke Law Library [email protected] Dennis Kim-Prieto, Rutgers-Newark Law Library [email protected] Brownfield & Kim-Prieto Rutgers – Newark