EVALUATING A TBLT SPANISH IMMERSION PROGRAM Greg Burwell – BPA – [email protected] Marta Gonzalez-Lloret –UHM – [email protected] Katie Nielson – UMD (CASL) – [email protected] COLLOQUIUM:
Download ReportTranscript EVALUATING A TBLT SPANISH IMMERSION PROGRAM Greg Burwell – BPA – [email protected] Marta Gonzalez-Lloret –UHM – [email protected] Katie Nielson – UMD (CASL) – [email protected] COLLOQUIUM:
EVALUATING A TBLT SPANISH IMMERSION PROGRAM Greg Burwell – BPA – [email protected] Marta Gonzalez-Lloret –UHM – [email protected] Katie Nielson – UMD (CASL) – [email protected] COLLOQUIUM: Evaluating task-based language programs Overview Background Student assessment Pilot study Ordinate study Student evaluation Discussion TBLT at the Border Patrol Academy (BPA) BPA Training Environment Spanish at the BPA (before and after) Students, instructors, and supervisors sought practical training Developed in 2006-2007 Needs Analysis Genuine audio/video Extensive input from actual Border Patrol agents TBLT at the Border Patrol Academy (BPA) First implemented in December of 2007 First cohort of 15 students graduated in March 2008 As of August 2009, 2,813 students have completed the program To date, the program has started 180 classes with over 3,000 students 100% pass rate TBLT at the BPA- Curriculum 8-week program following Long’s (1985, 2005a) stages for designing and implementing a TBLT program 7 target tasks Several target task types Multiple pedagogic tasks TBLT at the BPA - Curriculum Target Task Types: Extract subject Interview subject Determine citizenship status Inspect Vehicle TBLT at the BPA - Curriculum Output (guided) COMPLEXITY Input (video, audio) Role-play tasks + Lab Activities + La Chamba interactive video game TBLT at BPA- Assessment Weekly formative assessment: End-of-course summative assessment: • Computer-based language test • Performance-based assessment where trainees interact with native Spanish speaking role-players as they complete that module’s target task • One-on-one conference with instructor • • • • Critical scenarios with professional role-players Students role-play key course tasks over two days Criterion-referenced scoring Each scenario scored by two instructors TBLT at BPA - Assessment TBLT at BPA- Assessment Multiple versions of end-of-module assessments Questions considered in each iteration: Are these elements required to complete the task? Which are critical? How do we assess language? Global accuracy, fluency, and pronunciation ratings Inclusion of “using appropriate Spanish” with each required task Removal of any mention of language TBLT at BPA - Assessment TBLT at BPA - Assessment TBLT at BPA - Assessment TBLT at BPA - Assessment Instructor training Video evaluation Role-play evaluation Reliability Rubric modification Perceived grade inflation Language versus task accomplishment Non-obligatory steps removed Average grade changed from 94.10% to 95.48% Pilot Study – Oral narration task Participants Grammar-based group (GB) (n=20) TBLT group (TBLT) (n=20) Oral narrative based on picture story Hypotheses: • • TBLT will perform better on measures of fluency, lexical complexity and structural complexity GB will perform better on measures of grammatical accuracy Pilot study- cont Compared on: Fluency [syllables/second] (Gilabert, 2004) Lexical complexity [lexical words/words] (Robinson, 1995; Gilabert, 2004) [1st time use of lexical word/words] Ratio of lexical/functional words (Gilabert, 2004) Structural complexity [words/utterance] (Ortega, 1999) Grammatical accuracy TLU of noun-modifier agreement (Ortega, 1999) TLU of noun-verb agreement Pilot study- Results - Fluency 2.00 1.80 1.60 syllables/second 1.40 1.20 1.00 0.80 0.60 0.40 0.20 0.00 GB TBLT Pilot study- Results – Lexical complexity 1.20 1.00 0.80 GB 0.60 TBLT 0.40 0.20 0.00 Lexical complexity 1 Lexical words/words Lexical complexity 2 1st use lexical w/words Lexical complexity 3 Ratio of lexical/ functional words Pilot study- Results- Structural complexity 8.00 #of words/ utterance 7.00 6.00 5.00 GB 4.00 TBLT 3.00 2.00 1.00 0.00 GB TBLT Pilot study- Results –Grammatical Accuracy 1.00 0.90 0.80 0.70 0.60 GB 0.50 TBLT 0.40 0.30 0.20 0.10 0.00 Grammatical accuracy 1 -(S-N agrement) TLU of noun-modifier agreement Grammatical accuracy 2 (S-V agreement) TLU of noun-verb agreement Pilot study - Discussion Very small n-size due to difficulty obtaining samples Significant effect for fluency and structural complexity (.05) Students from TBLT course tend to produce more fluent and complex speech that is less grammatically accurate but contains a greater variety of lexical items. Ordinate study Participants First 256 TBLT students Performance on computerized oral proficiency interview Hypotheses: • • TBLT will improve students’ overall proficiency TBLT will affect overall proficiency regardless of language abilities at the start of the course Ordinate Study - continued Ordinate (Versant) Computerized Oral Proficiency Exam Used at BPA as a placement test Scored between 20 – 80 Overall Sentence Mastery Vocabulary Fluency Pronunciation Students took the test before and after the 8-week TBLT program Ordinate Study - Mean Scores 50 45 40 35 30 25 Pre-test Post-test 20 15 10 5 0 Overall Sentence Mastery Vocabulary Fluency Pronunciation Ordinate Study – Change Scores All change scores significant at .01 * Overall: Mean: 7.52 SD: 5.42 * Sentence mastery: Mean: 7.55 SD: 8.21 * Vocabulary: Mean: 7.16 SD: 10.99 * Fluency: Mean: 5.86 SD: 7.06 * Pronunciation: Mean 3.53 SD 4.92 Ordinate Study - Correlation Ordinate Study - Discussion Hypothesis #1 Overall proficiency will improve as a result of TBLT Yes: Post-test means improve an average of 7.5 points overall on an unrelated, standardized proficiency test Hypothesis #2 Overall proficiency will improve regardless of starting level Yes: Almost no correlation between pre-test score and change score Students’ evaluations Electronic (anonymous) surveys 2 Groups: Pre-experienced Current students of the TBLT Spanish program n=41 Experienced Current BA Agents trained with TBLT Spanish program n=14 Students’ evaluation (pre-experienced) Students’ evaluation In general students liked the best… The work done with the role-players. The direct application of the class to their job. “Role playing is very helpful. I feel that it helped me learn more spanish (sic) and become more comfortable speaking it.” “I enjoyed the hands on experience the most. Being able to read and write is important when learning a new language, but being able to incorporate it into your daily duties is beneficial.” “I have enjoyed that we get to have a lot of hands on in this program. Also helps a lot for us to be able to apply all that we have learned in basic and build apon (sic) basic.” Students’ evaluations Students suggestions: More conversational Spanish (not for the job) More grammar in class More role-play work with Spanish speakers Agents’ evaluation (experienced) The video game helped me learn important Spanish words &… I played the video game La chamba at least twice a month. The Spanish lessons reflect well what happens in the filed. The Spanish lessons gave me confidence to use Spanish in the filed. The lab exercises helped me learn Spanish. I apply most of what I learned in class in the field. The class exercises are now useful in the field. The vocabulary & expressions I studied are now useful in the field. The topics I studied are now useful in the field. I completed most of the lab exercises. The Spanish program (topics, activities, lab) was interesting. 0.00 1.00 2.00 3.00 Agents’ evaluations What they found most useful from the Spanish course was: Commands and expressions Knowing the procedures and forms in Spanish What they would have liked to learn: More conversation (outside scenarios) More vocabulary Grammar and verb conjugation TBLT at BPA Students are all passing Student speech is more fluent and complex than in the old course Overall student proficiency improves after the TBLT course Students find the course practical What does this mean? Difficulties with evaluating TBLT at the BPA Difficulty of training instructors properly on methods and principles of TBLT •Instructors reluctant to adopt new approach •Lack of trained personnel to collect and analyze data •Lack of support for follow-up research • THANK YOU GRACIAS MAHALO Questions?