The University of Melbourne 11 May 2015

Not just another diagnostic test!

Michael Jennings, Deb King (Melb), Adam Bridgeman (Syd), Liz Johnson (Deakin) Michael Jennings

Today’s talk is brought to you by…

… the Australian Government’s Office of Learning and Teaching.

Michael Jennings

B.Engineering Numbers

Yr

No.

’91

400

’01

600

’02

560

’03

530

’04

520

’05

570

’06

750

Yr

No.

’07

850

’09

970

’10

1000

’11

1100

‘12

1200

’13

1200

’14

1200 Table 1. Bachelor of Engineering first-year numbers (UQ enrolment data)

Prereqs

: 1991 - English, Maths 1 (Maths B), Maths 2 (Maths C), Physics, Chemistry 1995 onwards - English, Intermediate Maths (Maths B), Physics

OR

Chemistry Approx. 70% of students come in with both Inter. & Advanced Maths (B

&

C) Michael Jennings

History of diagnostic testing at UQ

Leo Howard Test – 1972 to1994 Paper test – 2007 Online test – 2009 to present Science diagnostic test – 2015

Michael Jennings

Current Engineering test – 2009 to present

• UQ T&L grant – cross-discipline team of Engineering & Science (inc. Maths) • Background questions • F-10 maths & Senior Intermediate Maths • Physics • Chemistry

Michael Jennings

Current Engineering test – OLT grant

Led by USQ, involving UQ, UNE, UTS, and UNewcastle • http://getset.uqcloud.net

20 maths questions –18 MCQs, 2 enter the answer – Can’t remember & never seen before options 12 physics and 12 chemistry questions Michael Jennings

Aim of the engineering test

• Is NOT to tell students how much they don’t know / forgotten • Is to gauge current mathematical / science understanding • Is to tell students what maths/science is used in their courses so they can choose the appropriate courses –

feedback matrix

• Is to identify at-risk students Michael Jennings

Current Science test – OLT extension grant – 2014/2015

Led by UQ, involving Melbourne, Sydney and Deakin • https://getset.science.uq.edu.au

19 maths/statistics questions – all MCQ – Can’t remember & never seen before options Michael Jennings

Results – engineering test

• Consistent over the years • Students who have done both Intermediate and Advanced Maths (Maths B & C) at school do better • Senior topics not done as well as junior ones • Differentiation and integration • Correlation between test results, ATAR (OP) and success in first-year engineering Michael Jennings

Engineering students’ thoughts

Benefit from doing quiz?

It gave me insight into the knowledge I need for 1st year It flagged some things I have forgotten and need to revise I feel this is the beginning of my learning journey at uni

% response

60 82 29 It made me feel that my needs were being assessed No, it achieved none of the above 29 2 Michael Jennings

Are you glad you did it?

Yes, I have a better idea of what I know and don't know Yes, I have a better understanding of 1st yr knowledge expectations Yes, I feel more confident I’m not glad I did it at all

%

73 62 26 4

What actions will you take as a result?

Revise my high school notes Find a relevant text book Ask someone who knows for help Do nothing as I'll pick it up during semester Do nothing because I did well

%

59 40 25 17 8 Michael Jennings

What did you dislike about doing the quiz?

I haven't disliked doing it at all I forgot a lot of what I knew during the holidays It has scared me It was stressful It didn't help It was time consuming and boring I was still in a holiday mood

%

38 43 11 6 2 8 35 Kavanagh, L., O’Moore, L, & Samuelowicz, K. (2009). Characterising the first year cohort knowledge. Proceedings of the AAEE 2009 conference.

Burton, L., Dowling, D., Kavanagh, L., O’Moore, L, & Wilkes, J. (2012). Examining First Year Students’ Preparedness for Studying Engineering. Proceedings of the AAEE 2012 conference.

Michael Jennings

BSc student’s feedback Semester 1, 2015 Yes, it gave me an insight into the knowledge that I need for 1st year Yes, it flagged some things that I have forgotten and need to review Yes, I feel that this is the beginning of my journey of studying science at UQ Yes, it made me feel that my needs are being assessed No, it achieved none of the above I'm not glad that I did completed the quiz I have a better idea of what I know and what I don't know I feel more confident I have a better understanding of first year knowledge expectations Revise my high school notes Find a relevant text book Ask someone who knows for help Do nothing because I did well Do nothing because I will pick it up during semester N/A - I didn't dislike it It was stressful It didn't help I forgot a lot of what I knew during the holidays It scared me I was still in a holiday mode It was time consuming and boring 60% 79% 30% 25% 3% 8% 73% 24% 58% 70% 30% 24% 12% 12% 41% 8% 4% 44% 16% 37% 6% Michael Jennings

2009 Michael Jennings

Results

2012 Michael Jennings

2013 Question No Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16 Q17 Q18 Q19 Q20 Q21 Q22 Q23 Q24 Q25 Concept Add two numerical fractions Add two algebraic fractions Solve an equation Expand a factorised quadratic Factorise a quadratic Solve a quadratic Solving simultaneous equations x= y= Simplify an expression with powers Understanding sine, cos, and tan Straight line equations What is the slope of the line?

What is the y intercept?

Simple derivative Optimising a function Visualising a derivative Adding different base logarithms Derivatives - Product rule Derivatives - Chain rule Simple integral Definite integral Calculate the area under graph Composition of functions Correct Answers % 97.54

77.05

92.87

89.47

91.95

83.76

70.04

71.98

76.46

84.89

88.99

78.52

92.72

77.67

65.22

64.1

49.38

73.45

89.66

64.81

76.6

64.62

Michael Jennings

Feedback matrix

• https://getset.science.uq.edu.au

Michael Jennings

OLT Extension Grant 2014-2015

• Melbourne, Sydney, Deakin science, maths and business programmes • UQ Science • Macquarie • Southern Cross, Charles Sturt, UWS, Curtin, Griffith, USQ, UTS, UNSW Michael Jennings

GetSet 2 – what’s new?

Core questions AND options CORE Option 1 Option 2 Option 3 Michael Jennings

You want GetSet? You got it!

However, you need: • TIME • $$ UQ Science 2015 • $43,000 grant – $8000 per discipline

Michael Jennings

What maths is in your course(s)?

What do you want to know? – Procedural vs conceptual understanding.

With what do you think students struggle?

When are you going to do the test?

What are you going to do with the info from the test?

Post-test? End of first-year test? Major test?

QS in Science –

www.qsinscience.com.au

Michael Jennings

Acknowledgements The 2009 online diagnostic test was funded by a The University of Queensland Teaching and Learning grant.

The project team was A/Prof Lydia Kavanagh (Faculty of Engineering, Architecture and Information Technology), Dr Liza O’Moore (School of Civil Engineering), Professor Peter Halley (School of Chemical Engineering), Professor Paul Lant (School of Chemical Engineering), Professor Peter Adams (Faculty of Science), Michael Jennings (School of Mathematics and Physics), Professor Lawrie Gahan (School of Chemistry & Molecular Biosciences) and Dr Anton Raynor (School of Mathematics and Physics), .

The OLT funded project involved USQ, UQ, UNE, UTS and UNewcastle.

http://www.olt.gov.au/project-get-set-success-using online-self-assessments-motivate-first-year-engineering-students engag

The feedback matrix

was designed by A/Prof Lydia Kavanagh (Faculty of Engineering, Architecture and Information Technology), Dr Liza O’Moore (School of Civil Engineering).

Michael Jennings

References

Burton, Lorelle, Dowling, David, Kavanagh, Lydia, O'Moore, Liza & Wilkes, Janelle (2012). Examining first year students' preparedness for studying engineering. In: Llewellyn Mann and Scott Daniel, AAEE 2012 Conference Proceedings.

23rd Annual Conference of the Australasian Association of Engineering Education (AAEE 2012)

, Melbourne, Australia, (1020-1028). 3 - 5 December 2012.

Jennings, M. (2011). The transition from high school to university: The University of Queensland perspective. In: John Hannah, Mike Thomas and Louise Sheryn, Te Ara Mokoroa: The Long Abiding Path of Knowledge - Proceedings of Volcanic Delta 2011.

Volcanic Delta 2011: The Eighth Southern Hemisphere Conference on Teaching and Learning Undergraduate Mathematics and Statistics

, Rotorua, New Zealand, (139-149). 27 November - 2 December 2011. Kavanagh, L., O’Moore, L., & Samuelowicz, K. (2009). Characterising the first year cohort knowledge.

2009 Conference Proceedings. Annual Conference of the Australasian Association of Engineering Education (AAEE 2009)

.

Michael Jennings