Transcript Student-owned computing, faculty development and the role

Student-owned computing,
faculty development and the
role of assessment
C. Dianne Raubenheimer
Rebecca Brent &
Amy Craig
The beginning
 Project initiated in 2001 with compact
planning funds.
 Pilot project ran from F 2001 – Sp
2005.
 Started small with 1 or 2 sections scaled up over time to +/- 25 sections.
 Developed appropriate support
mechanisms for the project.
Goals’ from Dean’s retreat, fall
2003
 Encourage students to purchase
laptops (expectation vs requirement).
 Develop infrastructure to support
technology in classrooms.
 Provide faculty training in the use of
technology and laptops.
 Each department to identify courses
in which to use laptops.
Laptop expectation
College of Engineering Incoming Freshman Survey 2006
Percentage of students bringing computers
Percentage of students bringing laptops
100
98.5
95.9
90
90
80
70
98
97.4
98.7
93
91.6
85
81.2
75
70.2
Percentage
60
50
49.2
40
38.6
30
29.1
Pilot Program F 2001 – Sp 2006
20
10
0
1998
1999
2000
2001
2002
2003
2004
2005
2006
Assessment
 Pilot program assessed over several years.
 The evaluation focused on 7 program
objectives, including (a) the impact of
teaching with wireless technology on
student performance, and (b) the impact of
teaching with wireless technology on
faculty workload, pedagogy and amount of
material delivered.
 The detailed report By Dr Joni Spurlin can
be found at
http://www.eos.ncsu.edu/soc/assessment/
Some early assessment results:
Student learning
 Enhanced Problem Solving - GC120, CSC
116, and MA141/241 used the laptop
program to integrate lab and lecture
sections of the course.
Assessment results show that student had
significant gains over non-laptop sections in
regard to:




the visualization of the course content,
several dimensions of problem solving,
graphics and computer-aided drawing,
programming was improved in several sections.
Some early assessment results:
Student attitudes
 85% felt laptops make learning more enjoyable.
 84% felt in-class use of instructional technology
stimulated learning.
 84% indicated the E115 experience increased comfort
in using their laptop.
 99% prefer working on their own computer instead of
in a computer lab.
 96% say laptop gave them freedom to work any
place, any time.
Some early assessment results:
Faculty attitudes
 Over the four year period, 57% said pace
had increased; 43% said variety
increased and 61% said course depth
increased.
 66% of the faculty agreed that students
were more involved in learning in their
laptop course
 20 out of 32 (62%) faculty said preparation
time for the laptop section was increased
because of modifications in pedagogy.
Faculty development
 Spring 2005, consultant (Rebecca Brent) brought in
to work with faculty on developing interactive
approaches to laptop use in the classroom.
 These were early adopters of technology.
 Sample projects
 Using short computer activities in nearly every
class to reinforce lecture material
 Developing spreadsheet simulation of
wastewater treatment plant
 Developing software proficiency test
 Teaching Excel with Visual Basic stressing
problem-solving algorithms
Results
Percentage of Students
Learning Benefits of Laptop Use
60%
50%
40%
Enhanced learning
30%
Enhanced computing
skills
20%
10%
0%
SA
A
N
N=220
D
SD
Some early lessons
 Effective use of laptops or classroom
computers takes time to learn.
 Faculty need help with classroom
management strategies (particularly offtask behavior).
 Renewed enthusiasm for teaching.
 Faculty support (from experienced peers,
pedagogical consultants, graduate
students) is vital for genuine course
transformation.
 Offer ongoing faculty development for
pedagogical aspects of computer use.
More lessons
 From the initial assessment and faculty
development processes, we learned that
assessing the impact of technology on
student learning needed to wait until
faculty had further developed their courses
and integrating the technology.
 First needed to assist faculty with HOW to
use technology and modify their pedagogy.
 When faculty work together on developing
curriculum and course materials, they can
have a positive effect on those around
them.
Reviewing the Goals – Sp 2006
 Encourage students to purchase laptops
- Satisfactory Progress
 Develop infrastructure to support
technology in classrooms - Satisfactory
Progress
 Provide faculty training in the use of
technology and laptops - Some Progress
 Each department to identify courses in
which to use laptops – Little Progress
Generating new goals
 New goals built on lessons learned and
previous assessment data, which highlighted
the need for
 Working strategically in departments
 Working with groups of faculty
 Focusing across the curriculum
 Assessing the impact on student learning
 Important to align program goals with
assessment processes with from the get-go.
 Assessment data should be used to facilitate
development processes.
New Goals – Fall 2006
 Goal 1: Engage students in program
decision making
 Goal 2: Disseminate materials and
information to faculty and students
 Goal 3: Create a more interactive and
informative website
Goal 4: Develop a framework for
use of computational tools within
departments
Specific
Outcome


Increase faculty
awareness of
the potential for
using laptops for
in class student
activities.
Develop a
sequence of
laptop
activities/modul
es for chosen
courses in each
department.
Strategies for
implementing
outcome
Assessment
methods

Present information
about the role of
laptops in instruction
to faculty in two
departments.

Baseline survey of
faculty use and
perceptions in two
departments (CBE
& ISE).

Work with HODs,
faculty, and teams of
graduate students.

Examine
syllabuses (pre
and post
implementation).
Case record report
of developmental
stages.

Goal 4 contd.
Specific
Outcome


Strategies for
implementing
outcome
Increase the
number of
faculty using
laptops for in
class student
activities

Increase
faculty
perception of
the value of
technology in
teaching.

Assessment
methods
Work with
individual faculty in
two departments to
incorporate laptops
into their
classrooms.

Work with
individual faculty in
two departments to
incorporate laptops
into their
classrooms.



Gather baseline data
on the number of
faculty using laptops
in their instruction.
Review how many
integrate laptops
after the project.
Baseline survey on
the value of laptops
in the classroom.
Follow-up surveys
after working with
faculty.
Goal 4 contd.
Specific
Outcome


Increase faculty
satisfaction
with teaching
with technology
and laptops.
Disseminate
project findings
to other
departments.
Develop a
framework for
laptop
integration
implementation
Strategies for
implementing
outcome


Work with individual
faculty in two
departments to
incorporate laptops
into their
classrooms.
Information sessions
at various HOD and
faculty forums.
Assessment
methods


Conduct in-depth
interviews with
faculty involved in
the project about
their experiences.
More departments
express interest
in laptops and
become involved
in successive
project cycles.
Goal 5: Create an interdisciplinary
engineering faculty community
Specific
Outcome

Create an
interdisciplinar
y engineering
faculty
learning
community.
Strategies for
implementing
outcome

Faculty are supported
by SOC in an FLC(s).
Assessment
methods




Faculty learning
community goals
inventory (presurvey).
Survey and faculty
reflections
focusing on the
specific FLC goals
& processes.
Surveys of specific
events.
Student learning
survey for faculty.
Goal 5 contd.
Specific
Outcome

Increase
number of
faculty engaged
in the
scholarship of
teaching and
learning
(SOTL).
Strategies for
implementing
outcome

Through the FLC,
and with support
from the director of
assessment, faculty
engage in research
on teaching and
learning.
Assessment
methods


Number of faculty
who attend
conferences and
make
presentations to
peers on
engineering
education
Catalog best
practices in
pedagogy tools
database created
by LITRE.
Goal 6: Assess the impact of
laptops on student learning
Specific
Outcome

Strategies for
implementing
outcome
Enhance

student
learning
through the use
of in-class
laptop
activities.
Assessment
methods
Work with individual 
faculty in the
project to develop
processes to assess
student learning
outcomes tied to the
use of technology. .

Various classroom
assessment
techniques
(CATs), tests,
projects and
rubrics.
Where possible,
compare student
performance with
previous
iterations of the
course.
Goal 6 contd.
Specific
Outcome

Strategies for
implementing
outcome
Increase student 
satisfaction with
the use of
laptops during
instruction.
Faculty use and
develop ways for
students to use
laptops in class.
Assessment
methods




Conduct baseline
survey of student
satisfaction with
use of laptops in
instruction.
Survey students
after faculty use
laptops for in class
instruction.
Focus group
interviews.
Comparisons of
GSS satisfaction
data.
Discussion time
 In pairs or groups – discuss goals and
assessment processes.
Goal 4: Engaging departments
 Replace traditional programming course.
 Focus on modeling, visualization and problem
solving in the specific engineering discipline.
 Use technology & computational tools for modeling
and solving real world problems (e.g. Excel and VBA
for programming).
 Change teaching style – e.g. integrating lab and
lecture; in class labs, homework labs and projects.
 Encourage curriculum articulation across a number
of courses in a department.
 Faculty engagement within the department to
discuss best practices and curriculum integration.
Scale-up and Scale-out
•
•
Curriculum integration with upper division courses
(scale-up).
Expansion to new departments.
ISE/TE 110
 Introductory course development and
implementation Spring 2006 –
worked with faculty development
consultant.
 Assessment of student perceptions of
teaching and learning (3 iterations of
data gathered to date).
Some assessment results: Student
confidence with programs
TE 110 F2006 N = 24
Rate your confidence with Excel
(1 = not confident,, 2= somewhat
confident,, 3 = confident,,
4 = very confident
Mean
TE 110 Sp 2007 N = 20
1
2
3
4
Mean
1
2
3
4
5. Entering values and formula
3.79
0.00
4.17
12.50
83.33
3.85
0.00
0.00
15.00
85.00
7. Using goal seek
2.50
16.67
29.17
41.67
12.50
2.80
5.00
30.00
45.00
20.00
13. Using Named Ranges
3.83
0.00
0.00
16.67
83.33
3.95
0.00
0.00
5.00
95.00
1
2
3
4
1
2
3
4
Rate your confidence with Excel
(1 = not confident,, 2= somewhat
confident,, 3 = confident,,
4 = very confident
Mean
Mean
14. Recording macros
3.83
0.00
0.00
16.67
83.33
3.95
0.00
0.00
5.00
95.00
19. Creating loops
2.83
20.83
16.67
20.83
41.67
3.25
15.00
5.00
20.00
60.00
22. Writing event handlers
2.71
16.67
25.00
29.17
29.17
3.15
5.00
15.00
40.00
40.00
Using laptops in class, students learn sufficient computing capacity.
Faculty use data to improve instruction in successive iterations.
Some assessment results:
Effectiveness of the In-Class Labs
ISE
Rate your opinions about
the class
29. I prefer lecture and
laboratory to be integrated
together, so that I can
practice what we learned in
lectures at the same time.
Strongly
disagree
Disagree
TE
Agree
Strongly
agree
Strongly
disagree
Disagree
Agree
Strongly
agree
0.00
7.14
42.86
50.00
4.17
0.00
29.17
66.67
34. I would prefer to listen
to the instructor rather than
have hands-on computer
activities in class.
33.33
62.96
3.70
0.00
34.78
56.52
4.35
4.35
43. I prefer to work through
the labs on my own, then
have the instructor go
through them.
15.38
57.69
19.23
7.69
25.00
62.50
8.33
4.17
0.00
3.57
67.86
28.57
4.17
4.17
66.67
25.00
45. I like the step by step
process of the labs
Students were engaged in active learning.
Some assessment results:
Effectiveness of Teaching Assistants
Rate your opinions about the
class
IE
Section
Strongly Disgree
disagree
42. Having the TA in class help w ith m y
learning.
TA training is essential.
0.00
0.00
Agree
42.86
Strongly
agree
57.14
TE
Section
Strongly Disgree
disagree
9.52
19.05
Agree
61.90
Strongly
agree
9.52
Assessment results: Comments
 Similarity of student responses in both
sections taught independently by the 2 with
different teaching styles instructors in
different departments.
 Close collaboration among faculty in
teaching and developing a course results in
an enriched curriculum.
 Faculty learned from each other, e.g. use of
tablet PC.
 Assessment results led to revisions in the
way course was taught in subsequent
iterations.
The ripple effect: Scale-up
 Assessment results discussed with other
faculty, who have embraced the level 100
course.
 Faculty at other levels of in the curriculum
(300 & 400) have started collaborating to
extend students’ skills across a sequence of
upper level courses.
 Baseline assessment will be done before
these courses are taught.
 There is an emerging culture of assessment
in the departments for curriculum
transformation.
Expanding to a new department
 In CBE many faculty teaching the same
course in different semesters, each with
different emphases and software tools.
 Large department, with inconsistent faculty
interaction.
 Support of departmental head for
curriculum change.
 Faculty development consultant worked
with group in spring 2007.
Use of baseline data
 Baseline survey of students’ perceptions of
competence in introductory 200 level course,
and 5 level 300 courses.
 Results showed no difference between
entering 200 and 300 level students in all
sections - that students lacked important
computational and problems solving skills
needed for the level 300 courses.
 Faculty recognized a bottle neck existed.
 Opened the space for a conversation for
faculty teaching the level 200 course.
Course revised
 Introductory course revised by Spring
2007 instructor.
 Assessment data showed significant
improvement against baseline in terms
of student’s self-reported skills.
 Significant improvement in attitude to
the use of laptops in class.
Some assessment results
1 = strongly disagree, 2 =
disagree, 3 = agree, 4 =
strongly agree
Pre-test Sp 07 CHE 225
1
2
3
4
Post-test Sp 07 CHE 225
Mean
1
2
3
4
Mean
Pvalue
Rate your opinion on the
following
I would prefer to be able to use a
laptop or computer in class rather
than just listen to the instructor.
2.38
33.33
45.24
19.05
2.81
3.13
6.25
56.25
34.38
3.22
>.02
Using a computer in class helps to
keep me focused on the work.
11.36
38.64
38.64
11.36
2.50
3.57
10.71
64.29
21.43
3.04
>.007
8.51
55.32
31.91
4.26
2.32
21.2
1
63.64
15.15
0.00
1.94
>.012
If I have a laptop or computer in
class I easily get distracted (e.g.
send email messages, IM).
Students engaged in active learning and computers were not a
distraction.
Current developments CBE
 Summer work group developing materials
and student workbook (CD and student
manual) for future iterations of the course.
 All faculty cycle through lower division
courses – this may increase their
predisposition to change other courses
(diffusion).
 Providing TA assistance is essential to
freeing up faculty time.
Next assessment steps
 Baseline assessment of student problem
solving abilities entering 100/200 level
courses.
 Re-assess mid-term and at the end of the
semester.
 Track student’s problem solving abilities
into subsequent semesters – longitudinal
plan.
 Correlate student perceptions of their
skills/particular outcomes with actual
student work of the same outcome.
Next implementation steps
Continue to use assessment data to inform the scale-up and
scale out processes, as done in the first 3 departments.
Questions or comments