Transcript eXplorance Blue Presentation

Current status of Blue
 Implementation of Blue is effective since October
 Presentations to Colleges/Schools and Departments
are currently ongoing
 The Electronic Course Assessment Implementation
(ECAI) committee (subcommittee of the FDAI
committee) is supervising the implementation, in
conjunction with the Provost’s Office and OIT
 ECAI is the point of contact for the Faculty Senate
with regard to all issues about electronic evaluation
 You can find information about online course
evaluation at: https://www.uaf.edu/provost/blue/
Adoption of Blue
 What are the implications
 What could be done to improve confidence on
the survey for each class
Implications of adopting Blue
Inter-system comparison
 Consistency of evaluation over time
 Quality
Intra-system comparison
 Representativeness
 Accuracy
 Quality
Inter-system comparison
 Q: How my evaluation in Blue measures against IAS?
 Qualitatively, the two questionnaire are different (no
“effectiveness of teaching” is assessed).
 Quantitatively, the scores are expected to be
different because the structure and the content of
the surveys are different.
 Each Blue survey should be compared to Campus-
wide Blue aggregate data.
 Few rounds of evaluation will be needed to be able
to make comparison with IAS, if needed.
 Revise unit criteria for tenure and promotion.
Inter-system comparison: quality
 Q: Students who are strongly negative about the course or
the instructor have been the most likely group to complete
the online evaluation.
 Results from many studies on this topic have proven this to be a
misconception, with results from online evaluations shown to be
as trustworthy as those from paper-based evaluations (Liu, 2005;
Thorpe, 2002; Johnson, 2002).
 A large scale study of the results of the Individual Development
and Educational Assessment (IDEA) student rating system
between 2002 and 2008 (Benton et al., October, 2010)
examined a total of 651,587 classes that used paper-based
evaluations and 53,000 classes that used web-based
evaluations. This comparison showed no meaningful differences
between survey methods.
Intra-system comparison:
representativeness
 Q: Those who have responded to the survey
have very different views than those who have
not, hence the results from the survey would not
reflect the opinion of the population as a whole.
 The link between response rate and non-response
bias has not been established (Marketing Research
and Intelligence Association, October 2003 and
2011; Curtin et al., 2000, Langer, 2003; Holbrook et
al., 2005).
Intra-system comparison:
accuracy
 Q: Small sample size leads to greater margin of
error of the results.
 Depends on the class size and the opinions’
skewness
What can be done to increase
confidence toward this system
 Response analysis will reveal biases and potential
correlation with the polled cohort.
 Results of the analysis will be shared with students
and instructors.
 If the wording and/or the content of one or
more questions appear to skew quality of
responses, those questions will be re-evaluated
and re-worded or eliminated/substituted.
What can be done to increase
confidence toward this system
 Response rates:
 Showing evaluation matters
 Communication
 Making it easy for students
 Providing incentives
 UAF evaluation portal: www.uaf.edu/inspireus
 Intensify the use of Blue:
 Mid-term evaluation
 Department-specific questions