Phil Ice - Intro to Analytics

Download Report

Transcript Phil Ice - Intro to Analytics

ODLAA

Data & Analytics: An Introduction for e-Learning Professionals

Phil Ice, Ed.D.

VP, Research and Development American Public University System [email protected]

Connections All analyses and stakeholders are interrelated

An Administrative Perspective

 Success and decision making are predicated on access to data  Understanding strengths and weaknesses is dependent on having access to all data within the institution  Data tells us what has happened and improves strategic planning moving forward

Institutional Assessment A D3M Culture

Organizational Capacity

 Data must have a “home”  Top down dissemination of analytics  Actionable reporting  In CONJUNCTION with other academic initiatives

Major Data Repositories

 Student Information System  Demographics  Institutional level transactions  Learning Management System  Learning transactions  Learning outcomes  Latent data  End of Course Survey  Perceptual data

Secondary Repositories (maybe)

 Student Services  Financial Aid  Faculty Records  Other – there’s always an “other”

Centralization

 Creation of a middleware database should be a priority for all institutions  SQL is a popular choice  Aggregate multiple data sources  Federation  Normalization

Levels of Analysis A range of approaches are required to satisfy stakeholder needs

STUDENT COUNT

STUDENT DEMOGRAPHICS

GRADES BY CORE COURSES

AMERICAN PUBLIC UNIVERSITY SYSTEM | EDUCATING THOSE WHO SERVE

Predictive Modeling Federation of multiple demographic and transactional data sets

UI / UX Visual Appeal and Ease of Navigation Data is NOT Enough

Multi-Institutional Initiative

 May 2011 – Bill and Melinda Gates Foundation provided a $1million grant to WCET  Six institutions aggregating data to look for trends in retention  Development of a POC to demonstrate multi-institutional federation and analysis  Development of a model for future work

EOC Survey Data

 End of course survey data is a very powerful tool for programatic improvement  Requires thorough understanding of student demographics  Demographic mix can alter outcomes  Multiple iterations of courses are possible if there are large variances in student characteristics  Remedial courses  Traditional vs. non-traditional learners

EOC Survey Problems

 The vast majority of online course surveys are derivative of face-to-face courses  Online surveys must account for unique pedagogies  Institutional inertia makes life difficult for instructional designers  ID efforts not measured effectively  ID / faculty roles are intertwined in most surveys

Measurement Needs

 Effectiveness of media and layout  Instructor role in discussion and interaction with students  Student interaction with other students  Effectiveness of activities  Cognitive engagement

Exemplar – CoI Framework

 A process model of learning in online and blended educational environments  Grounded in a collaborative constructivist view of higher education  Assumes effective online learning requires the development of a community of learners that supports meaningful inquiry and deep learning

Three Presences

Social Presence

 The ability of participants in a community of inquiry to project themselves socially and emotionally -- as ‘real’ people  The degree to which participants in computer mediated communication feel socially and emotionally connected

Cognitive Presence

 The extent to which learners are able to construct and confirm meaning through sustained reflection and discourse in a critical community of inquiry

Cognitive Presence - Elements

• • • •

triggering event (sense of puzzlement) exploration (sharing information & ideas) integration (connecting ideas) resolution (synthesizing & applying new ideas)

Teaching Presence

 The design, facilitation and direction of cognitive and social processes for the purpose of realizing personally meaningful and educationally worthwhile learning outcomes

Exemplar – CoI Framework

 9 social presence items (3 affective expression, 3 open communication, 3 group cohesion)  12 cognitive presence items (3 triggering, 3 exploration, 3 integration, 3 resolution)  13 teaching presence items (4 design & facilitation, 6 facilitation of discourse, 3 direct instruction)  Validated in 2007  Subsequent N > 1 million

Community of Inquiry Survey Instrument (draft v15) Developed by Ben Arbaugh, Marti Cleveland-Innes, Sebastian Diaz, Randy Garrison, Phil Ice, Jennifer Richardson, Peter Shea & Karen Swan Teaching Presence

Design & Organization

1. The instructor clearly communicated important course topi 2. The instructor clearly communicated important course goals.

3. The instructor provided clear instructions on how to participate in course learning activities.

4. The instructor clearly communicated important due dates/time frames for learning activities.

Facilitation of Discourse

5. The instructor was helpful in identifying areas of agreement and disagreement on course topics that helped me to learn.

6. The instructor was helpful in guiding the class towards understanding course topics in a way that helped me clarify my thinking.

7. The instructor helped to keep course participants engaged and participating in productive dialogue.

8. The instructor helped keep the course participants on task in a way that helped me to learn.

9. The instructor encouraged course participants to explore new concepts in this course.

10. Instructor actions reinforced the development of a sense of community among course participants.

Direct Instruction

11. The instructor helped to focus discussion on relevant issues in a way that helped me to learn.

12. The instructor provided feedback that helped me understand my strengths and weaknesses. 13. The instructor provided feedback in a timely fashion.

Social Presence

Affective Expression

14. Getting to know other course participants gave me a sense of belonging in the course.

15. I was able to form distinct impressions of some course participants.

16. Online or web-based communication is an excellent medium for social interaction.

Open communication

17. I felt comfortable conversing through the online medium.

18. I felt comfortable participating in the course discussions.

19. I felt comfortable interacting with other course participants.

Group cohesion

20. I felt comfortable disagreeing with other course participants while still maintaining a sense of trust.

21. I felt that my point of view was acknowledged by other course participants. 22. Online discussions help me to develop a sense of collaboration.

Cognitive Presence

Triggering Even

23. Problems posed increased my interest in course issues.

24. Course activities piqued my curiosity. 25. I felt motivated to explore content related questions.

Exploration

26. I utilized a variety of information sources to explore problems posed in this course. 27. Brainstorming and finding relevant information helped me resolve content related questions.

28. Discussing course content with my classmates was valuable in helping me appreciate different perspectives.

Integration

29. Combining new information helped me answer questions raised in course activities.

30. Learning activities helped me construct explanations/solutions.

31. Reflection on course content and discussions helped me understand fundamental concepts in this class.

Resolution

32. I can describe ways to test and apply the knowledge created in this course.

33. I have developed solutions to course problems that can be applied in practice.

34. I can apply the knowledge created in this course to my work or other non-class related activities.

Instructional Design 1

 CoI analysis informs optimal design strategy  Bifurcation of Teaching Presence is an indicator of need for review of “instructor voice”  Social Presence bifurcation is an indicator of the need for more collaboration  Strength of factor loadings can indicate areas where content review is needed

Instructional Design 2

 Socio-epistemological orientation – objectivist vs. constructivist  Two factor loading pattern indicative of an objectivist orientation  Constructivist paradigm important for those most impacted – 38-47 years of age

Faculty Improvement

 Informs strengths and weaknesses among individual faculty member  Used in conjunction with drop rates, grade distributions and third party observations  Informs support NOT punishment

Faculty Evaluation Combining descriptives, regression and factor analysis

Semantic Applications

 Content when developed, should consider the lowest level of granularity and highest level of reuse  Collaborative teams in traditional Instructional Design (ID) processes (ID, SME, Producer) can offer insights and additional meta-information that makes the system more self-aware  Enables future individualized and cohort learning profiles  Accelerate and insure the integrity of accreditation processes

Semantic Analysis Federation, Disaggregation, Relational Mapping, Ontological Ordering

GAP ANALYSIS REPORT

GAP ANALYSIS

The LMS Problem

 LMS’s have messy data bases  The primary function was not data collection  Years of additions have created the equivalent of a bowl of “data spaghetti”  Significant abstraction work is needed to draw out anything more than cursory data

Solutions

 Web analytics tools (Google Analytics, CoreMetrics, Omniture) are the future  Inserting Java code on pages and portions of pages  Highly granular transactional data can be derived  Not all web analytics tools are created equal

Platform Matters

Using Web Analytics

Converting Business to Education 5. Innovate: Experiment with new ideas to drive value A/B testing and multi screen delivery of individualized learning environments 4. Optimize: Test and enhance effectiveness Evaluate learning behaviors enhance content / pedagogical pathways Optimize Innovate Analyze 3. Analyze: Evaluate site performance Measure Report 1. Identify & Measure: Capture KPIs and other metrics Capture learner interactions and demographic information 2. Report: Generate reports on collected data

eLearning Optimization

Deliver reporting and dashboards to measure learning activity

Measure

Leverage these measurements to make informed decisions about how to create optimal learning experiences

Execute

Automate learner interactions and activities based on real time data, course goals and performance

Automate Extend

Bring together information from the LMS, SIS, and internet for a comprehensive view of customer interactions

Innovate

Find and apply new insights to optimize learning experiences based on a complete view of all interactions regardless of device Measure Optimize

Where are we headed?

Business Models Provide Guidance

Convergence

 Federation of Institutional Systems  Web Analytics  Quantification of Semantics  Round-Tripping Data Across the Enterprise  Multi-Institutional Comparisons  Programatic Globalization  QUESTIONS?