From Entrepreneurial to Enterprise IT Grows Up

Nate Baxley – ATLAS [email protected]

Rami Dass – ATLAS [email protected]

Learn@Illinois

• • A partnership between Colleges of LAS and Education Learning Management System (LMS) based on Moodle – – – – – Open Source Released in 2002 Currently at version 2.7

Over 80,000 installations worldwide http://moodle.org

Learn@Illinois A brief history

• • • • • Moodle came to LAS and Educaiton in 2003 – – http://courses.las.illinois.edu

http://learn.education.illinois.edu

Implemented on single machines 2010 – Sharing a single developer 2012 – Service was combined – http://learn.illinois.edu

Statistics (Fall 2014) – – 500 courses 26,000 enrollments 20,000 daily logins 23,000 unique users

Early Days Separate Installations

• • • • • A single web server in tandem with a separate database server Every year courses were copied manually along with a new database Rosters were managed with daily file uploads or registration keys sent out via email Code was pushed as needed with minimal testing Changes were often implemented as soon as they were ready

Early Days Separate Installations

(cont) • • • • Support was by email directly to the developers or service managers Monitoring was done through home grown script Logging was decentralized, each host kept logs locally

We lacked scalability and flexibility

Growing Up The Partnership Evolves

• • • • • Opportunity to design infrastructure for scalability and stability Code release procedures were tightened Support team was expanded Partnership became stronger Moodle 2.0 release coincided with partnership – Major code changes – Revisit old plugins

Growing Up Evolution of Infrastructure

• • • • • Scalable, flexible, and redundant Multiple web front ends were deployed behind a load balancer Offloaded scheduled task from the web front ends to a dedicated server Moved from a single individual to a team to better manage the rapid growth Transparent modifications to the systems with minimal interruptions to the users

Growing Up Dev & Release Process

• • • • • Improved reliability Thorough documentation of changes Reduced the frequency of releases Increased usage made ad hoc fixes less desirable Established a stricter dev/test/prod environment

Growing Up Dev & Release Process

(cont) • • • • Changes tracked in Redmine and bundled into releases Development team expanded to support other LMS CVS and GIT used for code versioning Code releases through direct file transfers

Today Infrastructure

• • • The database runs on a MySQL cluster with nodes spread across 3 data centers for high availability.

We have several web front ends spanned across multiple data centers The file server replicates to a “hot spare” server in another data center that can be manually switched to.

Today Infrastructure

(cont) • • • The framework is duplicated to varying degrees for the dev, test, and shadow systems Use Zabbix for monitoring, and get a lot of information on the health of the systems keeping us ahead of the curve Logging is centralized using Graylog to review the logs and event notifications

Today Application/Service

• • • • • Code releases are done through GIT 5 issue severity levels ranging from “emergency fix needed” to “annual major upgrades” Issues tracked from requirements gathering through testing and resolution Coordination of support team between partners Instructional designers at LAS, Education, and CITL help faculty design and build their courses

Future Growth

• • • • • Automatic failover for file server Implement configuration management to automate, standardize, and document infrastructure changes.

Future performance increases to handle growing enrollments. Dedicated "admin" web front end, tweaked to handle power user requests Moving beyond building maintenance tools to building teaching tools

Questions

• • Nate Baxley – College of LAS - ATLAS – – Client Relations Manager [email protected]

Rami Dass – – College of LAS – ATLAS Lead IT Infrastructure Engineer – [email protected]