Transcript Preliminary Troubleshooting Techniques for Blackboard
A System Administrator’s Guide to Preliminary Troubleshooting Techniques
Presented By: Darryl Wright and Volker Kleinschmidt April 12, 2005
And now a word from our lawyers…
• Any statements in this presentation about future expectations, plans and prospects for Blackboard and other statements containing the words "believes," "anticipates," "plans," "expects," "will," and similar expressions, constitute forward-looking statements within the meaning of The Private Securities Litigation Reform Act of 1995. Actual results may differ materially from those indicated by such forward-looking statements as a result of various important factors, including the factors discussed in the "Risk Factors" section of our most recent 10-K filed with the SEC. In addition, the forward-looking statements included in this press release represent the Company's views as of April 11, 2005. The Company anticipates that subsequent events and developments will cause the Company's views to change. However, while the Company may elect to update these forward-looking statements at some point in the future, the Company specifically disclaims any obligation to do so. These forward-looking statements should not be relied upon as representing the Company's views as of any date subsequent to April 11, 2005. Blackboard, in its sole discretion, may delay or cancel the release of any product or functionality described in this presentation.
Objectives
- Provide Blackboard ® System Administrators with an overview of their roles, responsibilities, and their impact on the success of the Blackboard Academic Suite™ at their institutions - Highlight best practices for keeping your Blackboard Academic Suite running smoothly and problem free - Improve and standardize the troubleshooting process on campus. Methods for better troubleshooting.
- Describe ticket submission and escalation process within Blackboard - Open Discussion: The evolution of your Blackboard Networked Learning Environment™
System Administrator Roles I
• • •
Back-end administrator
Installations; webserver admin; database configuration and maintenance; application monitoring; snapshot and data integration; and even OS maintenance tasks
Front-end GUI administrator
Course shell creation; user enrollment; export/import/copy; control over feature availability and design; set policies and business rules for account creation, content preservation etc.; portal administration
System Extension Developer
Creation of Building Blocks, Application Bridges and Portal Modules using Java and Blackboard’s APIs
System Administrator Roles II
• • •
CMS Help Desk Manager
Troubleshooting issues reported via email, phone, or a ticket system, or self-discovered; escalation of issues to Blackboard Technical Support; training and supervision of general help desk staff
Trainer & Documentation Author
One-on-one training and seminars for faculty, staff, and students; creating online documentation and tip sheets
Instructional Designer
In some cases the system administrator may also train instructors and staff on pedagogically sound uses of technology, and participate in the creation of course materials and the tracking and evaluation of student performance
Show of hands
Which roles do
you
fill at your institution? • Back-end administrator • Front-end administrator • System Extension Developer • CMS Helpdesk manager • Trainer / Documentation Author • Instructional Designer
User community mailing lists
• Back-end administrator • Front-end administrator • System Extension Developer • CMS Helpdesk manager • Trainer / Documentation Author • Instructional Designer • BBADMIN-L • BLKBRD-L • BB-OPEN_SRC • BB-FS • BB-ID
Evolution of System Administration Roles & Practices
EXPLORATORY SUPPORTED STRATEGIC MISSION CRITICAL TRANSFORMATIVE: THE NETWORKED LEARNING ENVIRONMENT
Business Continuity Strategic Owner Single Server Single Admin
Phase I
Multiple People Help Desk
Phase 2
Identified system manager Integrated Production Environment Change management practices System Update Deployment Plans Student /Faculty Training Programs Monitoring, Backup & Maintenance Practices Ticketing System
Phase 3 Phase 4
High Performance / High Availability Help Desk Management Long Range Planning Custom Instructional Tools
Phase 5
Best Practices
• Be Prepared • Avoiding is better than solving • Have a well-established troubleshooting process on campus • Train your users to be aware of pitfalls • Document local policies and differences • Top 10 tips for System Administrators
Best practices - Top 10 tips for System Administrators
1. Maintain current Operations Workbook 2. Document all server and application changes 3. Maintain a Test server environment 4. Set up and test a Disaster Recovery Plan 5. Plan ahead for increased usage 6. Train users well to avoid problems 7. Keep it clean - purge old records 8. Manage and monitor your application logs 9. Graph system metrics (MRTG, cricket) 10.Participate in user community
Network diagram (workbook)
• Use Visio templates or simple flowcharting tool • Make sure to label machines with function, hostname and IP-address • Indicate ports used on each connection • Include servers, firewalls, routers, proxies, load balancers • Keep it up-to-date
Disaster Recovery Plan
• Ready adequate replacement hardware • Set up full daily backups to it • Test recovery regularly • Train multiple staff on recovery • Document
all
necessary steps in standard location (Workbook, Backup/Recovery guide, local customizations) • Don’t commit replacement system to other production-grade tasks
Sample performance graphs
CPU-busy stats weekly graph dual weekly graph for total number of logins per day and number of recent logins (last 10 mins)
Part II
Troubleshooting
Troubleshooting Process on Campus
The SOAP Method
- A Basic Troubleshooting Method
Subjective
- Who?, What?, Where? and When?
Objective
- Steps to recreate, Error messages, log collection and access information
Assessment
- Review steps, errors, logs and assess whether it can be resolved or has to be escalated
Plan
- Resolution steps or Workaround
Examples of SOAP interaction:
- Used by the Helpdesk- troubleshooting end-user issues - Used by the System Administrator - recognizing, analyzing, reporting system-issues, focused collection of info for escalation
SOAP at the Helpdesk
Troubleshooting User-Reported Issues
• User- or course-specific issue?
– if so, need exact steps and location • Browser-specific?
– test on different machines and browsers • Recent changes?
– OS or software update, configuration change, malware infection • Restore known-good configuration – system restore, spyware removal, virus scan
Example:
Resolving a login problem
• Account exists?
• Correct password used? Caps?
• If campus-wide pwd, does it work elsewhere?
• Correct URL used?
• Machine-specific?
• Browser configuration (SSL, JS, cache) • Spyware and virus check • Create account, reset password, clean machine
SOAP for the GUI admin
• Word-of-mouth vs. direct report • Issue affects multiple users?
• Issue repeatable?
• Issue known in user community?
• Useful info in logs?
• Report results of this investigation when filing support case
Example:
Disappearing forum posts
• Group forum or course forum?
• Who has forum admin rights?
• Can authors delete own posts?
• whodunnit: check modperl access logs for POST requests in timeframe in question • Inform users that forums are not personal mailboxes and read messages should not be deleted • Restore backup to test machine and send text of lost messages to users for re-posting
SOAP for the back-end admin
• Intermittent problem vs. event driven • Issue affects server performance?
• Issue repeatable?
• Issue tracking in the system logs?
• Useful info in server OS logs?
• Report results of this investigation when filing Blackboard support case.
Example:
Database Connection Unavailable
• Blackboard application not working • Users reporting error messages upon connection • Problem is not browser specific.
• Review of Log files − Server OS, DB Logs and Blackboard log files • Log results – Blackboard log files revealed ‘full transaction logs’.
– Server OS logs reported low drive space on DB server.
• Database Maintenance Plans – System Administrator scheduled Database maintenance routines, Data integrity checks, Backup and truncation of logs. Updated Blackboard maintenance routines to include nightly and weekly DB backups
The Escalation Process within Blackboard Client Support
• Clarifying the process – towards realistic user expectations -sample case lifecycle -sample case interaction between customer and Blackboard support • How and where can you as the system administrator help to resolve cases quickly and smoothly? - SOAP
Sample case lifecycle
• Customer: case creation on Behind the Blackboard ® • TSM: replicates problem • TSM: researches KB, Wiki, tracking system for similar cases and known resolution • TSM: obtains access info from customer • TSM: creates bug report and escalates case • PSE: verifies problem and attempts resolution • PSE: escalates to Development • Dev, PSE: Analysis of customer impact; prioritization • Dev: Developer assigned; target release version determined • Dev: Issue is replicated; root cause is determined • PSE: communicates between customer and development • Dev: solution is created • Dev: fix goes to QA • Customer: Beta-release testing • Dev: Software updated released
Roadblocks in the lifecycle
or: why do things take so long?
• Communication failures • Everyone is too busy • Incorrect or incomplete information • Customer impact not clear • Issue is not high priority • Other issues hold up a release
Sample case interaction
• C: Our system keeps going down • S: Symptoms please? Logs? Access?
• C: Supplies some logs & access info • S: Need
recent
log and perf-report • S: Access not working, SSH blocked • C: now SSH works • S: escalating • S: perf-report useless, no DB info in it • … you get the picture…
Or how it should be…
• C: occasionally modperl crashes with CPU exhaustion • C: logs and perf-report from latest crash, access info • S: this has been reported by other customers – have you recently updated any building blocks?
• C: yes, block XYZ • S: any courses with course menu links to this BBB? If so, export sample such course, restore to test server, and see whether accessing that course makes modperl CPU usage rise over all bounds • C: verified, here’s sample course export • S: workaround: remove bad menu items in DB
How can YOU help?
• Clear problem description • Precise steps for replication • Preliminary analysis (SOAP) • Supply requested data promptly • • Check case status online
Focus on important issues
• We are all only human
Open Discussion
The Evolution of the Networked Learning Environment (John Fontaine, Jeff Kelley, John Knight)
– Managing the growth of your installation of Blackboard Academic Suite – Scaling to the Blackboard Enterprise Application – Strategy and planning for future growth – Blackboard Consulting’s role in evolution of the NLE – Consider Blackboard ASP Services
Evolution of System Administration Roles & Practices
EXPLORATORY SUPPORTED STRATEGIC MISSION CRITICAL TRANSFORMATIVE: THE NETWORKED LEARNING ENVIRONMENT
Business Continuity Strategic Owner Single Server Single Admin
Phase I
Multiple People Help Desk
Phase 2
Identified system manager Integrated Production Environment Change management practices System Update Deployment Plans Student /Faculty Training Programs Monitoring, Backup & Maintenance Practices Ticketing System
Phase 3 Phase 4
High Performance / High Availability Help Desk Management Long Range Planning Custom Instructional Tools
Phase 5
• • • •
Wrapping up…
Innovating Together in ‘05:
– Use SOAP method for troubleshooting – Preparation is better than problem solving
Resources Available:
–
Operations Workbook
http://behind.blackboard.com/b3/sysadminls/refcenter/r61/wkbk.htm
–
Training Center
http://behind.blackboard.com/b3/sysadminls/training/index.htm
− Web Forums
http://behind.blackboard.com/b3/sysadminls/forums/index.htm
Follow up Contact(s):
− Support site − http://behind.blackboard.com
− Product Support - Domestic 1-800-788-5264, International 1-202-715-6019 [email protected]
IF YOU ONLY REMEMBER 1 THING:
– Know your system well!