I N D I A N A U N I V Measuring Quality, Cost, and Value of IT Services Christopher S. Peebles, Craig A.

Transcript I N D I A N A U N I V Measuring Quality, Cost, and Value of IT Services Christopher S. Peebles, Craig A.

I N D I A N A U N I V E R S I T Y

Measuring Quality, Cost, and Value of IT Services

Christopher S. Peebles, Craig A. Stewart, Brian D. Voss, and Sue B. Workman Office of the Vice President for Information Technology and University Information Technology Services Indiana University Paper presented at the 55th Annual Quality Congress, Charlotte, NC Tuesday, 8 May 2001

I N D I A N A U N I V E R S I T Y

License Terms

• • Please cite as: Peebles, C.S., C.A. Stewart, B.D. Voss, and S.B. Workman. 2001. Measuring Quality, Cost, and Value of IT Services. Presentation. Presented at 55th Annual Quality Congress, Charlotte, NC, Tuesday, 8 May 2001. Available from: http://hdl.handle.net/2022/13897 This content is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/). This license includes the following terms: You are free to share – to copy, distribute and transmit the work and to remix – to adapt the work under the following conditions: attribution – you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work.

I N D I A N A U N I V E R S I T Y

IU in a nutshell

• Founded in 1820 • $2B Annual Budget • 8 campuses • >90,000 students • 3,900 faculty • 878 degree programs; >1,000 majors; > 60 programs ranked within top 20 of their type nationally • University highly regarded as research and teaching institution

I N D I A N A U N I V E R S I T Y

IT@IU in a nutshell

• Academic programs in IT through computer science, library and information sciences, engineering and technology, and most notably through new School of Informatics • CIO: Vice President Michael A. McRobbie • ~$70M annual budget • Technology services offered university-wide • UITS comprises ~500 FTE staff, organized into crosscutting units (e.g. finance and HR) and four technology divisions (Teaching & Learning Information Technology,Telecommunications, University Information Systems, Research and Academic Computing)

I N D I A N A U N I V E R S I T Y

Transformations

A Bit of Anthropology of Organizations and How IT Is Organized

I N D I A N A U N I V E R S I T Y

• Mary Douglas.

How Institutions Think

. Syracuse University Press, 1986. Dimensions of Grid and Group High Grid Low Grid anarchic isolates (marginal to society) individualistic (ego-focused network) hierarchies (nested/bounded groups) sect/enclave (bounded group ) Low Group High Group

I N D I A N A U N I V E R S I T Y

Culture, Strategy, Organization & Structure Continued

• James Cortada.

Best Practices in Information Technology.

Prentice Hall, NY, 1999 High Grid hackers systematic production continuous improvement Low Grid systematic customization Low Group craft culture High Group

I N D I A N A U N I V E R S I T Y

Value

• IT and Value Creation – It ’ s all about time: powers of automation and augmentation • IT and Value Destruction – It ’ s all about time: wasted time due to poor operating systems, poorly crafted applications, and mysterious, opaque user interfaces • IT and Value Protection – It ’ s all about time: time spent in support and education

I N D I A N A U N I V E R S I T Y

Measures of Performance and Success

• Do not have measures like EVA and “ profit ” for the success of university IT organizations as a measure • Must draw exemplars from business and benchmarks from wherever they are available • Organization performance: IBM “ Adaptive Organization ” and “ Customer Relationship Management ” • Measurement: “ The Balanced Scorecard ”

What Counts

” and “

Counting

I N D I A N A U N I V E R S I T Y

Performance Measures for All Organizations, Including University IT Organizations

• Robert Kaplan and David Norton.

The Balanced Scorecard.

HBS Press, Boston, MA, 1996

• Four dimensions of retrospective and prospective measures – Financial perspective: deployment (and growth) of revenue, ABC against internal (historical) and external benchmarks – Customer perspective: customer satisfaction measures, number of partnerships with faculty in teaching and research, support of university business processes, support of library processes – Internal perspective: process measures, classic IT measures of availability, cost-of-poor-quality, speed and depth of development cycles – Learning perspective: employee satisfaction, employee development (MSCE, CCNE, etc.), personal alignment of employee goals with position

I N D I A N A U N I V E R S I T Y

Measuring Quality and Cost

Customer Surveys Activity Based Costing

I N D I A N A U N I V E R S I T Y

Importance of Quality

• Claiming that quality is important is easy. Doing something useful is hard.

• Measurement of quality requires – Leadership initiative – Understanding your services – Commitment to two-way, fact based communication with customers • Motivations at IU – Initiative – Feeling of responsibility – Responsibility center management

I N D I A N A U N I V E R S I T Y

RCM

• RCM Implemented at IU in 1989 (Whalen 1991).

• IT organization(s) defined as a Responsibility Center; paid for through a non-discretionary tax on other RCs • IT customers are also captive users • Why not decentralize?

I N D I A N A U N I V E R S I T Y

User Satisfaction Survey

• Created out of leadership desire for quality and accountability, and desire for fact-based response to assaults on IT center ’ s budget • Key features of Survey – Administered by independent survey organization – Stratified random sampling; survey center finalizes and administers survey, performs randomization, tabulates responses, ensures anonymity – This assures credibility and quality of data – Longitudinal comparisons assured by consistency of questions from year to year

• Response rates: – IUB survey response rates in 1991 ranged from 23% (undergraduates) to 50% (faculty). – IUB response rates in 2000 ranged from 41% (undergraduates) to 71% (staff) – IUPUI survey initiated in 1997 with a 26% response rate for undergraduates – IUPUI response rate for 2000 at 36% (undergraduates) to 48% (staff) • High response rate due to small incentive ($5-$10 value) and to recognition of value of survey

I N D I A N A U N I V E R S I T Y

Survey Credibility and Utility

• External – Results disseminated rapidly and widely. Survey conducted Feb-end of April; results on Web in July or August – Results, and actions taken as a result, publicized widely throughout the year • Internal – Key component of annual internal quality assessment/plan – Often used in internal proposals as justification

I N D I A N A U N I V E R S I T Y

Survey Credibility and Utility: VAX phaseout as an example

• IU had largest academic VAX center in US, and had depended upon VAXes for ~15 years • IT organization announced plan to eliminate use of VAXes within 3 years in 1994 • Cost and quality measures informed this decision • Reaction: horror, humor, horror

I N D I A N A U N I V E R S I T Y

VAX phaseout reaction

• Horror: initial reaction of user community. Project leader set the following goal: Within two years, at least 95% of those surveyed would respond “ yes ” to the question “ Is the improvement in your computing environment sufficient to more than outweigh the cost to you of conversion ” • Humor: initial reaction of some colleagues • Horror: reaction of project team when they realized the project manager was serious

I N D I A N A U N I V E R S I T Y

VAX phaseout: the results

• The goal was met.

• The setting of user satisfaction (rather than computer center convenience) as a key goal resulted in openness to the project that was essential to its success • Full details on Web: Stewart, C.A.,

et al.

1998. Changing (almost) everything and keeping (almost) everyone happy. CAUSE/Effect 21:39-46. http://www.educause.edu/ir/library/html/cem9837.html

• Key lesson: if you are doing the right thing and communicating well, user opinion will not lag far behind expert opinion

I N D I A N A U N I V E R S I T Y

Activity based Costing

• “ There are no results inside an organization. There are only costs.

” – Peter F Drucker,

Managing the Nonprofit Organization: Principles and Practices.

NY, 1990. p. 120 Harper Collins, • Activity Based Costing-Activity Based Management – John Shank and Vijay Govindarajan.

Management

. Free Press, NY, 1993

Strategic Cost

– Robert Kaplan and Robin Cooper.

Cost and Effect.

HBS Press, Boston, MA, 1998

I N D I A N A U N I V E R S I T Y

Activity Based Costs and Management

ABCosts Wages and Benefits Training Hardware and Software (Expense or Depriciation) Maintenance and Other Contracts Expendable Supplies Circuit Charges Data Video Voice Organization Sustaining Activities Other Costs Organization Products People Processes Knowledge Quality Measures SERVICE Unit Costs

I N D I A N A U N I V E R S I T Y

The Ferengi “ First Rule of Acquisition ” : Once you have their money, never, ever give it back

I N D I A N A U N I V E R S I T Y

UITS Services — Bloomington Campus 1998-1999 Fiscal Year Report on Cost and Quality of Services

I N D I A N A U N I V E R S I T Y

UITS Services — Bloomington Campus 1998-1999 Fiscal Year Report on Cost and Quality of Services

I N D I A N A U N I V E R S I T Y

UITS Services — Bloomington Campus 1998-1999 Fiscal Year Report on Cost and Quality of Services

I N D I A N A U N I V E R S I T Y

ABC data

• Full reporting of data on Web at http://www.indiana.edu/~uits/business/scindex.html

I N D I A N A U N I V E R S I T Y

A Case Study of Activity Based Management

Reengineering E-Mail at Indiana University Bloomington

I N D I A N A U N I V E R S I T Y

Cost and Quality: an E-mail example

100 95 90 85 80 75 70 65 92 93 94 95 96

Year

97 98 99 00 Pine Exchange Vax Mail Overall

I N D I A N A U N I V E R S I T Y

Volume and Cost for e-mail services at Indiana University Bloomington, 1996 - 1998 Academic Years.

Number of Mail Accounts Number of Messages Mb Received Cost/Message Cost/Account/ Year 1996-97 66,000 307,000,000 1,258,000 $0.001

$8.96

1997-98 48,000 270,000,000 NA $0.002

$9.80

1998-99 46,000 179,000,000 391,825,000 $0.002

$6.11

I N D I A N A U N I V E R S I T Y

User perceived quality measures for five mail systems used at Indiana University Bloomington, 1996-1998 Academic Years 1996-1997 Percentage Satisfied Satisfaction Score 1997-98 Percentage Satisfied Satisfaction Score 1998-1999 Percentage Satisfied Satisfaction Score Pine Unix Mail (Elm) Eudora Group Wise Exchange 95.1% 82.1% 96.5% 89.0% 96.3% 4.1

3.5

4.2

3.8

4.2

90.5% 82.2% 87.8% 78.4% 85.0% 3.9

3.5

4.1

3.8

85.8% 62.2% 92.3% 76.9% 92.5% 3.7

3.0

4.1

3.5

4.2

I N D I A N A U N I V E R S I T Y

Comparative measures for e-mail support requests in Indianapolis and Bloomington during the academic year 1998-1999. Pine IU-B Outlook Exchange IUB Pine IUPUI Outlook Exchange IUPUI 1998-99 Total Number of Users 44,000 Number of Calls to Support Center 1,537 Call per User 1 per 28.6

Percentage of e-mail Calls to Support Center 23.8% Percentage of Total Calls to Support Center 2.4% 4,500 45,000 6,000 1,773 1,943 2,425 1 per 2.5

1 per 22.7

1 per 2.5

27.4% 22.7% 28.2% 2.7% 4.2% 5.3%

I N D I A N A U N I V E R S I T Y

Balanced Scorecard: example from Research & Academic Computing

• Service measures (for all, current levels and  – Quality – from user survey – Utilization – from user survey – Cost • Staff – Turnover from last year) – % receiving some sort of external certification • Results – Number and $s of grants applied for or supported – Number and $s received (not less than $1M in last 3 years) – Publications • Enabled by our services • By IT organization staff

I N D I A N A U N I V E R S I T Y

Quality Planning

• Internal quality plan written by each management unit, focusing on assessment and plans for improvement • User survey and ABC data key indicators • Any service with a satisfaction rating of less then 90% is generally taken to be cause for action • IT center budget office, which has customers only within the IT center, does its own annual survey!

I N D I A N A U N I V E R S I T Y

IT Support@IU

Support in Breadth

I N D I A N A U N I V E R S I T Y

IT Support at Indiana University

• IT has invaded our lives and culture • Demands for IT support increased exponentially • Budgets remained steady (at best) • How do we deliver a quality product under such a knowledge crisis?

I N D I A N A U N I V E R S I T Y

Some background

• Support free, unlimited to 115,000 IU Faculty, Staff, Students.

• End-user, general-purpose support provided by central computing organization (UITS) – Support Center – Knowledge Base – Student Technology Center Consultants – Residential IT Support – Education Program

I N D I A N A U N I V E R S I T Y

Support Center

• Provide support to 400,000 contacts/year • Campus-based centers • Variety of questions/platforms • Various support delivery methods – Call Center – Walk-in Center – Online

I N D I A N A U N I V E R S I T Y

Call Center

• Calls distributed to ACD routed queues – Computers/Applications w/ Windows OS – Computers/Applications w/ Macintosh OS – Computers/Applications w/ Unix OS – Mission Critical Systems – General IT questions

I N D I A N A U N I V E R S I T Y

Walk-In Center

• Campus-centric locations • Support requiring personal authentication • Computer configuration • General-purposes support of convenience

I N D I A N A U N I V E R S I T Y

Online Support

• Support via e-mail • Knowledge Base

I N D I A N A U N I V E R S I T Y

Managing the quantity of contact

• Proactive versus reactive support • Tools to reduce time of contact • Increase quality of answers

I N D I A N A U N I V E R S I T Y

Measuring the Quality of Contact

• Annual Department Survey • Daily Support Center Survey – Simple measures – Non-intrusive to customer – Indicator of quality of service

I N D I A N A U N I V E R S I T Y

The 3 Key Indicators

3. Did the customer receive a solution in a timely manner and delivered courtesy and respect ?

I N D I A N A U N I V E R S I T Y

What do we do with the info?

• Send to 45 users from day before • Every response is read and every “ no ” receives a response.

• Used as indicator of need for training or resource allocation • Positive feedback Date: Mon, 15 Jan 2001 11:15:49 -0500 (EST) From: kdyn To: Subject: Re: Dell Optiplex GX1..MS Natural Keyboard (#269.1445) THANKS, Jim. Please forward my message on to your boss – this is the BEST service I've ever gotten on ANY computer related problem anywhere. I'm glad my technology fee is paying your salary. THANKS.

I N D I A N A U N I V E R S I T Y

Results for first 10 months of 2000

Call Center Walk-In Center via Email

Total Responses % Satisfied Users

1067 148 195

92.7% 94.6% 88.2%

I N D I A N A U N I V E R S I T Y

Managing the Resources

• Utilize technology for what it does best • Utilize humans for what we do best

I N D I A N A U N I V E R S I T Y

Knowledge Base Background

• KB was born in 1988 • Internal tool to retain computing support information • Became a self-serve tool in 1995 • For more info see: “ What is the history of the Knowledge Base ” at http://kb.indiana.edu/data/acjq.html

I N D I A N A U N I V E R S I T Y

What is the KB?

The KB is a system with three physical components: 1.

It is a repository of problems and their resolutions (currently holding over 7,000 entries); It is a search engine to sort through that repository (currently handling over 50,000 queries each week); and It is a user interface that allows a user to engage the search engine to manage the data in the currently used by hundreds of thousands of users worldwide).

I N D I A N A U N I V E R S I T Y

KB Today

• 7000+ Entries • Search/Menu • KB Supports ~115K IU Users with specific campus information • Internal/External training tool • Available for use all over the world • Already received over 6M hits this year • See stats: http://kb.indiana.edu/info/usage.html

I N D I A N A U N I V E R S I T Y

Quality Control Process

• Extensive process control • Review and address the “ goose eggs ” • Measure the “ no, I did not get an answer ”

I N D I A N A U N I V E R S I T Y

Connecting the People and the Technology

• Trouble Ticket System replaced with Work Flow System • Eliminates the “ Post-It-Note ” syndrome • Manages workload, communication, information, and escalation.

• Eliminates duplication of work • Manages KB process

I N D I A N A U N I V E R S I T Y

Student Technology Center Consultants

• 1200+ Machines, 40 Labs, 200 Consultants • 50 percent of student body use consultant services • Training previously focused on technology • Change to service oriented training, given technology resources • Improved satisfaction from 75% to 90%

I N D I A N A U N I V E R S I T Y

Residential IT Support

• In-room services – 11,000+ students • Residential Technology Centers – 400 workstations • Concentrating on proactive support • Concentration on connection to network • Support Center first line of contact • Metrics too new to conclude

I N D I A N A U N I V E R S I T Y

Education Program

• Better educated users support themselves – Or ask harder questions….

• Free classroom training • Free CBT training • CBT topics integrated into KB entries • Satisfaction rating 94.9%

I N D I A N A U N I V E R S I T Y

Good Support – A total team effort

• Not just the Support Organization • Careful Implementation Plans • Extensive Change Management • Preparation of Support Resources – Consultant training – KB entry – Support Tools • Careful communication to Organization and Customer base

I N D I A N A U N I V E R S I T Y

IT Support@IU

Support in Depth

I N D I A N A U N I V E R S I T Y

Center for Statistical and Mathematical Computing

• • Formed in 1991, prototype for IT center specialized support services.

http://www.indiana.edu/~statmath/ • Formed as part of Interdisciplinary Consortium of Statistical Applications, with Dept. of Mathematics and Graduate School. • Math and Grad School provide statistical consulting by a faculty member and specialized classes

I N D I A N A U N I V E R S I T Y

Stat/Math Center, 2

• Most full time staff holders of terminal degrees (Ph.D or Ed.D) • Offers education, consulting, software management.

• Software grouped into three categories: – General utility. Expert consulting & provision (often via site licenses) – Specialized but critical utility: some consulting & provision – Utility within one department: provision • Consulting: offer phone, e-mail, and long term consulting. Long term done with team of researcher, statistics consultant, and statistical computing consultant

I N D I A N A U N I V E R S I T Y

Unix Workstation Support Group

• http://www.uwsg.indiana.edu/ • Supports people using Unix as a desktop workstation OS, from astrophysicists running a commercial Unix variant in their research lab to undergraduates running Linux in their dorm room • Offers classroom and online training, mirror sites, site licenses • Staff commonly present papers and give tutorials • Often collaborates with CS dept

I N D I A N A U N I V E R S I T Y

Library Electronic Text Resource Service

• http://www.indiana.edu/~letrs/ • Consulting and services in support of electronic delivery of texts • Partnership with University library, for both service and Victorian Women Writers Project ( http://www.indiana.edu/~letrs/vwwp/index.html

) • Led by manager with Ph.D. in English Literature

I N D I A N A U N I V E R S I T Y

Advanced Visualization Laboratory

• http://avl.iu.edu/ • Manages VR facilities, provides consulting on immersive and advanced visualization facilities • Writes community codes (3DIVE, XMView) in collaboration with CS and Chemistry departments • Managed by Ph.D. computer scientist

I N D I A N A U N I V E R S I T Y

Specialized Support Services: keys to success

• Pareto 80-20 rule. Critical additions to human knowledge (and accomplishments that distinguish university overall) often come from those with nonstandard computing needs • Specialized support services successful when: – Strategic need spans multiple academic units – Staffed by consultants expert in content and computing tools – Consultants and researchers view each other as partners – Formal partnership with one or more academic units helpful

I N D I A N A U N I V E R S I T Y

IT Support@IU

Distributed Support

I N D I A N A U N I V E R S I T Y

Distributed Support Services

• Classic set of support models

(McClure, Smith, Lockard)

– Centralized – Decentralized – Haphazard – Distributed • Planned/Proactive (IU Approach) • Reactive • Feral

I N D I A N A U N I V E R S I T Y

Leveraged Support Model

• Roles

(Voss, Workman, Alspaugh, et al)

– Support provided centrally – Support provided by the users themselves – Support provided by local IT staff • Central IT organization responsible for ensuring all elements

I N D I A N A U N I V E R S I T Y

Quality Distributed Support Evolution at Indiana University

• Distributed Support Assistants (DSA) Program • Technical Information for Excellent Support (TIES) Program – Education/Certification (Ed/Cert) • Partners in Computing Support (PICS) Program

I N D I A N A U N I V E R S I T Y

DSA Program – Seeding Quality Local Support

• Two-year “ Try before you Buy ” Effort – Funding assistance from IT organization – Staff mentoring and coordination • Advantages – Seeing the value of local support – Consistency of local support with central IT – Partnering – common purpose, non-compete

I N D I A N A U N I V E R S I T Y

DSA Program – Seeding Quality Local Support

• Unforeseen Consequences – Development of local IT job market – Close-knit Community – Career paths – in and out of central IT • Covered 26 campus entities – Success led to ‘ wild growing ’ local support in other units

I N D I A N A U N I V E R S I T Y

TIES Program

• As support grew wild, it needed training and contact with central IT – thus TIES was born – 10-12 weekly sessions each semester – Training and information sharing focus – Integration with DSAs elsewhere • As time passed, need for detailed skills that were certifiable – Ed/Cert developed

I N D I A N A U N I V E R S I T Y

PICS Program

• More than just a user group – a set of services – Tools acquired centrally, served to local staff – Forums for discussion, sharing, and learning – Special Focus on serving our ‘ partners ’ • Forum for getting feedback on services • Forum for getting input from partners

I N D I A N A U N I V E R S I T Y

Organizing for Supporting Local Support Providers

• LSPs need forward looking focus – Departmental Computing Advising and Support – Departmental Support Lab • LSPs need back-up support – Local Support Provider Services – LAN Lab – 2 nd and 3 rd level expertise in key applications and technologies

I N D I A N A U N I V E R S I T Y

An example …

Date: Tue, 30 Jan 2001 10:17:46 -0500 From: LSP Services To: [email protected]

--------------------------------------- Windows 2000 and the Active Directory is almost upon us. As most of you have hopefully heard by now, the Active Directory is an upgrade to the universities current Windows NT 4 infrastructure. UITS is planning on a production deployment by late February. Please join us in an information session this coming Friday morning, February 2nd 9:00 am in BS 2007. We will be discussing the design and architecture of the new Active Directory as well as some new technologies that will be introduced in this upgrade. Representatives from LSP Services, Messaging and the Security office will be there. This will be followed up by a question and answer session. If you have any questions please don't hesitate to bring them with you and ask! Hope to see you there!

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= LSP Services University Information Technology Services [email protected]

317.274.6577

I N D I A N A U N I V E R S I T Y

Satisfaction

• User Survey results for UITS distributed support services 2000 1999 1998 1997 Satisfaction with UITS Distributed Support Services 93.3% 90.1% 90.8% 91.4% • User Survey results for local support providers 2000 1999 1998 Satisfaction with LSPs in departments 89.1% 85.4% 85.7% 1997 86.8%

I N D I A N A U N I V E R S I T Y

Key Factors in Delivering Quality Local Support Services

• Organizing Central IT to deliver support • Emphasizing Local Support Providers • Emphasis on building LSP skills • Emphasis on building User skills • Providing Specialized Support services • Providing Tools that enable user self-support (Knowledge Base)

I N D I A N A U N I V E R S I T Y

Other Environmental Success Factors

• Institutional/Enterprise Software Licensing – Standardization and consistency – Latest versions – lessen support burden • Departmental IT Equipment Modernization and life-cycle funding – Minimizing and eliminating problems and workload caused by out-of-date equipment – Standardization – volume purchases

I N D I A N A U N I V E R S I T Y

Support is the Critical Element

• Adds value through standardization of hardware and software • Adds value through education of customers and IT professionals • Prevents some of the destruction of value through poorly designed software and poorly manufactured hardware,

I N D I A N A U N I V E R S I T Y

IT@IU

The Value Equation

U N I V E R S I T Y I N D I A N A

For faculty: How helpful has the information technology environment at IU been in your teaching activity?

40 35 30 25 Percent 20 15 10 5 0 1997 1998 Year 1999 2000 Not at All Helpful Somewhat Helpful Very Helpful

U N I V E R S I T Y I N D I A N A

For faculty and graduate students: How helpful has the information technology environment at IU been in your research activities?

45 40 35 30 Percent 25 20 15 10 5 0 1997 1998 Year 1999 2000 Not at All Helpful Somewhat Helpful Very Helpful

I N D I A N A U N I V E R S I T Y

For undergraduate and graduate students: How helpful has the information technology environment been in your learning experience at IU?

45 40 35 30 Percent 25 20 15 10 5 0 1997 1998 Year 1999 2000 Not at All Helpful Somewhat Helpful Very Helpful

I N D I A N A U N I V E R S I T Y