Transcript KMS Performance assessment

Contents
 Why measure the performance of KMS
 Level of analysis used in KMS measurement



Process level
KMS Level
Overall organization
 Existing models for knowledge assets valuation




ROI
BSC
KM lifecycle
Employee surveys
 Case study
Frame work for KMS implementation
Problem/ opportunity
identification
Why KMS
Knowledge diagnosis
and audit
Strategy selection
Codification /
personalization
What KMS
KMS implementation
roadmap
KMS life cycle
development
How to develop KMS
How to sustain effectiveness from KMS
KMS maturity
management
KMS evaluation
and assessment
Why measure the performance of KMS
As an intangible asset
Knowledge results are difficult to measure
and quantify
The question is “how can organizations
measure KMS benefit?”
Why measure the performance of KMS
 Tip the scale for (or against) the undertaking of a
specific project
 Provide a concrete way of defining what a KMS will
achieve and whether it has met its goals
 The need to improve the transparency of value
creation to stakeholders – for governance and
financing person.
Benefit of KMS measure
 Representation a way of learning what works and what




does not
Securing funding for KM implementation
Providing targets and feedback on implementation
Assessing the success of implementation
Deriving lessons for future implementation
Non existing of standardized method for evaluating KM results
in general and specially in monetary terms
Managers often use easily estimable quantitative factors – ROI as
to justify the dollar invested and gain support from
management in times of tight IT budgets
 While these benefits have not been tagged.
 better informed employee
 Streamline d communication with customers
 Repeatable and consistent process
If change can be measured, it has to be cross-checked
whether these are the result of KM activities
The question, then
How can organizations measure KMS benefits?
Level of analysis used in KMS measurement
Process level
 KMS Level
 Overall organization

Level of analysis used in KMS measurement :
Process level
Process – “ordering of work”; activities across time and
place; with a beginning, an end; clearly define input,
output
assessing
How KMS influence the performance of these activities
Level of analysis used in KMS measurement :
Process level
How KMS influence the performance of process activities
Evaluate
• Time saved in performing an activity
• Quality of output by an activity
Need
To analyze the evaluation overtime
Broader perspective
Workers or customers perspective such as customer loyalty, employee satisfaction,
quality and knowledge creation
Level of analysis used in KMS measurement :
KMS level
KMS stores access statistics on individual
records in the database
•
•
How many times a particular document has been
accessed
Documents have been accessed by whom
Level of analysis used in KMS measurement :
KMS level
•
•
How many times a
particular document
has been accessed
Documents have been
accessed by whom
Challenge
How to associate these
usage statistics with other
outcomes
• number of downloads
• number of posting
• May not be good
indicators of knowledge
sharing behavior of
employee
Level of analysis used in KMS measurement :
KMS level
How to associate these usage
statistics with other outcomes
IBM measures
• reuse and associated with profit (ROI), to estimate
ROI gained from the reuse assets
• Potential area of improvement
• 5000 people is carried out that aims to answer questions.
• A review f profit measures is done each quarter
• Reuse is measures and associated with profit
What Does Return On Investment - ROI Mean?
A performance measure used to evaluate the efficiency of
an investment or to compare the efficiency of a number of different
investments. To calculate ROI, the benefit (return) of an investment is
divided by the cost of the investment; the result is expressed as a
percentage or a ratio.
The return on investment formula:
ROI = (Gain from investment – Cost of investment) / Cost of investment
In the above formula "gains from investment", refers to the
proceeds obtained from selling the investment of interest. Return on
investment is a very popular metric because of its versatility and
simplicity. That is, if an investment does not have a positive ROI, or if
there are other opportunities with a higher ROI, then the investment
should be not be undertaken.
http://www.investopedia.com/terms/r/returnoninvestment.asp
How to define KM metrics
The best way to get Concrete metrics in the context of KMS
Consists in
Determining the objectives of the initiative and its expected
outcomes at the start
Link to
Business objectives
How to define KM metrics
KMS should meet specific objectives
The objectives should be very specific
So as to be further evaluated and should focus on
a particular business process and activities
Managers need to ensure that employees’ interests are
aligned with the company’s KM and their execution
How to define KM metrics
Clearly KMS objectives
Selecting the mix metric to measure
The metric should assessed before the project is
initiated in order to compare the results after
implementation.
How to define KM metrics
Importance
• Consider “before and after implementation”
• The consistency of the results over the short and
long run
To be able to make
adjustments as required
How to define KM metrics
Process analysis and selection
Sales
Improve sale
by 20%
R&D
Customer
support
Process level
System
level
Process
level
Use level
System
level
Use level
Measuring sales before KMS implementation
Process analysis and selection
Sales
Sale objective
achieved
R&D
Customer
support
Process level
System
level
Process
level
Use level
System
level
Use level
Measuring sales after KMS implementation
How to define KM metrics
The measurement should be:
• Valid : measure what intend to measure not
something else
• Reliable: give consistent results
• Actionable: give information that can be
acted upon if necessary
How to define KM metrics
The measurement should be:
•
•
•
•
•
•
Target value
Time frame
Who or what will be measured
Measurement frequency
Any assumption
Dependencies on other project or systems
How to define KM metrics
 In general organization initiate with participation
matrices such as access to knowledge base, number
of posting, number of download for a particular
document
 Cost and time saving
Once a KM initiative is mature and the
organization whishes to institutionalize its KM
Program
Performance review and Balanced scored are
methods are used
KM Measurement Process
What is the
business
objective?
What KM
methodology and
tools will we us?
Who are the stake
holders and what
they need to
know?
What is the
business
objective?
Modify
measures
Modify KM
Process
Aid decision
making
What should we
measure?
How should we
collect and
analyze the
measures?
What do the
measures tell us
and how should
we change?
Existing models for knowledge assets
valuation
ROI
 Balanced Scored Card
 KM lifecycle
 Employee surveys

Existing models for knowledge assets
valuation : ROI
Return on Investment
What benefits did we get to justify
the costs involved?
Cost associated with an
investment in IT is
straight forward to
identify
Cost and benefit of KM are
difficult to pin down
For benefit, how can we measure things such as
increased knowledge sharing, faster learning
or better decision making
Existing models for knowledge assets
valuation : BSC
BSC is being widely adopted in KM
The advantage of this approach, in terms of KM,
is that it links learning to process performance
Which in turn is linked with overall
organizational performance.
Existing models for knowledge assets
valuation : BSC
BSC has been developed by Kaplan and Norton
in 1996
Focused on linking an organization’s strategy and
objectives to measure from four key perspective
1.
2.
3.
4.
Financial
Customer
Internal process
Learning and growth
Existing models for knowledge assets
valuation : BSC
In contrast to traditional accounting measure, BSC shifts the
focus from purely financial measures to include intangible
success factors
These roughly equate to the three components of IC ,
human capital & learning, structure capital & process,
Customer capital & customer
Existing models for knowledge assets
valuation : BSC
Financial : How do we view our “share holder”
2. Customer : How do our customer see us? Are we
meeting their need and expectations?
3. Internal process : What do we need to do well in
order to succeed? What are the critical processes that
have the greatest impact on our clients and our
financial objectives?
4. Learning and growth : How can we develop our
ability to learn and grow in order to meet our
objectives in the above three areas.
1.
Existing models for knowledge assets
valuation : KM lifecycle
Measure progress of KM activates in turn of
their maturity
APQC: ROAD Map to Knowledge Manage results: Stages of
implementation
To provide the organizations with a map to guide
them from getting started right through to
institutionalizing knowledge management
Existing models for knowledge assets
valuation : KM lifecycle
The map has five stages:
1. Getting start
2. Develop a strategy
3. Design and launch a knowledge management
initiative
4. Expand and support
5. Institutionalize knowledge management
Measure associate with each stage
Existing models for knowledge assets
valuation : Employee surveys
Given the importance of people in KM
Can be used to assess
1. Organizational culture
2. People opinion, attitude, behavior
Change or Not
Existing models for knowledge assets
valuation : Employee surveys
Employee surveys can be subjective
So
May or may not reflect reality
But
Can be very benefit , as people
perceptions will determine their
behaviors with respect to KM
Case study
Case study: Wilson Sonsini Goodrich & Rosati
 Law firm with 600 lawyers
 Provide legal services to technology and growth
business enterprises at all stages of development.
 Its headquarters are in California, with offices in six
other US cities.
Knowledge intensive
firm
Why WSGR needs KM, KMS
 12 lawyers – 100 lawyers – 600 lawyers
 One office – 7 offices in 6 different locations
 Knowledge dispersion
 Clients’ expectation increased
Need a strategy to response to client
faster, and better quality
How WSGR developed KM
 KM Program started in 1998
 Report to IT group until 2001
 In 2001, Director of KM was nominated and formal
team of KM was put in place
 One team – 2 staff – one focused on the litigation
aspect (the court), the other focused on transaction
and advising clients (negotiation)
Case study: Wilson Sonsini Goodrich & Rosati
The firm runs and manage KM program by a
small full time team.
To sell and deliver the services as effectively
and efficiently s possible.
Business
objective
Case study: Wilson Sonsini Goodrich & Rosati
The KM program in WSGR focus on getting new lawyers
and other professionals up to speed quickly, and
continuously developing their legal knowledge and
professional skills
To
Support superlative client service by putting the essential
expertise and experience of the whole firm by a fingertips readily or instantly available ,of each lawyer.
Deal with different knowledge challenges that the
organization is facing
How WSGR developed KM
 Three types of knowledge are focused:
1.
2.
3.
Work product
Project
people
How WSGR developed KM
Three types of knowledge are focused:
Work product collections
Model and precedent documents to guide lawyers as
starting point for drafting new deal or case documents.
This includes sample document , patterns, high quality
templates, forms, and transactional documents for using
as guideline for lawyers.
How WSGR developed KM
Three types of knowledge are focused: Project databases
Principle deal types are profiled such as private and
public financing, mergers and acquisition
Attach profiles as possible
Lawyers can search profiles for deal similar to those they
are working on.
Good stating point
To show a client that they have valuable experience with
a certain deal type, industry, geography
How WSGR developed KM
Three types of knowledge are focused:
People, Expertise locator
Dedicated e-mail addressees are provide for tough.
Practiced – related question or referrals and
questions and answers are made searchable to
all staff via online debases.
The expert location process
Expert 1
Online
submission
of problem
Automatically
directed to the best
available experts
Expert 2
Expert 3
Resolve
Why WSGR evaluate KM program
KM team in WSGR runs program to sell and deliver legal
services as effectively and efficiently as possible.
KM team costs the firm money for salaries, technology,
and fee-earners’ time
No direct revenue
Why WSGR evaluate KM program
To justify the firm’s investment
KM team needs to find the way to show how the
program benefit
Output, Usage, Results are employed
to measure KM benefit
How WSGR evaluate
KM program
How WSGR evaluate KM program
 Output: measure
 quantity of easily accessible
 High quality knowledge
provides an answer to the
essential question
Answers the essential
questions
“Have you built it?”
The firm KM resource must have critical mass of
material before lawyers use it
How WSGR evaluate KM program
 Usage: measure
 Who use the KM resource
 In what way they use
 How often they use
Answers the essential
questions
“Did they come?”
Heavy usage of a resource generally indicates
that lawyers value it highly and
How WSGR evaluate KM program
 Results: measure
 The outcomes of using the KM resource
answers the most important questions, How
did the clients and the firms benefit?
Great benefit KM should be replicated and
expanded wherever possible.
One the provides little benefit should be either
repaired or terminated
Measuring output : weekly or monthly
 Work product collection - precedent documents
 Measure : number of items - briefs in the brief bank ,
- form in the form collection
Is the firm collect more or less, relative to the pace of over all
firm and the specific practices?
How does it help?
Number that are too low relative to the size or growth of the
practice force the firm to figure out why, and should do
something to increase them.
Measuring output : weekly or monthly
 Work product collection – precedent documents
In early 2003 – too low number of briefs in brief bank compare to the
peace of litigation practice which is quite busy.
In March 2003, the firm design to increase the number of
briefs in brief bank by improving the collection process.
Inform litigating leader to tell everyone to file each brief with the
KM department at the same time that it was filed with the court.
Having secretaries responsible for file maintenance (rather than
lawyers) send the briefs.
These two changes caused double of briefs in every months,
without major change in the pace of litigation business.
Measuring output : weekly or monthly
 Project data based- collections of deals that are profiled.
 Measure : percentage of deals closed in a given month for
which profile meet a minimum standard. Percentage are
analyzed over all and by the practice groups
How does it help?
Number that are too low force the firm to figure out why, and
to do something to increase them.
Measuring output : weekly or monthly
 Expertise locator – e-mail based Q&A facility. Use by lawyers to
locate internal and external experts and examples of their
expertise.
 Measure : number of questions sent and responses each week
How does it help?
Unlike the document and matter data based, the main problem of
this is there are too many easy and poorly draft questions. The
channel becomes devalued because recipients lose interest and
through out.
To keep the quality of e-mail high lawyers are trained to do
homework before sending e-mail questions and to explain clearly
both reason for the question and the question its self.
Send reminder when necessary – cooling off effect.
Measuring usage : site and document usage
 Site usage
 Measure : a website counter facility is used to track how
many unique users visit KM site each week
How does it help?
It represents the awareness of and demand for , each online
KM resource, reaction to training , and where the M team
should be investing their time and expertise.
Measuring usage : site and document usage
 Document usage
 Measure : number and type of users who view each KM
document
How does it help?
They show which KM documents are the most demand and
should be revised most frequently; which lawyers can be called
for feedback.
Measuring result
In WSGR results are measured only
anecdotally – user stories that show how a
layer has used a KMS to get client better,
faster, more efficient results.
Discuss how WSGR use this KM Measurement
Process
What is the
business
objective?
What KM
methodology and
tools will we us?
Who are the stake
holders and what
they need to
know?
What is the
business
objective?
Modify
measures
Modify KM
Process
Decision
making
What should we
measure?
How should we
collect and
analyze the
measures?
What do the
measures tell us
and how should
we change?