Transcript Session C102 http://www.ukoln.ac.uk/web-focus/events/conferences/ili-2010/ Monitoring and Maximising Organisational Impact using the Social Web Brian Kelly UKOLN University of Bath Bath, UK Acceptable Use Policy Recording/broadcasting of this talk, taking photographs, discussing the content.

Session
C102
http://www.ukoln.ac.uk/web-focus/events/conferences/ili-2010/
Monitoring and Maximising
Organisational Impact using the Social Web
Brian Kelly
UKOLN
University of Bath
Bath, UK
Acceptable Use Policy
Recording/broadcasting of this talk,
taking photographs, discussing the
content using Twitter, blogs, etc. is
permitted providing distractions to
others is minimised.
Email:
[email protected]
Twitter:
http://twitter.com/briankelly/
Blog:
http://ukwebfocus.wordpress.com/
Tag for del.icio.us ‘ili-2010-talk'
UKOLN is supported by:
This work is licensed under a Attribution-NonCommercialShareAlike 2.0 licence (but note caveat)
Introduction
2
About Me
Brian Kelly:
• UK Web Focus: a national advisory post
• Long-standing Web evangelist (since Jan
1993)
• Based at UKOLN, University of Bath, with
remit to advise HE/FE sector
• 790+ blog posts since Nov 2006
• Over 350 presentations given since 1997
• Speaker at all ILI conferences except ILI
2008
Introduction
3
About This Talk
Summary
• The Social Web is becoming accepted as a
valuable tool for use by information professionals.
• But how do we measure the effectiveness of our
use of the Social Web?
• How do we provide evidence of value of our
services?
• What are the limitations of gathering and using
metrics?
• What should we be doing?
Contents
Structure of this talk
4
Reshaping The Gartner Curve
Listening to users
Supporting early
mainstream adopters
Gathering evidence
Demonstrating ROI
Advocacy
Need to (a) avoid the chasm; (b) manage expectations
and (c) move to productive service
5
Some Statistics
Statistics about the UK Web Focus blog (from
1 Nov 2007- 23 Aug 2010)
41 : Wikio ranking
770 : no. of posts
3,584 : no. of comments
270,978 : no. of views
561,401 : no. of spam comments (since reset)
Impressive?
Not really! Nos. meaningless without context
6
What’s The Point?
Arguments against gathering statistics:
• We know we’re doing a good job
• Our users are happy with our services
• The effort with distract from core business
Arguments for gathering statistics:
• The government is unconvinced!
• The Daily Mail is unconvinced!
• Evidence may indicate that services are no
longer being widely used
• Evidence may indicate that services are
popular and required extra resources
7
Political Context
• David Camefon
X
8
Need To Understand Trends
How effective are mailing lists?
• Very: everyone uses
mailing lists
• Not much: we now use
RSS, blogs, Twitter, …
Not much: Correct! Evidence
shows significant decrease in
Web support team lists over
past 5 years
Very: Correct! lis-* lists still
popular despite small
There’s a context to answers.
Evidence can help inform decisions.
9
Who Are We Doing This For?
Who should we be providing impact data for?
• Decision makers: so they have evidence on
which service to fund from smaller pot
• Our funders: they can mandate this
• Our peers: they can learn – and we should
support such learning and sharing
• Our users: if our services are well-used, they’ll
be interested in this
• Ourselves: we can learn from our successes and our failures
Moves towards greater transparency (e.g. salaries of
senior managers; MP’s expenses; etc.) should be
reflected by those working in libraries
10
Case Studies
Let’s look at:
• Metrics for use of various Social Web
services
And explore:
• Metrics which can be determined for little
effort
• Significance of the numbers
• How the metrics can be gamed
Before:
• Reviewing limitations
• Making suggestions
11
Blog Usage Statistics
UK Web Focus blog usage since launch in Nov 2007
2009: 247 visits/day
2010: 224 visits/day
Stories:
• Preparation for workshop on SEO
• Running tool to create PDF copy of blog posts
What does this tell us:
• Ways of measuring & influencing nos. of visits (but
not necessarily indicative of purpose of service)
12
Blog Metrics: Technorati
Technorati indexes ~1 M
English language blogs
• Nos. of links from
other (indexed) blogs
weighted by their own
link popularity (cf
Google Page Rank)
• Comparisons with
peers
But:
• Need to register blog
• Sometimes breaks
13
The Evidence
•Evidence of the importance of Twitter for
driving traffic to blogs
UK Web Focus Blog:
Total of 250,00+ views.
Average 250/day over 3.5 years
But how do they arrive at the blog?
14
“The Power Of Passed Links”
The Value Of Twitter Is In “The Power Of Passed
Links”
Wilson predicts that at current growth rates, Twitter
“will surpass Google for many websites in the next
year.” And that just as nearly every site on the Web
has become addicted to Google juice, they will
increasingly try to find ways to get more links from
Twitter. Because Twitter equals traffic. …
Moreover, he asserts that these Twitter links “convert
better” than search links because they are often prefiltered and come in the form of a recommendation
from someone you are following.
•TechCrunch, June 2009
15
Claiming a blog in
Technorati
Note need to publish
(temporarily) unique
code to demonstrate
ownership (useful to
do at start).
Thank you for submitting your blog claim on
Technorati. Technorati will need to verify
that you are an author of the site
http://blogs.ukoln.ac.uk/good-apis-jisc/ by
looking for a unique code. We have just
assigned the claim token 2XAVJZX29XXX
to this claim.
16
Blog Metrics: Wikio
Wikio: similar
tool for blog
rankings
Three UK HE
blogs in top
~50 technology
blogs
17
Claiming a
blog in Wikio
18
There Are Limitations …
League tables can
be flawed
• Non-existent
blog
• Excellent blog
• Blog for
completed
project (no
recent posts)
Need for scepticism
19
… And Benefits
20
Monitoring usage statistics can provide indications of
successful strategies for user engagement:
• Something good happened on 19 Feb 2007!
• This was date of post on email subscription
service to blog posts
Summary on Blog Metrics
We have seen:
• There are services for providing blog
metrics
• Signup up at blog launch can be useful
• The services can be flawed
• Monitoring and keeping records can help
to spot success stories
But:
• What about limitations, ethics, …?
• What about other Social Web services?
21
Case Study 2a: Twitter
Twitoaster
provides
summaries
of Twitter
responses
& RTs
22
Exploring The Evidence
Twitter can
be a
significant
driver of
traffic
Where would blog be now
without Twitter? Twitter
seems to have decimated
RSS traffic.
23
Summarizr
Summarizr can provide
statistics on event
hashtag usage e.g. top
twitterers, hashtags &
URLs tweeted, etc.
Summarizr
processes
data stored in
Twapper
Keeper Twitter
archiving
service
24
Summarizr
Summarizr
also provides
information on
geo-located
tweets
25
Possible
benefits:
• Are they local
or remote?
• Is geolocation
used?
• What are
barriers?
Wikipedia
There is a
lack of
awareness
of the
usage
statistics
for
Wikipedia
articles
Missing data
As in other examples the statistics may be flawed
26
Slideshare
How effective is Slideshare?
27
Slideshare Evidence
(1)
Slides uploaded on behalf
of Professor Carole Goble
•Carol Gole’s
slides
28
Slideshare Evidence (2)
•Tweet from
Steve
Wheeler, 25
August 2010
What is a seminar? A mechanism
for creating & delivering content
which is consumed by others?
When does the digital resource
become more impact than the
physical equivalent
29
JISC VfM Report (2005)
“Whilst a Value for Money exercise measures and
compares costs, it must also take account of the
mix of quality, resource use, fitness for purpose,
timeliness and convenience to judge whether or
not together, they constitute good value. As such,
Value for Money is one measurement of good
practice. Some elements under consideration may
be subjective, difficult to measure, unavailable,
intangible or misunderstood. Judgement is
therefore required when considering whether value
for money has been achieved.”
30
TASI Example
‘Value for Money’ exercise performed in 2005
suggested that advice from TASI saved user time and
effort and that large savings were being made.
The following calculation was proposed:
• Suppose each workshop attendance or enquiry
leads to a saving of one/person day. Suppose that
each report read saves someone half an hour.
Value a person/day at £109.
• A conservative estimate suggests that in 2004-05,
for every pound of funding received, TASI saved
the community at least £16 pounds.
An illustration of the need for evidence on
which to base scenarios
31
Facebook Trends
32
Jun 2008:
“The Open University Facebook page is the top of all
University pages, with 7,539 fans (2,874 a month ago).
The other most popular UK Universities are Aston
University (2,976 fans), Royal Holloway (1,765),
Aberystwyth University (1,655 fans), University of
Central Lancashire (1,475 fans), Keele University
(1,420 fans), Cardiff University (1,357 fans) and the
University of Surrey (1,166 fans).”
28 Sep 2010 (~start of academic year):
OU: 28,949
Aston: 8,445
Royal Holloway: 9,093 Aberystwyth: 7,326
UCLan: 7,982
Keele Uni: 6,716
Cardiff Uni: 18,689
University of Surrey: 8,063
Understanding Trends
What can be learn from Facebook trends
• Evidence which can address ideology
• Diversity of approaches (e.g. University
Freshers Fb page)
• Identification of fragmentation of Facebook
presence
• Inform discussions on resourcing, policies,
etc.
33
Stories + Statistics
We need (a) Evidence to support scenarios and (b)
Stories to make evidence comprehensible
Assertion
Twitter can help to
develop professional
relationships
Response
Really?
Where’s the
evidence?
What is the cost?
34
Story
Following a Twitter response
from @slewth I noticed her
interests & a link to her blog.
I read her blog posts – and
exchanged tweets & emails.
A few months later we had
collaborated on a paper.
At W4A 2010 paper was award
a prize as the best
communications paper 
Bigger Picture
35
Metrics are flawed as they don’t take into account:
• Diversity of uses of Social Web (e.g. blog for
reflective thinking vs dissemination)
• Size of organisation and levels of resources
• Temptation to ‘game the system’ and aim to
maximise metrics rather than underlying purpose
We could:
• Refuse to engage with current tools & approaches
• Wait for the perfect analytic framework
But:
• We’ll lose out in understanding current approaches
• Is there a perfect framework?
• What if evidence is needed to justify existence?
The Bigger Picture: OU
BBC News, 29 June 2010
36
“… it’s a whole new
generation of engaging
digital content. 6.12m
downloads, 64k
visitors, 180k
downloads a week.
Lots of top-20 hits.
Over 50% outside the
UK. How much does it
cost? Very costeffective versus putting
signs on the side of
buses.”
Martin Bean, OU VC
Amazon Rankings
Amazon provides rankings
for its books, even though
they have different
purposes, audiences, …
37
Manipulating Statistics
Some suggestions:
• Create blogs which have links back to
your services
• Automatically submit comments to open
Social Web services containing links
• Use keywords for your competitors
• Give your content away with a CC
attribution licence
• Keep an eye out for queries and link to
your services when appropriate
• Include your URL in your email signature
38
Ethics
39
Concerns:
• There are lies, dammed lies and statistics
• Metrics can be gamed
Ethical approaches:
• Openness:
 Being honest about limitations of metrics
 Being open about the purposes of the services
• Diversity of approaches to evaluation:
 Provide a range of statistics
 Statistics plus stories
• Staff development:
 Ensure staff are aware of need for metrics
 Ensure that are aware of ethical approaches
Policies
Can you measure the
effectiveness of a
service if you don’t
have a clear idea of
its purpose?
There’s a need for
(simple) policies on
purpose of Social
Web Services
40
Maximising Impact
How do you go about maximising impact?
• Create content!
• Empower your staff
• Join in the discussions
• Go where the discussions take place
• Monitor the impact and identify success
stories
41
Questions
Questions are welcome
Name: Brian Kelly
Address: UKOLN, University of Bath, BATH, UK
Email: [email protected]
Web site: http://www.ukoln.ac.uk/
Blog: http://ukwebfocus./wordpress.com/
Twitter: briankelly and
ukwebfocus (automated feed)
42