- D-Scholarship@Pitt
Download
Report
Transcript - D-Scholarship@Pitt
Here Comes the Sunburst:
Measuring and Visualizing
Scholarly Impact
John Barnett
Scholarly Communications Librarian
Jennifer Chan
Assistant Scholarly Communications Librarian
Office of Scholarly Communication and
Publishing
University Library System
University of Pittsburgh
Here Comes the Sunburst:
Measuring and Visualizing
Scholarly Impact
University of Pittsburgh
Pittsburgh campus + regional
campuses in Bradford,
Greensburg, Johnstown, and
Titusville
16 undergraduate, graduate,
and professional schools
456+ degree programs
2012: conferred 8,949 degrees
University of Pittsburgh
Top 10 American higher ed. in
federal funding (NSF)
Top 5 in annual research support
(NIH)
5,369 faculty; 4,470 full-time faculty
Research conducted: more than 300
centers, institutes, laboratories,
clinics
University Library System
ARL
22nd largest academic library system in North
America
25 libraries; 6.6 million volumes
279,000 current serials
Sum of the Parts
Office of
Scholarly
Communication
and Publishing
ULS
Department of
Information
Technology
Liaison
Librarians
Why Pitt?
Strategic goal:
Innovation in scholarly communication
Providing services that scholars understand, need,
and value
Putting ourselves in faculty “spaces”
Re-envisioning our librarian liaison program
Deepening our understanding of scholarly
communications issues
Why PlumX?
Making research “more assessable
and accessible”
– Gathering information in one place
– Making it intelligible and useful
Measuring and visualizing research
impact
Correlating metrics from traditional and new forms of
scholarly communication
Allowing researchers, labs, departments, institutions to
track real-time scholarly impact
Promoting research, comparing with peers, connecting
with new research
Altmetrics Project Timeline
Spring
2012:
Fall 2012
• First meeting
with Plum
Analytics
• Gathered data
from pilot
participants
Summer
2012:
• Announcement
of Pitt as Plum
Analytics’ first
partner
Spring
2013
• Faculty
surveyed;
enhancements
made
Winter 2013
Fall 2013
• PlumX pilot
system made
public
• IR widget
launched;
rollout
preparations
Pilot project aims
Develop a tool for measuring and visualizing
research impact
Gathering information in one place
Intelligible and useful
Impact in social media and other scholarly
communication methods
Traditional measures counted as well
See where to disseminate works to increase impact
Traditional vs. new
• Traditional measures are also
counted
• Findings are complementary
to conventional methods of
measuring research impact
(e.g., H-Index)
• Not intended to replace them
New measures
More comprehensive: Altmetrics = ALL METRICS
–
–
–
–
–
Citations
Usage
Captures
Mentions
Social Media
Covers impact of online behavior
– Because scholars increasingly work online
Measures impact immediately
– Because citation counts take years to appear in literature
Pilot Process
Created
Altmetrics
Task Force
CV receipt
and
records
creation
Engaged
Liaison
Librarians
Built
Sharepoint
Site
Pilot
Selection
Pilot Project Participants
• 32 researchers, various
disciplines
• 9 schools
• 18 departments
• 1 complete research group
• Others joined as they
learned about the project
Pilot Project Participants
discipline
school/department
Selected
faculty
participants,
diversified by:
online behavior
level of career
advancement
Technologies
Internal
–
–
–
–
IR built on Eprints Platform
Sharepoint
Microsoft Office Suite
PMID/DOI data import tool
External
– PlumX
– DOIs
– PMID
Data collection for pilot project
• Created records in D-Scholarship@Pitt, our
institutional repository
• Focused on articles, books, book chapters,
proceedings
• Scholarly output with standard identifiers
• DOI, ISBN, PubMed
ID, official URL, etc.
• Scholarship
produced since
2000
Other Library work
• Developed guidelines to standardize record creation
• Data entry from faculty CVs into IR (2 to 3 student
workers with QA by librarians)
• Librarian liaisons and other staff trained in record
creation
• SharePoint site used to track work completed
• Coordination with pilot faculty
• Gathered feedback and administered online survey
Sharepoint
Altmetrics Meetings Minutes
Faculty CVs
Excel spreadsheets
Word docs
External Data Sources
Metadata sources
Faculty CVs . . . But verify metadata!
Books: PittCat, WorldCat, Books in Print, publisher
sites, online retailers
Journals: Serials Solutions list, journal websites,
JournalSeek, UlrichsWeb, DOAJ, PubMed
Conference presentations: Websites, PittCat,
indexes, WorldCat
PMID Import Tool
Custom build by SysAdmin for Eprints Platform
Utilizing PMIDs from PubMed, able to import
records that prepopulate metadata fields
– Item Type, Title, Abstract, Creators, Publication Title, ISSN,
Volume/Issue, Page ranges, Date and Date type, DOI,
MeSH Headings, Grant Information, Keywords, etc.
Data Ingestion
Full-text sources
DOAJ
ERIC
PLOS
SSRN*
Other repositories*
Federal government websites*
Conference websites*
* Use with caution
Plum Analytics processing activities
Harvest records from Pitt IR for each participant
Build profile for each researcher in PlumX
Harvest additional online artifacts NOT in Pitt IR
Use data mining to harvest publically available
metrics from hundreds of sites on the Web
Create visualizations to display metrics on PlumX
interface
Key features
Faculty profiles
Online ‘artifacts’
–
–
–
–
–
Article
Book
Book chapter
Video
Etc.
Impact graph
Sunburst
Faculty profile
Online ‘artifact’ display
Impact graph
Sunburst
Feedback
• Solicited via email and online survey
• Generally positive in most cases
• Data corrections
• Errors in profiles
• Links to wrong data
• Quickly corrected by Plum staff
• Requests for results from additional online sources
(Google Scholar, SlideShare, Reddit, etc.)
• PlumX collects data from these but did not gather information
in advance for profiles
The survey says
Surveyed pilot project faculty in spring 2013
@ 1/3rd responded to the survey
Meaning 13 out of 32 participants responded
Accurate and useful data
90
76.92
80
70
60
Strongly disagree
Disagree
Agree
Strongly agree
50
40
30
20
10
15.38
7.69
0
0
Summary profile data page
The bar graph
70
58.33
60
50
40
30
25
20
16.67
10
0
0
Usefulness of interactive bar graph
Strongly disagree
Disagree
Agree
Strongly agree
The sunburst
60
50
50
41.67
40
Strongly disagree
Disagree
Agree
Strongly agree
30
20
8.33
10
0
0
Usefulness of sunburst
Traditional & new measures
60
54.55
50
40
36.36
Strongly disagree
Disagree
Agree
Strongly agree
30
20
9.09
10
0
0
Conveying traditional and new measures
Usefulness of altmetrics
60
54.55
50
40
Strongly disagree
Disagree
Agree
Strongly agree
30
20
18.18
18.18
9.09
10
0
The value of altmetrics
Learning something new
40
36.36
35
30
27.27
27.27
25
Series 1
Series 2
Series 3
Series 4
20
15
10
9.09
5
0
Learning something new about my own research
Comments
Affiliations/bio inaccurate or has missing
information
“Mentions” by whom & when?
Publications misclassified
– Books vs. conference proceedings
Data not collected
– Google Scholar
– Slideshare
Comments
Filter out unwanted information
Data are wrong—and not useful
Overabundance of information in sunburst
“I only care what a select group of scholars thinks
of my work”
“I did not find this useful for my discipline”
Observations
Lacked information about faculty practices
Are the results useful to all faculty, all disciplines?
May appeal more to faculty who are early in their
careers or whose work is more contemporary
Will the data be used against faculty or programs?
Labor-intensive strategy
When it comes down to it . . . Does anyone care?
Embeddable widgets
(in development)
For researchers, to add to:
• their own Web pages
• department directories
• IR researcher profile page
For individual artifacts,
to build article level metrics
for imbedding in:
• IR document abstract page
• Article abstract page for
journals we publish
Roll-out challenges
Who creates profiles? Who edits?
What information should be included in profiles?
Who can view them?
Separate data gathering from D-Scholarship
deposits?
Who promotes the service? Who trains?
Timing . . .
Future plans
Data checking
Additional data gathering
Record merging/deduping
Ability to edit user profiles and artifact records locally
Open API
To allow integration with other online systems
More exhaustive scholarly practices survey for all faculty
Rollout to all Pitt Researchers
Will use automatic feed from Pitt IR to PlumX
Discussion
How would you “sell” PlumX to additional faculty?