Transcript Online real-time tweets extraction, mapping and dissemination

Online real-time tweets extraction,
mapping and dissemination
Xiannian Chen and Gregory Elmes
West Virginia University
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
1
Geospatial data
• Challenges that geospatial data face
• Traditional geospatial data updates are slow.
• The real-time data is critical in disaster management.
• Crowdsourcing (Jeff Howe 2005, Brabham 2008)
• VGI (Volunteered Geographic Information, Goodchild 2007)
• Real time geographic information from volunteers
• “geographic information acquired and made available to others
through the voluntary activity of individuals or groups, with the
intent of providing information about the geographic world”
(Elwood, Goodchild, and Sui 2011)
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
2
Values of VGI
• Captures of real time changes
• Quick dissemination
• A significant data source.
• Individuals’ understandings of a local event
• Rich in context
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
3
API and REST
• Application Programming Interface
• A library that may include specification for routines, data
structures, object classes, and variables
• An interface for software components to communicate
• All social media websites provide API
• REST (REpresentational State Transfer, Fielding 2000)
• A HTTP-based style to describe the resources it represents.
• uses HTTP for all four CRUD (Create/Read/Update/Delete)
• The access of REST services is a simple URL
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
4
Twitter and Tweets
• Twitter: a real-time information network
• Tweets
• created_at
• coordinates
• favorite_count, retweet_count
• id
• place
• text
• user
• more information:
https://dev.twitter.com/docs/platform-objects/tweets
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
5
Twitter Search API
• Twitter provides REST API to access tweets
• The resources are represented in RESTful style, which mean, in
theory, users can get tweets by typing the URL with right
parameters.
• Tweets, especially geo-tagged tweets
• A sample URL for query:
• https://api.twitter.com/1.1/search/tweets.json?q=shooting&geocode
=41.148693,-81.346292,100km&result_type=recent&until=201405-30&since=2014-05-29&count=100
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
6
Major query parameters
• q – the search keywords
• geocode – location with radius. Format: lat, long, 50mi.
• since, until – the search time frame. Format: yyyy-mm-dd
• count – the number of tweet in a query. <100
• result_type – types of search results preferred to receive
• mixed: Include both popular and real time results in the response.
• recent: return only the most recent results in the response.
• popular: return only the most popular results in the response.
• more information:
https://dev.twitter.com/docs/api/1.1/get/search/tweets
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
7
Authorization
• Required in Twitter API v1.1.
• Response of un-authorized request.
• {"errors":[{"message":"Bad Authentication data","code":215}]}
• Using Twitter website to generate the authorization code
• API key
• API secret
• Access token
• Access token secret
• Use and activate authorization through server-side
programming technologies, e.g. PHP, Python.
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
8
System architecture for live VGI saving
Server
for Tweets
Tweet storing
Client
Application
Tweet collection
Web
Server
Server-GIS
Application
Server
Map
Server
Live data saving
Live mapping
Geodatabase
Server
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
9
Implementation of live data saving
Tweets
(FeatureClass
Field: Keyword
Field: Location
Field: Message
:
:
:
Tweets
(GIS Layer)
Symbology
based on
keyword
FeatureServer Service
RESTful map service
ArcGIS Server
ArcGIS
JavaScript API
Geo Filter
Web GIS platform
Geo-tagged
Tweets
Twitter Search
API
Tweets
ArcGIS Desktop
Geodatabase
Load
(ArcSDE)
Publish
(ArcGIS Server)
Chen & Elmes @ West Virginia University
Write
Filter
(ArcGIS JavaScript API) (JavaScript)
2014 WVAGP GIS Conference, Charleston, WV
Collect
(Twitter API, PHP)
10
A live demo
• http://157.182.212.204/tweets
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
11
Issues
• Quality
• Accuracy
• Uncertainty
• Credibility
• Heterogeneities
• Liability
• Privacy
• Confidentiality
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
12
Thanks and questions
Chen & Elmes @ West Virginia University
2014 WVAGP GIS Conference, Charleston, WV
13