Transcript What to Do? A Research Agenda Jim Gray Microsoft Research “Everything that can be invented has been invented.” Commissioner, U.S.

What to Do?
A Research Agenda
Jim Gray
Microsoft Research
“Everything that can be invented has been invented.”
Commissioner, U.S. Office of Patents, 1899
From http://inventors.about.com/library/lessons/bl_appendix5.htm
…absolutely no basis to support Duell's alleged statement.
Just the opposite is true.
Duell's 1899 report documents an increase of about 3,000
patents over the previous year, and
nearly 60 times the number granted in 1837.
Further, Duell quotes President McKinley's annual message saying, "Our future
progress and prosperity depend upon our ability to equal, if not surpass, other nations
in the enlargement and advance of science, industry and commerce. To invention we
must turn as one of the most powerful aids to the accomplishment of such a result."
Duell adds, "May not our inventors hopefully look to the Fifty-sixth Congress for aid
and effectual encouragement in improving the American patent system?" These are
unlikely words of someone who thinks that everything has been invented.
“We have patents on the Byte and the Algorithm”
Dave Huffman
•
•
•
•
EE has the electron
Physics has matter/energy
Chemistry has molecules and reactions
Economics has the transaction.
• They all need us to do anything
• The IT revolution is just starting.
Science The Endless Frontier
Vannevar Bush -> Harry Truman, July 1945
1. Introduction:
4.
Scientific Progress is Essential
Science is a Proper Concern of Government
Government Relations to Science - Past and Future
Freedom of Inquiry Must be Preserved
2.
The War Against Disease:
In War
In Peace
Unsolved Problems
Broad and Basic Studies Needed
Coordinated Attack on Special Problems
Action is Necessary
3.
Science and the Public Welfare:
Nature of the Problem
A Note of Warning
The Wartime Deficit
Improve the Quality
Remove the Barriers
The Generation in Uniform Must Not be Lost
A Program
5. A Problem of Scientific Reconversion:
Effects of Mobilization of Science for War
Security Restrictions Should be Lifted Promptly
Need for Coordination
A Board to Control Release
Publication Should be Encouraged
6. The Means to the End:
Relation to National Security
Science and Jobs
The Importance of Basic Research
Centers of Basic Research
Research Within the Government
Industrial Research
International Exchange of Scientific Information
The Special Need for Federal Support
The Cost of a Program
Renewal of our Scientific Talent:
New Responsibilities for Government
The Mechanism
Five Fundamentals
Military Research
National Research Foundation
http://www.nsf.gov/od/lpa/nsf50/vbush1945.htm
Request From George Djorgovski
• What do you think we (CACR) should do?
– Long term
– 2 or 3 focus areas
– Leverage our skills.
Honest Answer
• I do not know.
•
•
•
•
•
•
Hire smart over-achievers
Give them (barely) enough resources
Ask them what they have accomplished
Foster mutual respect
Take credit for the successes
Allow some of them to fail.
Talking the Talk and
Walking the Walk
• We have 7 people – what’s our agenda?
• Observe it is all about:
Data → Information → Knowledge → Wisdom
People == Communication is the “killer app”
• ½ Personal Information Management
• ½ Corporate Information Management
Memex
As We May Think, Vannevar Bush, 1945
“A memex is a device in which an individual
stores all his books, records, and
communications, and which is mechanized
so that it may be consulted with exceeding
speed and flexibility”
“yet if the user inserted 5000 pages of
material a day it would take him hundreds
of years to fill the repository, so that he can
be profligate and enter material freely”
25Kday life ~ Personal Petabyte
Lifetime Storage
1PB
1000.
100.
10.
TB
1.
0.1
0.01
0.001
Msgs
web
pages
Tifs
Books
jpegs
1KBps
sound
music
Videos
Will anyone look at web pages in 2020?
Probably new modalities & media will dominate then.
Challenges
•
•
•
•
•
•
•
•
Capture: Get the bits in
Organize: Index them
Manage: No worries about loss or space
Curate/ Annotate: automate where possible
Privacy: Keep safe from theft / disclosure.
Summarize: Give thumbnail summaries
Interface: how ask/anticipate questions
Present: show it in understandable ways.
MyLifeBits Software
Radio
capture tool
TV capture
tool
Telephone
capture tool
MyLifeBits
store
Internet
TV EPG
download
tool
database
Browser
tool
MyLifeBits
Shell
PocketPC
transfer
tool
PocketRadio
player
Radio EPG
tool
MAPI
interface
Legacy
email client
files
Legacy
applications
IM capture
Voice
annotation
tool
Text
annotation
tool
Import files
MyLifeBits Interesting Ideas
http://www.research.microsoft.com/barc/mediapresence/MyLifeBits.aspx
• Capture is “easy”
– All TV, all Phone, all web pages, all mail,….
– Senscam: a picture a minute
– GPS corrleation (and other correlation)
• Interesting search strategies
– Pivot and cluster
• Struggling with metadata (annotations)
• Interesting visualizations
(ambiance, timelines, spatial, conceptual).
• Very topical
interest from public, industry, developers
Relevance To CACR
• (re)Invent the laboratory notebook
• Scientists are still mostly working with paper
and pencil (especially in the lab).
• Change the way scientists do information
management.
• Give them good analysis/visualization tools
• Allow them to publish their data/workbooks
80% of data is personal / individual.
But, what about the other 20%?
• Business
– Wall Mart online: 1PB and growing….
– Paradox: most “transaction” systems < 1 PB.
– Have to go to image/data monitoring for big data
• Government
– Government is the biggest business.
• Science
– LOTS of data.
Data Challenges I'm
Struggling With
1. Sneakernet is probably the best way to moving WAN data at 1GBps
File transfer efforts are currently 550MBps via Internet2.
How to manage the multi-petybyte file repository we are about to generate.
2. The TerraServer has evolved from a mainframe to a bunch of bricks.
The new design has been operating for a year and we are quite pleased with it.
But we face "how-do-you-manage a bunch?" and what the best geoplex strategy?.
3. The SkyServer website is built using database technology and web services.
Now moving the web services inside the database.
Others are working to design a scale-out version of the server.
There are several interesting data challenges in these changes.
4. Using relational tuples to represent spatial volumes as constraints.
Point-in-polygon and polygon-overlap queries can then be quickly evaluated.
I will briefly describe this idea.
How Do You Move A Terabyte?
Context
Speed
Rent
$/TB
$/Mbps
Mbps $/month
Sent
Time/TB
Home phone 0.04
40
1,000
3,086
6 years
Home DSL
0.6
50
117
360
5 months
T1
1.5
1,200
800
2,469
2 months
T3
43
28,000
651
2,010
2 days
OC3
155
49,000
316
976
14 hours
OC 192
9600
1,920,000
200
617
14 minutes
100 Mpbs
100
1 day
Gbps
1000
2.2 hours
Source: TeraScale Sneakernet, Microsoft Technical Report May 2002, MSR-TR-2002-54
http://research.microsoft.com/research/pubs/view.aspx?tr_id=569
Moving Data Bricks
• WAN costs >> 100$/Mbps/month
>> 1$/GB
• Beowulf networking
10,000x cheaper than WAN
factors of 105 matter.
• The cheapest and fastest way
to move a Terabyte cross country
is sneakernet.
24 hours = 4 MB/s
50$ shipping vs 1,000$ wan cost.
Giga Byte Per Second File
Mover
• CERN to Pasadena
– Windows TCP/IP stack improvements
– Opteron demo
– Disk-to-Disk at 550MBps now (~2 TB/Hour)
– Near the PCI-X limit.
• GOAL: 1GBps disk-to-disk
– 75% there
CERN-Caltech Trasfer Speeds
CERN-Caltech Trasfer Speeds
Newisys->Newisys
Newisys->Newisys
1000
900
800
PCI -X limit
limit
MBps
MBps
1000
900
800
tcp
700
600
500
400
300
200
100
0
Mar-04
File Transfer MBps
1 Stream tcp MBps
May-04
Jun-04
Aug-04
Sep-04
700
600
500
400
300
200
100
0
Mar-04
File Transfer MBps
1 Stream tcp MBps
Jun-04
Sep-04
Jan-05
Apr-05
But then what?
Managing Petabytes
•
•
•
•
•
•
CERN files are 30MB
They produce 1 B files/year.
How name them?
How manage them?
Depends on workload: how use them.
It’s a DB problem.
Data Challenges I'm
Struggling With
Jim Gray, Microsoft Research
1. Sneakernet is probably the best way to moving WAN data at 1GBps
File transfer efforts are currently 550MBps via Internet2.
How to manage the multi-petybyte file repository we are about to generate.
2. The TerraServer has evolved from a mainframe to a bunch of bricks.
The new design has been operating for a year and we are quite pleased with it.
But we face "how-do-you-manage a bunch?" and what the best geoplex strategy?.
3. The SkyServer website is built using database technology and web services.
Now moving the web services inside the database.
Others are working to design a scale-out version of the server.
There are several interesting data challenges in these changes.
4. Using relational tuples to represent spatial volumes as constraints.
Point-in-polygon and polygon-overlap queries can then be quickly evaluated.
I will briefly describe this idea.
TerraServer / TerraService
http://terraService.Net/ http://TerraServer-USA.com/
• USGS Photo of US
• Online since June 1998
• Operated by Microsoft
• 20 TB data source
• 10 M web hits/day
• A web service
• Our laboratory
TerraServer – What’s new
• Web Service and Web Server
• New ~1 ft2/pixel full color image
of 120 urban areas
• Storage Bricks
–
–
–
–
–
–
–
Commodity servers”
4 TB raw / 2 TB Raid1 SATA storage
Dual 2 GHz + 4GB RAM
3 Bricks = TerraServer data
Data partitioned
Moving to Yukon
Working on low TCO
auto-manage
• Low Cost Availability Pair & Spare
–
–
–
–
RAID1 Mirroring
Mirrored Bunches (Yukon log ship?)
Spare Brick
Web Application
• Load balances mirrors
• Uses surviving database on failure
KVM / IP
TerraServer Challenges
• Best Geoplex strategy?
• Moving Web Services into the DB?
• Managing bunches (lower TCO).
World Wide Telescope
• Premise: Most Astronomy data is online
• So, the Internet is the world’s best
telescope:
– It has data on every part of the sky
– In every measured spectral band:
– As deep as the best instruments (2 years
ago).
– It is up when you are up.
The “seeing” is always great
(no working at night, no clouds no moons
no..).
– It’s a smart telescope:
links objects and data to literature on them.
SkyServer.SDSS.org
Built with Johns Hopkins U.
• A modern archive
– Raw Pixel data lives in file servers
– Catalog data (derived objects) in Database
– Online query to any and all
• Also used for education
– 150 hours of online Astronomy
– Implicitly teaches data analysis
• Interesting things
–
–
–
–
–
Spatial data search
Query interface via Emacs, Perl, Java…
Popular -- 1% of Terraserver 
Cloned by other surveys (a template design)
Based on Web Services
Quick Overview (Services)
• SkyServer (skyserver.sdss.org)
– Web site delivers Sloan Digital Sky Survey data
– Also has education
– 1,000x less popular than Terraserver,
but HUGE for a science website.
• A Batch Job System with Personal DBs
– Lets users run jobs http://casjobs.sdss.org/CasJobs/
– Parameters & Answers to & from Personal DB
– Simple batch job scheduler.
• Web Services: http://www.voservices.org/
–
–
–
–
Photographic objects
Spectrographic objects
Transformation functions
7 out of the 8 are .NET.
Federation: SkyQuery.Net
• Combines 15 archives
• Send query to portal,
portal joins data from archives.
• Evolving Portal to have
– Personal databases (workbenches)
– Batch scheduling of monster queries.
Image
Cutout
SDSS
2MASS
SkyQuery
Portal
FIRST
INT
The Data Challenges
• Parallel data search (data pump).
How to partition?
How manage load
• Moving web services to DB
What is the right approach?
• Move objects into DB
Spatial access methods
Data analysis in the DB.
• Managing Petabytes
The Knowledge Challenge
• We need to “objectify science”
• What is a Gene? Star? River? Cell? …
– What is the definition (formal)
– What attributes do they have?
– What are their dynamics?
• Defining this (the “O” word) has to happen
in each discipline.
• This is the Knowledge level
(rather than the data level)
Data Challenges I'm
Struggling With
1. Sneakernet is probably the best way to moving WAN data at 1GBps
File transfer efforts are currently 550MBps via Internet2.
How to manage the multi-petybyte file repository we are about to generate.
2. The TerraServer has evolved from a mainframe to a bunch of bricks.
The new design has been operating for a year and we are quite pleased with it.
But we face "how-do-you-manage a bunch?" and what the best geoplex strategy?.
3. The SkyServer website is built using database technology and web services.
Now moving the web services inside the database.
Others are working to design a scale-out version of the server.
There are several interesting data challenges in these changes.
4. Using relational tuples to represent spatial volumes as constraints.
Point-in-polygon and polygon-overlap queries can then be quickly evaluated.
I will briefly describe this idea.
A Detail: 3 Ways We Do Spatial?
• Hierarchical mesh (extension to SQL)
– Uses table valued stored procedures
– Acts as a new “spatial access method”
– Porting to Yukon CLR for a 10x speedup.
• Zones: fits SQL like a glove
– Amazingly simple, amazingly good.
• Constraints: a really novel idea
– Lets us do algebra on regions.
• Paper:There Goes the Neighborhood: Relational Algebra for Spatial Data Search
• Idea in backup slides.
Equations Define Subspaces
y
• For (x,y) above the line
ax+by > c
• Reverse the space by
-ax + -by > -c
• Intersect a 3 volumes:
a1x + b1y > c1
a2x + b2y > c2
a3x + b3y > c3
x=c/a
x
y=c/b
y
x
HTM Approach
• Table-valued function
find points near a point
– Select * from fGetNearbyEq(ra,dec,r)
• Use Hierarchical Triangular Mesh www.sdss.jhu.edu/htm/
– Space filling curve, bounding triangles…
– Standard approach
• 13 ms/call… So 70 objects/second.
• Too slow, so precompute neighbors:
Materialized view.
• At 70 objects/sec
it takes 6 months to compute a billion objects.
Areas defined by String
(or struct in C# world)
•
•
•
•
•
•
•
•
•
•
•
circleSpec :=
|
rectSpec :=
polySpec :=
|
hullSpec :=
|
convexSpec
regionSpec
areaSpec :=
|
CIRCLE J2000 ra dec radArcMin
CIRCLE CARTESIAN x y z radArcMin
RECT J2000 {ra dec}2
POLY J2000 {ra dec}3+
POLY CARTESIAN { x y z }3+
CHULL J2000 {ra dec}3+
CHULL CARTESIAN { x y z }3+
:=
CONVEX { x y z d}+
:=
REGION { convexSpec }+
circleSpec | rectSpec | polySpec
hullSpec | regionSpec
Working in 3D
Avoids Spherical Geometry
• P = (px,py,pz)
Inside circle C centered at x,y,z,
With radius r radians if
P•C > cos(r)
• Arbitrary polygons
are intersections of
these regions.
Cos(r)
Find Points Inside Area
• fGetNearbyObjEq(ra,dec,r)
fGetNearbyObjXyz(x,y,z,r)
fGetNearestObjEq(ra,dec,r)
fGetNearestObjEq(x,y,z,r)
• fGetObjInside(region)
• Recently Alex added Healpix,
Igloo is also a nice iso-area decomposition
HTM reprise
• Good for point-in-area.
• Not good for area-overlaps-area
(but can simplify areas and test for empty)
To Repeat (for area algebra)
Equations Define Subspaces
y
• For (x,y) above the line
ax+by > c
• Reverse the space by
-ax + -by > -c
• Intersect a 3 volumes:
a1x + b1y > c1
a2x + b2y > c2
a3x + b3y > c3
x=c/a
x
y=c/b
y
x
Domain is Union of Convex Hulls
Not a
convex hull
+
• Simple volumes are
unions of convex hulls.
• Higher order curves also
work
• Complex volumes have
holes and their holes
have holes. (that is
harder).
Now in Relational Terms
create table HalfSpace (
domainID
int not null
-- domain name
foreign key references Domain(domainID),
convexID int not null,
-- grouping a set of ½ spaces
halfSpaceID int identity(), -- a particular ½ space
x
float not null,
-- the (a,b,..) parameters
y
float not null,
-- defining the ½ space
z
float not null,
c
float not null,
-- the constant (“c” above)
primary key (domainID, convexID, halfSpaceID)
(x,y,z) inside a convex if it is inside all
lines of the convex
(x,y,z) inside a convex if it is NOT OUTSIDE ANY line of the convex
select convexID
from HalfSpace
where @x * x + @y * y + @x * z <
group by all convexID
having count(*) = 0
-- return the convex hulls
-- from the constraints
l -- point outside the line?
-- consider all the lines of a convexID
-- count outside == 0
The Algebra is Simple (Boolean)
@domainID = spDomainNew (@type varchar(16), @comment varchar(8000))
@convexID = spDomainNewConvex (@domainID int)
@halfSpaceID = spDomainNewConvexConstraint (@domainID int, @convexID int,
@x float, @y float, @z float, @l float)
@returnCode = spDomainDrop(@domainID)
select * from fDomainsContainPoint(@x float, @y float, @z float)
Once constructed they can be manipulated with the Boolean operations.
@domainID = spDomainOr (@domainID1 int, @domainID2 int,
@type varchar(16), @comment varchar(8000))
@domainID = spDomainAnd (@domainID1 int, @domainID2 int,
@type varchar(16), @comment varchar(8000))
@domainID = spDomainNot (@domainID1 int,
@type varchar(16), @comment varchar(8000))
What! No Bounding Box?
• Bounding box limits search.
A subset of the convex hulls.
• If query runs at 3M halfspace/sec then no
need for bounding box,
unless you have more than 10,000 lines.
• But, if you have a lot of half-spaces
then bounding box is good.
Zone Approach
• Divide space into zones
• Key points by Zone, offset
(on the sphere this need wrap-around margin.)
• Point search
look in a few zones
at a limited offset: ra ± r
a bounding box that has
1-π/4 false positives
• All inside the relational engine
• Avoids “impedance mismatch”
r
• Can “batch” all-all comparisons
• 33x faster and parallel
√(r +(ra-zoneMax) )
cos(radians(zoneMax))
6 days, not 6 months!
2
ra-zoneMax
x
2
zoneMax
Ra ± x
In SQL
select o1.objID
from zone o1
where o1.zoneID between
floor((@dec-@r)/@zoneHeight) and
floor((@dec+@r)/@zoneHeight)
and o1.ra between @ra - @r and @ra + @r
and o1.dec between @dec-@r and @dec+@r
-- find objects
-- in the zoned table
-- where zone #
-- overlaps the circle
-- quick filter on ra
-- quick filter on dec
Bounding
box
and ( (sqrt( power(o1.cx-@cx,2)+power(o1.cy-@cy,2)+power(o1.cz-@cz,2))))
< @r
-- careful filter on distance
Eliminates the
~ 21% = 1-π/4
False positives
Summary
• SQL is a set oriented language
• You can express constraints as rows
• Then You
– Can evaluate LOTS of predicates per second
– Can do set algebra on the predicates.
• Benefits from SQL parallelism
• SQL == Prolog? 
Talking the Talk and
Walking the Walk
• We have 7 people – what’s our agenda?
• Observe it is all about:
Data → Information → Knowledge → Wisdom
People == Communication is the “killer app”
• ½ Personal Information Management
• ½ Corporate Information Management