Grid Challenges It’s the vision, stupid …but it NEEDS TO be followed by operational standards based on real applications… The Global Grid Forum 25 June 2003 Gordon.
Download ReportTranscript Grid Challenges It’s the vision, stupid …but it NEEDS TO be followed by operational standards based on real applications… The Global Grid Forum 25 June 2003 Gordon.
Grid Challenges It’s the vision, stupid …but it NEEDS TO be followed by operational standards based on real applications… The Global Grid Forum 25 June 2003 Gordon Bell Microsoft Corporation A quick look at some past visions and a challenge NREN >> Internet WWW Challenge: Will match any Grid enabled application that wins a Gordon Bell Prize for parallelism FCCSET NREN Plan 11/1987 10G1G- 3G a factor of 1000 makes a difference 45 M 100M10M- 1.5 M Optical Phase 2 1M100K10K- Phase 1 56K 1988 1990 1992 1994 1996 1998 2000 3 Originating Bandwidth (Gb/s) U.S. Interstate Comm. traffic L Roberts ’92 10,000- ARPAnet Goals c1972 = Grid Goals Video Conf. 1,000- Voice 100- Video on Demand NSF bb• 10FAX Email Broadcast TV 1- |1990 | |2000 | |2010 | |20204 Growth in hype vs reality WWW books, Infoway newspapers regulation Infoway speculation “how great it’ll be” (politicians , telecoms & futurists) Infoway addiction conferences lawsuits c 1995 Data from Gordon’s WAG 5 Articles per newspaper versus orders per second sent via Internet orders per second articles per newspaper c 1995 Data from Gordon’s WAG 6 Articles about security, privacy, & fraud versus commerce ($M) actual commerce articles about risk and NOT doing commerce organized crime on Internet c 1995 Data from Gordon’s WAG 7 The virtuous cycle of bandwidth supply and demand Increased Demand Standards IP Create new service Telnet & FTP EMAIL Increase Capacity (circuits & bw) Lower response time WWW Audio Video Voice! Grids Video Conf. FTP Web Svcs 8 For More Information Grid Book c1998 from 1996 - www.mkp.com/grids The Globus Project™ - www.globus.org OGSA - www.globus.org/ogsa Global Grid Forum 651 pp. 22 chapters, 41 authors - www.gridforum.org Grid Computing 2003 1080 pages 43 chapters, O(100) authors From Carl Kesselman, ISI Progress...a review Grid started out with great promise…c1998 Interesting use at NASA for coupled programs NMI (National Middleware Infrastructure) …State_Tools.gov, funded by NSF.gov clearly open, clearly not “free” not IETF model Tools vs. standards & evolving working code Some examples: C1980: Seti@home, folding@home, >> Napster p2p 2001 15 TB Terraserver > Terraservice w/Web Services 2003 Alex Szelay & Jim Gray: Skyserver/skyservice Cornell Theory Center Web Services based apps NEES—good poster child. An XML task GRADs and Teragrid… dream or research or just $$s? TerraServer c1998: The “Whole Earth” Database TerraServer Experience c2001 Successful Web Site To the rescue! 50,000 daily users satisfied with “human-accessible” data 59 GB imagery transmitted daily New Feature Requests Programmable access to meta-data User selectable image sizes, i.e. “a map server” Permission to use TerraServer data within server applications .NET TerraService Architecture Standard Browsers Map UI Web Forms Existing DB Server Map Server Http Handler Smart Clients Windows Forms .NET Framework 668 m Rows SQL 2000 TerraServer Web Service 1.0 TB Db SQL 2000 1.0 TB Db ADO.NET OLEDB SQL 2000 1.0 TB Db Data Intensive Science: the Next Frontier The W.M. Keck Fellows in Advanced Scientific Data Analysis Alex Szalay The Johns Hopkins University Department of Physics and Astronomy National Virtual Observatory NSF ITR project, “Building the Framework for the National Virtual Observatory” is a collaboration of 17 funded and 3 unfunded organizations Astronomy data centers National observatories Supercomputer centers University departments Computer science/information technology specialists PI and project director: Alex Szalay (JHU) CoPI: Roy Williams (Caltech/CACR) Scientific Data Exploration 1. Thousand years ago: science was empirical describing natural phenomena 2. Last few hundred years: theoretical branch using models, generalizations 3. Last few decades: a computational branch simulating complex phenomena 4. Today: data exploration is emerging synthesizing theory, experiment and computation with advanced data management and statistics Living in an Exponential World Astronomers have a few hundred TB now 1 pixel (byte) / sq arc second ~ 4TB Multi-spectral, temporal, … → 1PB 1000 They mine it looking for new (kinds of) objects, more of interesting ones (quasars), density variations in 400-D space, correlations in 400-D space 100 10 1 0.1 1970 1975 Data doubles every year Data is public after 1 year So, 50% of the data is public Same trend appears in all sciences 1980 1985 1990 1995 2000 CCDs Glass Why Is Astronomy Special? IRAS 25m It has no commercial value No privacy concerns, freely share results with others Great for experimenting with algorithms It is real and well documented High-dimensional (with confidence intervals) Spatial, temporal Diverse and distributed Many different instruments from many different places and many different times The questions are interesting There is a lot of it (soon petabytes) GB: It is not over-funded aka it’s poor ROSAT ~keV 2MASS 2m DSS Optical IRAS 100m WENSS 92cm NVSS 20cm GB 6cm Making Discoveries When and where are discoveries made? Always at the edges and boundaries Going deeper, collecting more data, using more colors…. Metcalfe’s law Utility of computer networks grows as the number of possible connections: O(N2) VO: Federation of N archives Possibilities for new discoveries grow as O(N2) Current sky surveys have proven this Very early discoveries from SDSS, 2MASS, DPOSS What can be learned from Sky Server? It’s about data, not about harvesting flops 1-2 hr. query programs versus 1 wk programs based on grep 10 minute runs versus 3 day compute & searches Database viewpoint. 100x speed-ups Avoid costly re-computation and searches Use indices and PARALLEL I/O. Read / Write >>1. Parallelism is automatic, transparent, and just depends on the number of computers/disks. Limited experience and talent to use dbases. Soon: The Virtual Observatory Many new surveys are coming SDSS is a dry run for the next ones LSST will be 5TB/night All the data will be on the Internet ftp, web services… Data and applications will be associated with the instruments Distributed world wide, cross-indexed Federation is a must Will be the best telescope in the world World Wide Telescope Finds the “needle in the haystack” Successful demonstrations in Jan’03 Emerging Concepts Standardizing distributed data access Web Services, supported on all platforms XML: Extensible Markup Language SOAP: Simple Object Access Protocol WSDL: Web Services Description Language Standardizing distributed computing Grid Services Custom configure remote computing dynamically Build your own remote computer, and discard Virtual Data: new data sets on demand Both needed for Data Exploration Computational Science Simulations based on Web Services Gerd Heber Cornell Theory Center [email protected] Three Flavors of Adaptivity Application-level Algorithm-level Mathematical model High/low confidence Discretization method Solution technique System-level Resource availability Fault tolerance International Conference on Computational Science 2003 The Problem Do distributed, coupled and adaptive multi-physics simulations of Mechanics of chemically-reacting flows (Damage) Thermo-Mechanics of solids Components provided as Web Services International Conference on Computational Science 2003 International Conference on Computational Science 2003 Geography Cornell University Theory Center Department of Computer Science Department of Civil Engineering University of Alabama Mississippi State University College of William and Mary International Conference on Computational Science 2003 Workflow International Conference on Computational Science 2003 Components MiniCAD Meshers Surface (Delaunay, quality guarantees) Volume (Dmesh, Jmesh, Gmesh) Fluid/Thermal simulation (Loci, CHEM) Thermo-mechanical component (CPTC) Fracture mechanics Visualization (OpenDX + SQL Server) International Conference on Computational Science 2003 Web Services “Web Services are self-contained, modular applications that can be described, published, located, and invoked over a network, …” (IBM) Service oriented architecture: Publish, find, bind XML, SOAP, UDDI, WSDL International Conference on Computational Science 2003 Features and Requirements Distributed expertise Platform and language neutrality No porting Network accessibility (“firewall compliant”) Security Industry standards Metadata State Students shouldn’t waste too much time with coding! International Conference on Computational Science 2003 GrADS Vision • Build a National Problem-Solving System on the Grid • Software Support for Application Development on Grids • Challenges: — Transparent to the user, who sees a problem-solving system — Goal: Design and build programming systems for the Grid that broaden the community of users who can develop and run applications in this complex environment — Presenting a high-level application development interface* – If programming is hard, the Grid will not not reach its potential — Designing and constructing applications for adaptability — Late mapping of applications to Grid resources — Monitoring and control of performance – When should the application be interrupted and remapped? *GB note: This is a superset of the previously unsolved clusters programming problem! GrADSoft Architecture Performance Feedback Software Components Source Application WholeProgram Compiler Performance Problem Configurable Object Program Resource Negotiator Real-time Performance Monitor Negotiation Scheduler Binder Libraries Grid Runtime System Network for Earthquake Eng. Simulation NEESgrid: US national infrastructure to couple earthquake engineers with experimental facilities, databases, computers, & each other On-demand access to experiments, data streams, computing, archives, collaboration Carl Kesselman, ISI NEESgrid: Argonne, Michigan, NCSA, UIUC, USC From www.neesgrid.org A Universal Architecture for Web Services… Microsoft Vision Security Reliable Messaging “Scales Up” Transactions on large Routing systems … “Scales In” on a machine “Scales Down” to devices “Scales Away” spans organizations & geographies “Scales Out” by adding machines Messaging Infrastructure Distributed applications Vertical processes Embedded systems Network equipment … 39 Web Services: Level I Foundation to Build Upon Basic profile Defined by WS-I XML, SOAP, WSDL, UDDI Broad vendor support WS-I assures widespread compatibility Level II Secure, Reliable, Transacted Connected Applications Secure Reliable Messaging XML Transports … Transacted Metadata Management Business Process Level III From Infrastructure to Solutions Application schemas Domain specific profiles Vertical industry services Vison: Community/Data-Centric Computing Versus Machine-Centered Centers Goal: Enable technical communities to create and take responsibility for their own computing environments of personal, data, and program collaboration and distribution Design based on technology and cost, e.g. networking, apps programs maintenance, databases, and providing 24x7 web and other services Many alternative styles and locations are possible Service from existing centers, including many state centers Software vendors could be encouraged to supply apps web services NCAR style center based on shared data and apps Instrument- and model-based databases. Both central & distributed when multiple viewpoints create the whole. Wholly distributed services supplied by many individual groups Community/Data Centric: “web service” Community is responsible Planned & budget as resources Responsible for its infrastructure Apps are from community Computing is integral to work In sync with technologies 1-3 Tflops/$M; 1-3 PBytes/$M to buy smallish Tflops & PBytes. New scalables are “centers” Community can afford and evolve Dedicated to a community Program, data & database centric May be aligned with instruments or other community activities Output = web service; Can communities form that can supply services? Commitment to standards A general architecture comes much from understanding the problems Understanding the problems comes from actually solving such problems This is bottom-up, based on experience Microsoft is committed to develop community-wide web services standards… Is the Grid Forum equally committed? The End How can GRIDs become a real, useful, computer structure? Get a life. Use the standards and tools. Adopt an application and/or community…now!