The Future of Distributed Systems . Jim Gray Researcher Microsoft Corp. [email protected] ™ Outline Global forces Moore’s, Metcalf’s, Bell’s, Bills, Andy’s laws Micro dollars per transaction Cyber-content is key value because distribution.
Download ReportTranscript The Future of Distributed Systems . Jim Gray Researcher Microsoft Corp. [email protected] ™ Outline Global forces Moore’s, Metcalf’s, Bell’s, Bills, Andy’s laws Micro dollars per transaction Cyber-content is key value because distribution.
The Future of Distributed Systems . Jim Gray Researcher Microsoft Corp. [email protected] ™ 1 Outline Global forces Moore’s, Metcalf’s, Bell’s, Bills, Andy’s laws Micro dollars per transaction Cyber-content is key value because distribution costs go to zero Distributed Systems Concepts and terms Key software technologies objects, transactions 2 Metcalf’s Law Network Utility = Users2 How many connections can it make? 1 user: no utility 100,000 users: a few contacts 1 million users: many on Net 1 billion users: everyone on Net That is why the Internet is so “hot” Exponential benefit Moore’s First Law XXX doubles every 18 months 60% increase per year Exponential growth: Micro processor speeds Chip density Magnetic disk density Communications bandwidth WAN bandwidth approaching LAN speeds 1GB 128MB 1 chip memory size ( 2 MB to 32 MB) 8MB 1MB 128KB 8KB 1970 bits: 1K 1980 1990 4K 16K 64K 256K 1M 4M 16M 64M 256M The past does not matter 10x here, 10x there, soon you’re talking REAL change PC costs decline faster than any other platform 2000 Volume and learning curves PCs will be the building bricks of all future systems Bumps In The Moore’s Law Road $/MB of DRAM 1000000 DRAM: 1988: United States anti-dumping rules 1993-1995: ?price flat Magnetic disk: 1965-1989: 10x/decade 1989-1996: 4x/3year! 100X/decade 10000 100 1 1970 1980 1990 2000 $/MB of DISK 10,000 100 1 .01 1970 1980 1990 2000 Gordon Bell’s Seven Price Tiers 10$: 100$: 1,000$: 10,000$: 100,000$: 1,000,000$: 10,000,000$: wrist watch computers pocket/ palm computers portable computers • personal computers (desktop) departmental computers (closet) site computers (glass house) regional computers (glass castle) Super server: costs more than $100,000 “Mainframe”: costs more than $1 million Must be an array of processors, disks, tapes, comm ports Bell’s Evolution Of Computer Classes Technology enables two evolutionary paths: 1. constant performance, decreasing cost 2. constant price, increasing performance Log price Mainframes (central) Minis (dep’t.) WSs PCs (personals) Time ?? 1.26 = 2x/3 yrs -- 10x/decade; 1/1.26 = .8 1.6 = 4x/3 yrs --100x/decade; 1/1.6 = .62 Software Economics An engineer costs about $150,000/year R&D gets [5%…15%] of budget Need [$3 million… $1 million] revenue per engineer Intel: $16 billion Profit 22% R&D 8% SG&A 11% Tax 12% P&S 47% Microsoft: $9 billion Profit 24% R&D 16% SG&A 34% Tax 13% Product and Service 13% IBM: $72 billion Profit Tax 6% 5% R&D 8% Oracle: $3 billion Profit 15% Tax 7% SG&A 22% P&S 59% P&S 26% R&D 9% SG&A 43% Software Economics: Bill’s Law Fixed_Cost Price = + Marginal _Cost Units Bill Joy’s law (Sun): don’t write software for less than 100,000 platforms @$10 million engineering expense, $1,000 price Bill Gate’s law: don’t write software for less than 1,000,000 platforms @$10 engineering expense, $100 price Examples: UNIX versus Windows NT: $3,500 versus $500 Oracle versus SQL-Server: $100,000 versus $6,000 No spreadsheet or presentation pack on UNIX/VMS/... Commoditization of base software and hardware Gordon Bell’s Platform Economics Traditional computers: custom or semi-custom, high-tech and high-touch New computers: high-tech and no-touch 100000 10000 Price (K$) Volume (K) Application price 1000 100 10 1 0.1 0.01 Mainframe WS Computer type Browser Grove’s Law The New Computer Industry Horizontal integration is new structure Each layer picks best from lower layer Desktop (C/S) market 1991: 50% 1995: 75% Function Operation Integration Applications Middleware Baseware Systems Silicon & Oxide Example AT&T EDS SAP Oracle Microsoft Compaq Intel & Seagate Outline Global forces Moore’s, Metcalf’s, Bell’s, Bills, Andy’s laws Micro dollars per transaction Cyber-content is key value because distribution costs go to zero Distributed Systems Concepts and terms Key software technologies objects, transactions 12 1987: 256 tps Benchmark 14 M$ computer (Tandem) A dozen people False floor, 2 rooms of machines Admin expert Hardware experts A 32 node processor array Simulate 25,600 clients Network expert Manager Performance expert DB expert Auditor OS expert A 40 GB disk array (80 drives) 13 1997: 10 years later 1 Person and 1 box = 1250 tps 1 Breadbox ~ 5x 1987 machine room 23 GB is hand-held One person does all the work Cost/tps is 1,000x less 1 micro dollar per transaction Hardware expert OS expert Net expert DB expert App expert 4x200 Mhz cpu 1/2 GB DRAM 12 x 4GB disk 3 x7 x 4GB disk arrays 15 What Happened? Moore’s law: Things get 4x better every 3 years (applies to computers, storage, and networks) New Economics: Commodity class price/mips software $/mips k$/year mainframe 10,000 100 minicomputer 100 10 microcomputer 10 1 time GUI: Human - computer tradeoff optimize for people, not computers 16 What Happens Next Last 10 years: 1000x improvement Next 10 years: ???? Today: 1985 1995 text and image servers are free 1 m$/hit cost 70,000m$/hit advertising revenue Advertising pays for them Content is only “real” expense “You ain’t seen nothing yet!” 2005 17 Kinds Of Information Processing Point-to-point Immediate Timeshifted Broadcast Conversation Money Lecture Concert Network Mail Book Newspaper Database It’s ALL going electronic Immediate is being stored for analysis (so ALL database) Analysis and automatic processing are being added Low rent min $/byte Shrinks time now or later Shrinks space here or there Automate processing knowbots Immediate OR time-delayed Why Put Everything In Cyberspace? Point-to-point OR broadcast Network Locate Process Analyze Summarize Database Billions Of Clients Every device will be “intelligent” Doors, rooms, cars… Computing will be ubiquitous Billions Of Clients Need Millions Of Servers All clients networked to servers May be nomadic or on-demand Fast clients want faster servers Servers provide Shared Data Control Coordination Communication Clients Mobile clients Fixed clients Servers Server Super server Thesis Many little beat few big $1 million 3 1 MM $100 K $10 K Pico Processor Micro Mini Mainframe Nano 1 MB 10 pico-second ram 10 nano-second ram 100 MB 10 GB 10 microsecond ram 1 TB 14" 9" 5.25" 3.5" 2.5" 1.8" 10 millisecond disc 100 TB 10 second tape archive Smoking, hairy golf ball How to connect the many little parts? How to program the many little parts? Fault tolerance? 1 M SPECmarks, 1TFLOP 106 clocks to bulk ram Event-horizon on chip VM reincarnated Multiprogram cache, On-Chip SMP Future Super Server: 4T Machine Array of 1,000 4B machines 1 bps processors 1 BB DRAM 10 BB disks 1 Bbps comm lines 1 TB tape robot A few megabucks Challenge: Manageability Programmability CPU 50 GB Disc 5 GB RAM Cyber Brick a 4B machine Security Availability Scaleability Affordability As easy as a single system Future servers are CLUSTERS of processors, discs Distributed database techniques make clusters work The Hardware Is In Place… And then a miracle occurs ? SNAP: scaleable network and platforms Commodity-distributed OS built on: Commodity platforms Commodity network interconnect Enables parallel applications Outline Global forces Moore’s, Metcalf’s, Bell’s, Bills, Andy’s laws Micro dollars per transaction Cyber-content is key value because distribution costs go to zero Distributed Systems Concepts and terms Key software technologies objects, transactions 25 Outline Concepts and Terminology Why Distributed Distributed data & objects Distributed execution Three tier architectures Transaction concepts 26 What’s a Distributed System? Centralized: everything in one place stand-alone PC or Mainframe Distributed: some parts remote distributed users distributed execution distributed data 27 Why Distribute? No best organization Companies constantly swing between Centralized: focus, control, economy Decentralized: adaptive, responsive, competitive Why distribute? reflect organization or application structure empower users / producers improve service (response / availability) distributed load use PC technology (economics) 28 What Should Be Distributed? Users and User Interface Thin client Presentation Processing workflow Data Trim client Fat client Business Objects Database Will discuss tradeoffs later 29 Transparency in Distributed Systems Make distributed system as easy to use and manage as a centralized system Give a Single-System Image Location transparency: hide fact that object is remote hide fact that object has moved hide fact that object is partitioned or replicated Name doesn’t change if object is replicated, partitioned or moved. 30 Outline Concepts and Terminology Why Distribute Distributed data & objects Partitioned Replicated Distributed execution remote procedure call queues Three tier architectures Transaction concepts 44 Distributed Execution Threads and Messages Thread is Execution unit threads (software analog of cpu+memory) Threads execute at a node Threads communicate via Shared memory (local) Messages (local and remote) shared memory messages 45 Peer-to-Peer or Client-Server Peer-to-Peer is symmetric: Either side can send Client-server client sends requests server sends responses simple subset of peer-to-peer 46 Remote Procedure Call: The key to transparency y = pObj->f(x); Object may be local or remote Methods on object work wherever it is. Local invocation x f() return val; y = val; val 48 Remote Procedure Call: The key to transparency Remote invocation y = pObj->f(x); x proxy Obj Local? x marshal stub x un marshal pObj->f(x) f() x Obj Local? f() return val; y = val; val val return val; un marshal val marshal val 49 Object Request Broker (ORB) Orchestrates RPC Registers Servers Manages pools of servers Connects clients to servers Does Naming, request-level authorization, Provides transaction coordination (new feature) Old names: Transaction Processing Monitor, Web server, Transaction NetWare Object-Request Broker 50 History and Alphabet Soup 1995 CORBA Solaris Object Management Group (OMG) 1990 X/Open UNIX International 1985 Open software Foundation (OSF) Microsoft DCOM based on OSF-DCE Technology DCOM and ActiveX extend it Open Group OSF DCE NT COM 51 ActiveX and COM COM is Microsoft model, engine inside OLE ALL Microsoft software is based on COM (ActiveX) CORBA + OpenDoc is equivalent Heated debate over which is best Both share same key goals: Encapsulation: hide implementation Polymorphism: generic operations key to GUI and reuse Versioning: allow upgrades Transparency: local/remote Security: invocation can be remote Shrink-wrap: minimal inheritance Automation: easy COM now managed by the Open Group Linking And Embedding Objects are data modules; transactions are execution modules Link: pointer to object somewhere else Think URL in Internet Embed: bytes are here Objects may be active; can callback to subscribers Bottom Line Re ORBs Microsoft Promises Cairo distributed objects, secure, transparent, fast invocation Netscape promises the CORBA Both will deliver Customers can pick the best one Transaction Object-Request Broker 54 Outline Concepts and Terminology Why Distributed Distributed data & objects Distributed execution Three tier architectures remote procedure call queues what why Transaction concepts 57 Work Distribution Spectrum Thin Presentation and plug-ins Workflow manages session & invokes objects Business objects Database Fat Presentation workflow Business Objects Database Fat Thin 61 Transaction Processing Evolution to Three Tier Intelligence migrated to clientsMainframe Mainframe Batch processing (centralized) Dumb terminals & Remote Job Entry cards green screen 3270 TP Monitor Intelligent terminals database backends Workflow Systems Object Request Brokers Application Generators Server ORB Active 62 Web Evolution to Three Tier Intelligence migrated to clients (like TP) Web WAIS Character-mode clients, smart servers Server archie ghopher green screen Mosaic GUI Browsers - Web file servers GUI Plugins - Web dispatchers - CGI Smart clients - Web dispatcher (ORB) pools of app servers (ISAPI, Viper) workflow scripts at client & server NS & IE Active 63 PC Evolution to Three Tier Intelligence migrated to server Stand-alone PC (centralized) PC + File & print server message per I/O PC + Database server message per SQL statement PC + App server message per transaction IO request reply disk I/O SQL Statement Transaction ActiveX Client, ORB ActiveX server, Xscript 64 The Pattern: Three Tier Computing Presentation Clients do presentation, gather input Clients do some workflow (Xscript) Clients send high-level requests to ORB (Object Request Broker) ORB dispatches workflows and business objects -- proxies for client, Business Objects orchestrate flows & queues Server-side workflow scripts call on distributed business objects to execute task workflow Database 65 The Three Tiers Web Client HTML VB Java plug-ins VBscritpt JavaScrpt Middleware VB or Java Script Engine Object server Pool VB or Java Virt Machine Internet HTTP+ DCOM ORB ORB TP Monitor Web Server... Object & Data server. DCOM (oleDB, ODBC,...) IBM Legacy Gateways 66 Why Did Everyone Go To Three-Tier? Manageability Business rules must be with data Middleware operations tools Performance (scaleability) workflow Server resources are precious ORB dispatches requests to server pools Technology & Physics Presentation Put UI processing near user Put shared data processing near shared data Business Objects Database 67 Why Put Business Objects at Server? MOM’s Business Objects DAD’sRaw Data Customer comes to store Takes what he wants Fills out invoice Leaves money for goods Easy to build No clerks Customer comes to store with list Gives list to clerk Clerk gets goods, makes invoice Customer pays clerk, gets goods Easy to manage Clerks controls access Encapsulation 68 What Middleware Does ORB, TP Monitor, Workflow Mgr, Web Server Registers transaction programs workflow and business objects (DLLs) Pre-allocates server pools Provides server execution environment Dynamically checks authority (request-level security) Does parameter binding Dispatches requests to servers parameter binding load balancing Provides Queues Operator interface 69 Server Side Objects Easy Server-Side Execution A Server ORB gives simple execution environment Object gets Network start invoke shutdown Everything else is automatic Drag & Drop Business Objects Queue Connections Context Security Thread Pool Configuration Management Receiver Service logic Synchronization Shared Data 70 A new programming paradigm Develop object on the desktop Better yet: download them from the Net Script work flows as method invocations All on desktop Then, move work flows and objects to server(s) Gives desktop development three-tier deployment Software Cyberbricks Why Server Pools? Server resources are precious. Clients have 100x more power than server. Pre-allocate everything on server preallocate memory pre-open files pre-allocate threads N clients x N Servers x F files = N x N x F file opens!!! pre-open and authenticate clients Keep high duty-cycle on objects (re-use them) Pool threads, not one per client Classic example: TPC-C benchmark IE 2 processes everything pre-allocated Pool of DBC links HTTP 7,000 clients IIS SQL 72 order entry , payment , status (oltp) delivery (mini-batch) restock (mini-DSS) Metrics: Throughput, Price/Performance Shows best practices: everyone three tier 2 processes at server everything pre-allocated HTTP Transaction Processing Performance Council (TPC): standard performance benchmarks 5 transaction types IIS = Web Pool of DBC links ODBC Classic Three-Tier Example TPC-C 7,000 Web clients SQL 73 Outline Laws & micro$/transaction Distributed Systems Why Distributed Distributed data & objects Distributed execution Three tier architectures why: manageability & performance what: server side workflows & objects Transaction concepts Why transactions? Using transactions 75 Thesis Transactions are key to structuring distributed applications ACID properties ease exception handling Atomic: all or nothing Consistent: state transformation Isolated: no concurrency anomalies Durable: committed transaction effects persist 76 What Is A Transaction? Programmer’s view: Bracket a collection of actions A simple failure model Only two outcomes: Begin() action action action action Commit() Success! Begin() Begin() action action action action action action Rollback() Fail ! Rollback() Failure! 77 Why ACID For Client/Server And Distributed ACID is important for centralized systems Failures in centralized systems are simpler In distributed systems: More and more-independent failures ACID is harder to implement That makes it even MORE IMPORTANT Simple failure model Simple repair model 81 Outline Why Distributed Distributed data & objects Distributed execution Three tier architectures Transaction concepts Why transactions? Using transactions programming workflow 90 References Essential Client/Server Survival Guide 2nd ed. Client/Server Programming with Java and CORBA Orfali, Harkey, J Wiley, 1997 Principles of Transaction Processing Orfali, Harkey & Edwards, J. Wiley, 1996 Bernstein & Newcomer, Morgan Kaufmann, 1997 Transaction Processing Concepts and Techniques Gray & Reuter, Morgan Kaufmann, 1993 91 ™ 92