Transcript Document
Lean SOA Enterprise Refactoring Collaborative Service Specification by Example Presented by Edward Ost © Talend 2011 1 Purpose – Enterprise Refactoring Leverage agile Behavior Driven Design (BDD) principles for service specification in a collaborative SOA Governance environment to implement Lean Continuous Delivery to incrementally transform the Enterprise © Talend 2011 2 Agenda Lean SOA Continuous Delivery for SOA Semantic Contracts SOA Evolution at DISA © Talend 2011 3 Narrowing the Agility Gap Critical Architecture Dependencies Business Business Processes A • Infrastructure must be controlled to guarantee quality of service for mission critical processes • Lack of common understanding of requirements • Business processes evolve • Rigid IT hinders innovations B Applications IT Infrastructure Agility is the ability to adapt faster than the frequency of change Key Concepts: Federated EHR Services Architecture EHR IS Enabled Legacy System INTEGRATED EHR SOLUTION EHR IS EHR SERVICES PLATFORM (ESP) Tier 3 Plug-&-Play Data Stores Population Health Data & Services Population Health Data & Services EHR Data & Services Registries Data & Services EAI IP Tier 2 EHR Information Services Tier 1 Plug-&-Play Applications EHR IS Enabled PHR System Information Access & Longitudinal Record Services (LRS) EHR IS Locator ESP Based System Data Access Virtualization Layer EHR Information Services (EHR IS) EHR IP EHR IP EHR IP Point of Service Applications Personal Health Record Systems EHR Viewers Application Virtualization Layer IP is SOA Interoperability Profile aka Service Interface EAI is Enterprise Application Integration 5 CIO Conference: iEHR, VLER, ESB v3 See notes page 11/25/2011 Legacy Transition Strategy INTEGRATED EHR SOLUTION LEGACY EHR SYSTEM EHR SERVICES PLATFORM (ESP) EHR-IS ENABLED LEGACY INFRASTRUCTURE Population Health Data & Services Data Warehouse Data & Services EHR Data & Services Legacy Point of Service Application Registries Data & Services Data Warehouse Data & Services EHR Data & Services Personal Health Record System Legacy EAI IP EHR-IS EAI IP Information Access & Longitudinal Record Services (LRS) Information Access & Longitudinal Record Services (LRS) EAI IP EHR Information Services (EHR IS) EHR IP EHR IP EHR IP Point of Service Application Personal Health Record Systems EHR-IS Bi-Directional Information Exchange EHR Information Services (EHR IS) EHR Viewer EAI is Enterprise Application Integration IP is SOA Interoperability Profile aka Service Interface EHR-IS enabled legacy systems allow users to transition at a convenient time and then legacy systems can be shut down. 6 CIO Conference: iEHR, VLER, ESB v3 6 11/25/2011 EHR Information Services (EHR IS) Within EHR Services Platform (ESP) EHR SERVICES PLATFORM (ESP) ORGANISATIONAL INFOSTRUCTURE Population Health Data & Services Registries Data & Services Client Registry Outbreak Management EHR Data & Services PHS Reporting Shared Health Record Drug Information Data Warehouse & Services Diagnostic Imaging Health Information Laboratory Provider Registry Location Registry Business Rules EHR Index Terminology Registry Message Structures Normalization Rules Information Access & Longitudinal Record Services (LRS) IP is SOA Interoperability Profile aka Service Interface Security Mgmt Privacy Mgmt Configuration Rules/Data Rules/Data Rules/Data EAI is Enterprise Application Integration Common Information Interoperability Framework (CIIF) and Common Services EAI IP EHR IS EAI IP Secure Communications Bus EHR IP Public Health Services Public Health Provider POINT OF SERVICE APPLICATIONS 7 CIO Conference: iEHR, VLER, ESB v3 EHR IP EHR IP Pharmacy System Radiology Center PACS/RIS Pharmacist EHR IP Radiologist Lab System (LIS) Lab Clinician EHR IP EHR IP Hospital, LTC, CCC, EPR Physician/ Provider EHR IP Physician Office EMR Physician/ Provider EHR Viewer Physician/ Provider See notes page 11/25/2011 SOA Increases Project Dependencies Partial Function Incomplete Data Integrate via Web Services Orchestrate Cleanse data MDM Hub ESB Analyze data CRM Production Budgeting New Accounting Confidential Data Future Release Extract, Transform and Load data Migrate legacy applications Sales Datamart DWH © Talend 2011 Stale Data Access Fees Exchange data with partners Sensitive Data Mkt Datamart Manage a single version of the truth Legacy Accounting External Customers, Providers… External Control Unavailable 8 Unit of Business Functionality A SOA Service is cohesive unit of business functionality with direct value to a business user or business process with a machine readable contract and measurable SLA. Business Processes compose Services Services are realized by Components Components are hosted by Platforms Components are developed as parts of Applications SOA increases the granularity and modularity of business functions © Talend 2011 9 Enterprise Refactoring Low Risk Adoption Low Risk Adaptation Custom Application Packaged Application Low Risk Evolution Integration Server Legacy Application Custom Application Low Risk Retirement Architecture Agility Low Risk Adoption Low Risk Adaptation Custom Application Packaged Application Low Risk Evolution Custom Application Legacy Application Low Risk Retirement HL7 Service Aware Interoperability Framework (SAIF) Enterprise Compliance and Conformance Framework (ECCF) ECCF Conceptual Perspective Logical Perspective Implementable Perspective Enterprise Dimension “Why” - Policy Information Dimension Engineering Dimension Technical Dimension “How” - Behavior “Where” - Implementation “Where” - Deployments Inventory of • Domain Entities • Activities • Associations • Information Requirements • Information Models o Conceptual o Domain Inventory of • Functions-services Requirements • Accountability, Roles • Functional Requirements, Profiles, Behaviors, Interactions • Interfaces, Contracts Inventory of • SW Platforms, Layers • SW Environments • SW Components • SW Services • Technical Requirements • Enterprise Service Bus Key Performance Parameters Inventory of • HW Platforms • HW Environments • Network Devices • Communication Devices Technical Requirements Business Policies Governance Implementation Guides Design Constraints Organization Contracts Information Models Terminologies Value Sets Content Specifications Specifications • Use Cases, Interactions • Components, Interfaces Collaboration Participations Collaboration Types & Roles Function Types Interface Types Service Contracts Models, Capabilities, Features and Versions for • SW Environments • SW Capabilities • SW Libraries • SW Services • SW Transports Models, Capabilities, Features and Versions for • HW Platforms • HW Environments • Network Devices • Communication Devices Business Nodes Business Rules Business Procedures Business Workflows Technology Specific Standards Schemas for • Databases • Messages • Documents • Services • Transformations Automation Units Technical Interfaces Technical Operations Orchestration Scripts SW Specifications for • Applications • GUIs • Components SW Deployment Topologies HW Deployment Specifications HW Execution Context HW Application Bindings HW Deployment Topology HW Platform Bindings Inventory of • User Cases, Contracts • Capabilities Business Mission, Vision, Scope “What” - Content Computational Dimension Five (5) Categories: Capability | Mission | Business Process | Infrastructure/Enterprise Architecture | Interoperability 11/25/2011 CIO Conference: iEHR, VLER, ESB v3 12 Lean SOA © Talend 2011 13 Agile Manifesto Also Important Fundamental Processes and tools Comprehensive documentation Contract negotiation Following a plan Individuals and interactions Working software Customer collaboration Responding to change © Talend 2011 Works well wellwithin acrossaan enterprise Works single team http://agilemanifesto.org/ 14 Lean Software Development Principles Principles Myths 1. 2. 3. 4. 5. 6. 7. 1. 2. 3. 4. 5. 6. 7. © Talend 2011 Eliminate Waste Build Quality In Create Knowledge Defer Commitment Deliver Fast Respect People Optimize the Whole Early specification reduces waste The purpose of testing is to find defects Predictions ensure predictability Planning estimates are commitments Haste makes waste There is one best way Decompose and optimize Implementing Lean Software Development, Mary and Tom Poppendieck 15 Lean SOA Analysis Agile Planning Agile Methodologies Test Driven Development Continuous Deployment Operations SOA Development Continuous Delivery Design By Contract Delivery Continuous Integration © Talend 2011 Integration Behavior Driven Development 16 Increased Granularity – Decreased Cycle Time Are we building the right product? Validate business value Lower risk of User rejection Faster feedback from Users Incorporate feedback from Users sooner Are we building the product right? Improved visibility into functionality rather than SLOC Improved fidelity to actual environment Improved predictability based on empirical results © Talend 2011 17 Agile Methodologies • Rapid Incremental Iterations • Frequent Release • Integrated Team Analysis Analysis Analysis Design Design Design Code Code Code Test Test Test Iteration 1 Iteration 2 Iteration 3 © Talend 2011 Lean Product Management Agile Methodologies 18 Unit of Value – Run Tested Features The desired software is broken down into named features (requirements, stories) which are part of what it means to deliver the desired system. For each named feature, there are one or more automated acceptance tests which, when they work, will show that the feature in question is implemented. The RTF metric shows, at every moment in the project, how many features are passing all their acceptance tests. Ron Jeffries http://xprogramming.com/xpmag/jatRtsMetric © Talend 2011 19 Minimize SDLC Latency “How long would It take your organization to deploy a change that involved just one single line of code? Can you do this on a repeatable, reliable basis? What gets in the way of getting software out the door?” Mary and Tom Poppendieck, Implementing Software Development, p.59 Latency is different than throughput © Talend 2011 20 Continuous Delivery for SOA © Talend 2011 21 Multiple Stakeholders Organizations Enterprise Architecture Data Management Development Test Operations Business Users Roles Managers Analysts Architects Developers Testers Administrators Users Continuous Delivery is the basic process for lean collaboration © Talend 2011 22 Continuous Delivery 1. Configuration Management Continuous Delivery 2. Agile Testing 3. Build Pipeline Semantic Contracts SOA Integration Test Always Production Ready © Talend 2011 23 Agile Delivery Processes Continuous Integration Continuous Delivery 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 1. Maintain a Code Repository Automate the Build Make the Build Self-Testing Everyone Commits Every Day Every Commit is Built Build Fast Clone the Production Environment Make Builds Available Ensure Visibility Automate Deployment 2. 3. 4. 5. 6. 7. 8. Create a repeatable, reliable process for releasing software Automate almost everything Keep everything in version control If it hurts, do it more often, bring the pain forward Build quality in Done means released Everyone is responsible for delivery Continuous improvement Continuous Delivery assures Common Awareness and Understanding between teams © Talend 2011 24 SOA Continuous Delivery Integration © Talend 2011 QA + IA Release + Operations 25 Dev SCM Build Service Integration Integration Performance UAT Update Build Metadata Local Build CheckIn CI Build Trigger Pass Continuous Delivery Build Trigger Update Fail Local Build CheckIn CI Build Trigger Pass Continuous Delivery Build Trigger Pass Continuous Delivery Build Trigger Pass Continuous Delivery Build Trigger How can we adapt this for SOA? Pass Continuous Deployment Build Trigger Pass Continuous Integration Continuous Delivery Continuous Deployment Adoption Curve Continuous Integration Continuous Delivery Large Scale Continuous Deployment Adoption Continuous Deployment © Talend 2011 CI, Pipelines and Deployment, Chris Read 27 Essential Complexity Processes and Tools should promote Interactions Sufficient Documentation should emerge with Software Contracts should express common understanding Collaborative Plans should emerge as the basis for responding to Change Develop Monitor & Contol © Talend 2011 Runtime Deploy 28 How can we adapt Lean Principles to SOA? Address coordination between teams through Governance Service Lifecycle Management rather than Application Lifecycle Management Incorporate and automate early integration testing into SOA SDLC Establish stewardship for test and delivery artifacts Build visibility and trust through Continuous Delivery Apply BDD principles via test contracts for common understanding © Talend 2011 29 SOA Extensions for Continuous Delivery Perform full integration / deploy tests in each environment Include ‘upgrade’ tests for both provider and consumer Include multiple concurrent version tests Include backward compatibility test Include retirement tests Include tests for meta-data and management information for mixed environments (e.g. changed SLA thresholds, new log information) Share assets in a collaborative environment © Talend 2011 30 Semantic Contracts for Test Driven SOA © Talend 2011 31 Great software requires collaboration Collaborate on complicated problems Get them right early in development Enables customers, testers, and programmers to learn what their software should do Continuously compare customers' expectations to actual results © Talend 2011 32 Services Require Collaboration Across Boundaries © Talend 2011 33 Service Contracts are Central Manager Operations Manageable Lifecycle, release schedule User User Experience Analyst Business Value Security Security Policy BDD Tests Scalable, composable, extensible, re-use WSDL, API Developer Service Metadata Contracts are the basic artifact of coordination © Talend 2011 Tester Architect 34 Service Contract Challenges Increased Interdependence Unit of Functionality: Services versus Application Unit of Deployment : Components versus Application Organizational Unit: Product versus Project Service Layers Different Context of Use Balance Re-use, Loose Coupling, Consistent Semantics Independent Service Lifecycles Multiple Concurrent Versions © Talend 2011 35 Design By Contract Test Driven Development GUI Selenium Domain Fit, FitNesse, Rspec, JBehave Domain Driven Design Refactoring Unit Test Junit, TestNG © Talend 2011 36 SOA Building Blocks Orchestration Business Processes 1 Choreography All of these need to be tested Talend BPM Talend ESB Runtime 2 3 4 Features 5 Tasks Camel Bundles Service Enablement SOA Services CXF ActiveMQ Camel Packages Modules © Talend 2011 Fine Grained Modules Classes 37 Types of Testing Module Component / Service System Right Code TDD BDD / Semantic Tests Acceptance Testing Coding Right xUnit Integration Tests Stress and Load Test Stakeholders Developers Analysts Testers Developers Operations Testers Users Operations Methodology Continuous Integration Continuous Delivery Continuous Deployment Defining Tests is a collaborative way to establish semantics with verifiable contracts. © Talend 2011 38 Test Artifacts Definition and Intent SOA Extensions Mock Consumer Mock Provider Reference Consumer Reference Provider Test Documentation Test Code Automation Environment Configuration Data Binary Artifact Source Code © Talend 2011 Delivered as a design time service 39 Service Versioning Versions are governed by policies which described the rules for: Updating major version Updating minor version Updating patch version Backward compatibility Support and expiry time for past major versions Typically, version numbers are expressed as <Major>.<Minor>.<patch> OSGi uses semantic versioning Major, minor, micro, qualifier JCP has no dependency and module management standard OSGI provides the industry de-facto standard © Talend 2011 40 Managing Complexity Environments Data •Development •Continuous Integration •Service Integration •Performance •UAT •Production •Content •Source •Binaries •Configuration •Test Data •Performance Results •Transactions Manage meta-data across environments and test scenarios © Talend 2011 41 Refactoring Branch By Abstraction 1. Create an abstraction over the part of the system that you need to change. 2. Refactor the rest of the system to use the abstraction layer. 3. Create new classes in your new implementation, and have your abstraction layer delegate to the old or the new classes as required. 4. Remove the old implementation. 5. Rinse and repeat the previous two steps, shipping your system in the meantime if desired. 6. Once the old implementation has been completely replaced, you can remove the abstraction layer if you like http://continuousdelivery.com/2011/05/make-large-scale-changes-incrementally-with-branch-by-abstraction/ © Talend 2011 BBA applies Protected Variation (GoF) to create Service Legacy Adaptors 42 Legacy Adapter Consumer V1 Consumer V2 ESB ESB ESB Provider V1 Provider V2 The ESB must be able to manage multiple concurrent endpoint versions © Talend 2011 43 Test Fixtures and Unit Under Test Test Fixture Message Sender Input Unit Under Test Diagnostic Test Fixture Output Composite Destination ? Expected Test Results © Talend 2011 Test Assertion A B 44 Reference Consumer v1 Reference Provider v1 Prioritize Features Integration Test v1 Mock Consumer Mock Provider Specification By Example Provider fail Consumer Feature Story Refactor Refactor fail fail Regressio n Test Acceptance Criteria Regressio n Test fail fail fail fail Stable Regression Provider v1.1 Reference Consumer v1 Reference Provider v1 Consumer v1.1 Add Functionality Add Functionality fail fail Provider Test Candidate Provider Stable Regression fail Specification by Example Consumer Test Provider v1.2 Mock Consumer Mock Provider Consumer v1.2 Candidate Consumer Automated Developer Test Integration Test Reference Provider v2 Reference Consumer v2 Reference Provider v2 Reference Consumer v1 Reference Provider v1 Reference Consumer v2 Integration Test v2 Automated Semantic Test Integration Test Specification by Example eg.Division Direct application to the business domain numerator More precise than imperative requirements 10 Drive out ambiguity and edge cases 12.6 Help prioritize functionality 22 Can be automated 9 Communicate intent 11 Captures context 100 Can be incrementally elaborated for collaboration denominator quotient? 2 5.0 3 4.2 7 3.14285714 2~=3.14 3 3.0<5 2 4<5.5<6 4 [25.0] expect ed [33] Specification by Example can be applied at many levels of abstraction © Talend 2011 46 Capturing Test Data Capture Fixture AMQ Composite Destination Message Sender Message Receiver Actual Target Logging Queue Replay Fixture Message Playback Store File Endpoint © Talend 2011 Test Message Queue Test Rate Throttle 47 Behavior Driven Design for SOA Use terminology from the problem domain for agile acceptance tests (e.g. aviation) focused on behavior of business services Address Technical Stories regarding scalability and other nonfunctional aspects by using the enterprise integration pattern (EIP) domain specific language Include analysts, developers, testers, and operations from provider and consumer(s) stakeholder teams © Talend 2011 48 Test Driven SOA with Specification by Example Generic Test Principles Specification by Example Specific - explicitly defined and definite Measurable - observable and quantifiable Achievable – realistic scenario Relevant – relates to business value Time-bound – observable within timespan of test Drive out ambiguity Use concrete examples Capture and share intent Do not specify implementation Specifications are not workflow scripts Separate business rules from UI tests Focus on the problem domain Develop a domain language Cognitive Diversity – pair testing © Talend 2011 49 Fitnesse – Framework for Integrated Tests Lightweight, open-source framework Integrated Wiki Collaboratively define Tests Run tests, see results, compare, track progress Clear, early, frequently, quickly, automated © Talend 2011 50 IT Code is Waste Automation removes manual processes from production line Domain-centric focus on value Organizational seams are likely friction points Repeatable test environment is the basis for collaboration between development teams Integration Test Fixtures leverage ESB Scalable testing for early risk resolution Non-functional characteristics Improved modularity, flexibility, portability © Talend 2011 51 SOA Continuous Evolution at DISA © Talend 2011 52 Agile SOA Planning IaaS delivers IT as a utility service Enterprise Architecture PaaS Build Pipeline during Design PaaS ESB at runtime SaaS for customer Domain Cloud IaaS / PaaS SOA Governance SaaS / BPM / EIM SOA Governance manages SaaS to realize BPM and EIM © Talend 2011 53 Forge.mil A Combat Support Agency Forge.mil Community – Stakeholders and SMEs Developers Testers Users Program Managers Warfighters Process and Methods SME Collaborative Development/Test Environment Improve DoD’s ability to rapidly deliver dependable Software, Services and Systems Cloud Computing Services Tools and Resources Community Shared Knowledge and Best Practices Continuous Integration Agile Software Development Continuous Delivery Testing Services 54 Release Feedback A Combat Support Agency DEV / INT / TEST UAT IV&V Code Quality Compliance Security • • • PROD Inter-Op Integration Regression Compliance Regression Performance Security Faster feedback Test as much as possible early and often (“discover the pain and bring forward”) Perform longer running tests in parallel ( or after ‘release candidate’ ) Perform full integration / deploy tests in each environment (include ‘upgrade’ tests) Environments become more production-like Increasing confidence in build’s production readiness 55 DISA Capability Maturity Continuous Delivery DISA Forge.mil Continuous Integration Continuous Delivery Delivery Pipelines Automation DISA PaaS SOA Runtime Continuous Deployment Cloud Infrastructure / Environment Management Version Control Continuous Integration Automated Testing Configuration Management Package Management Agile (Scrum / XP) © Talend 2011 56 Continuous Evolution… A Combat Support Agency Continuous Integration Continuous Delivery Continuous Deployment Integration of code modules and sub-systems Rapid delivery of nascent capability to target systems for test and feedback Rapid deployment of newly developed capability to live production systems • Continuous commits to ‘trunk’ • Every commit results in a Build • Automate the Build & keep the Build fast • Continuous code quality checks • Unit Tests • Static Code Analysis • Integration and smoke tests Established • Automate deploys to ‘Pre-Production’ • Program controlled release and environments deployment of approved features • Deeper software quality and system security scans • Automated inter-op testing by use of simulated (virtual) • Continuous Information Assurance and Production interfaces and STIG compliance services • Monitoring of delivery pipeline and deployed systems • Release ‘dashboard’ Evolving Future 57 Summary Continuous Deployment is the natural evolution of Lean Software Continuous Integration Continuous Delivery Continuous Deployment SOA increases Business Modularity and Flexibility Lean Deployment Pipeline is needed in order to realize SOA benefits Lean SOA requires a shift from ALM to SLM (SEI SPL) SOA collaboration and coordination requires Continuous Delivery to address Semantic Contracts © Talend 2011 58 Resources http://martinfowler.com/articles/continuousIntegration.html http://continuousdelivery.com/ http://behaviour-driven.org/ http://continuousdelivery.com/wp-content/uploads/2011/04/deployment_production_line.pdf Bridging the Communications Gap, Specification by Example and Agile Acceptance Testing, Adzic The Rspec Book, Behavior Driven Development with Rspec, Cucumber, and Friends, David Chelimsky Test Driven, Practical TDD and Acceptance TDD for Java Developers, Lasse Koskela Mary and Tom Poppendieck Lean Software Development, An Agile Toolkit Implementing Lean Software Development © Talend 2011 59