The OWASP Application Security Metrics Project OWASP AppSec Seattle Oct 2006 Bob Austin Application Security Metrics Project Lead KoreLogic, Inc. [email protected] 804.379.4656 Copyright © 2006 - The OWASP Foundation Permission is granted.
Download ReportTranscript The OWASP Application Security Metrics Project OWASP AppSec Seattle Oct 2006 Bob Austin Application Security Metrics Project Lead KoreLogic, Inc. [email protected] 804.379.4656 Copyright © 2006 - The OWASP Foundation Permission is granted.
The OWASP Application Security Metrics Project OWASP AppSec Seattle Oct 2006 Bob Austin Application Security Metrics Project Lead KoreLogic, Inc. [email protected] 804.379.4656 Copyright © 2006 - The OWASP Foundation Permission is granted to copy, distribute and/or modify this document under the terms of the Creative Commons Attribution-ShareAlike 2.5 License. To view this license, visit http://creativecommons.org/licenses/by-sa/2.5/ The OWASP Foundation http://www.owasp.org/ Presentation Objectives Drivers for Security Metrics Review the Project Plan. Work Accomplished To Date, Next Steps Provide Application Security Metrics Resources Solicit Feedback and Participation OWASP AppSec Seattle 2006 2 The Best Metrics….Can Answer Hard Questions How secure am I? Am I better than this time last year? Am I spending the right amount of money? How do I compare to my industry peers (senior management’s favorite question)? Source: Dr. Dan Geer OWASP AppSec Seattle 2006 3 Forrester Survey: “What are your top three drivers for measuring information security?” Justification for security spending 63% Regulations 51% Loss of reputation 37% Better stewardship Better stewardship 26% Report progress to to business Report progress business Manage risk 23% Base: 40 CISOs and senior security managers 11% Source: “Measuring Information Security Through Metrics And Reporting”, Forrester Research, Inc., May 2006” OWASP AppSec Seattle 2006 4 Forrester Survey: What do CISOs want to measure? “As a CISO, if you have a choice of measuring and monitoring up to five areas in security, which ones would you measure?” Regulatory compliance 62% Incident handling and response 59% 55% Corporate governance Risk management process adherence 52% Security awareness 52% Vulnerability and patch management 52% Application security Base: 34 CISOs and senior security managers. 34% Source: “Measuring Information Security Through Metrics And Reporting”, Forrester Research, Inc., May 2006” OWASP AppSec Seattle 2006 5 Project Goal and Roadmap Project Goal: Address the current lack of effective application security metrics by identifying, sharing and evolving useful metrics and metric processes to benefit the OWASP community. Current Project Contributors: Jeff Williams (Aspect Security), Cliff Barlow (KoreLogic), Matt Burton (Mitre) Phase One Develop Project Approach Conduct research. Identify leading practices, standards Develop/Conduct Initial Survey Publish Survey Results Current Project Status Phase Two Solicit OWASP Feedback and Perform Gap Analysis Identify Short List of Needed Metrics Create Approach to Develop Metrics http://www.owasp.org/index.php/Category:OWASP_Metrics_Project OWASP AppSec Seattle 2006 6 Phase One – Application Security Metrics Baseline Survey Plan Information Capture Analysis Survey Results Interviews Research Key findings (from survey results) Identify key findings (common themes, barriers, concerns) Assess survey participant-provided metrics for applicability for OWASP community use Application security metrics Provide set of “best practices” associated with establishing an application security metrics program. Use results to design Phase Two of the Project Metrics Survey http://www.owasp.org/index.php/Metrics_Survey_Form OWASP AppSec Seattle 2006 7 Useful Resources from Research OWASP CLASP Project – “Monitor Security Metrics” Dr. Dan Geer’s “Measuring Security” Tutorial Other Initiatives: Securitymetrics.org, Metricon 1.0 Secure Software Development Life Cycle: “The Security Development Lifecycle”, Howard and Lipner, “Security in the Software Lifecycle”, DHS, Cybersecurity Div. Information Security Metrics Standard - ISO 27004 Dr. Larry Gordon, Cybersecurity Economics Research Projects Resources from NIST: Security Metrics Guide for Information Technology Systems, Guide for Developing Performance Metrics for Information Security NIST Software Assurance Metrics and Tool Evaluation (SAMATE) OWASP AppSec Seattle 2006 8 Organizing Metric Types Process Metrics Information about the processes themselves. Evidence of maturity. Vulnerability Metrics Metrics about application vulnerabilities themselves Management Metrics specifically designed for senior management Examples Secure coding standards in use Avg. time to correct critical vulnerabilities Examples By vulnerability type By occurrence within a software development life cycle phase Examples % of applications that are currently security “certified” and accepted by business partners Trending: critical unresolved, accepted risks OWASP AppSec Seattle 2006 9 Opportunities for Metrics - Secure Development Life Cycle (SDL) Were software assurance activities conducted at each lifecycle phase? Secure questions during interviews Concept Security push/audit Threat analysis Learn & Refine External review Designs Complete Team member training Security Review Test plans Complete Data mutation & Least Priv Tests Source: Microsoft Code Complete Deploy Post Deployment Review old defects = on-going Check-ins checked Secure coding guidelines Use tools 10 OWASP AppSec Seattle 2006 Examples of Application Security Metrics Process Metrics Is a SDL Process used? Are security gates enforced? Secure application development standards and testing criteria? Security status of a new application at delivery (e.g., % compliance with organizational security standards and application system requirements). Existence of developer support website (FAQ's, Code Fixes, lessons learned, etc.)? % of developers trained, using organizational security best practice technology, architecture and processes Management Metrics % of applications rated “business-critical” that have been tested. % of applications which business partners, clients, regulators require be “certified”. Average time to correct vulnerabilities (trending). % of flaws by lifecycle phase. % of applications using centralized security services. Business impact of critical security incidents. OWASP AppSec Seattle 2006 11 Examples of Application Security Metrics Vulnerability Metrics Number and criticality of vulnerabilities found. Most commonly found vulnerabilities. Reported defect rates based on security testing (per developer/team, per application) Root cause of “Vulnerability Recidivism”. % of code that is re-used from other products/projects* % of code that is third party (e.g., libraries)* Results of source code analysis**: Vulnerability severity by project, by organization Vulnerabilities by category by project, by organization Vulnerability +/- over time by project % of flaws by lifecycle phase (based on when testing occurs) Source: * WebMethods, ** Fortify Software OWASP AppSec Seattle 2006 12 The Path Forward Complete KoreLogic-sponsored surveys Encourage others to complete survey forms Create metrics taxonomy. Test drive it. Collaborate/share with other metrics initiatives “Will Work for Metrics”. Volunteers needed! Solicit survey participants. Collect survey data. Help analyze survey data Donate useful application security metrics Help plan Phase Two OWASP AppSec Seattle 2006 13 Our Security Metrics Challenge “A major difference between a "well developed" science such as physics and some of the less "well-developed" sciences such as psychology or sociology is the degree to which things are measured.” Source: Fred S. Roberts, ROBE79 “Give information risk management the quantitative rigor of financial information management.” Source: CRA/NSF, 10 Year Agenda for Information Security Research, cited by Dr. Dan Geer OWASP AppSec Seattle 2006 14 Supplemental Slides and Metrics Resources OWASP AppSec Seattle 2006 15 Resources – Security Metrics Security Metrics Standards: ISO 27004 - a new ISO standard on Information Security Management Measurements. Other metrics initiatives - Securitymetrics.org Metricon 1.0 presentations, http://www.securitymetrics.org/content/Wiki.jsp?page=Metricon1.0 Dan Geer’s measuringsecurity tutorial. Pdf, http://geer.tinho.net/usenix Developing metrics programs: Security Metrics Guide for Information Technology Systems, http://csrc.nist.gov/publications/nistpubs/800-55/sp800-55.pdf Guide for Developing Performance Metrics for Information Security, http://csrc.nist.gov/publications/drafts/draft-sp800-80-ipd.pdf Establishing an Enterprise Application Security Program, Tony Canike, OWASP 2005 Metrics-related Tools: NIST Software Assurance Metrics and Tool Evaluation (SAMATE), http://samate.nist.gov/index.php/Main_Page Metrics-related Models, Frameworks: http://www.sse-cmm.org/model/model.asp Current Articles on Metrics www.csoonline.com/metrics/index.htm Metric-related Financial and Econometric Resources: Economics and Security Resource Page, Ross Anderson), http://www.cl.cam.ac.uk/~rja14/econsec.html Dr. Larry Gordon, University of Maryland, Cybersecurity Economics Research Projects, http://www.rhsmith.umd.edu/faculty/lgordon/Cybersecurity%20Economics%20Research%20 Projects.html OWASP AppSec Seattle 2006 16 Resources – Software Assurance “A Clinic to Teach Good Programming Practices”, Matt Bishop, http://nob.cs.ucdavis.edu/bishop/talks/2006-cisse-2/clinic.html Team Software Process for Secure Systems (TSP-Secure), http://www.sei.cmu.edu/tsp/tsp-security.html OMG’s Software Assurance Workshop 2007, http://www.omg.org/news/meetings/SWA2007/call.htm DHS Cybersecurity Division Software Assurance Initiatives: Software Assurance Measurement Workshop, Oct, 2006 Software Assurance Program, http://www.psmsc.com/UG2006/Presentations/11_DHS_SwA_Overview_for_PSM.pdf Software Assurance Forum, https://buildsecurityin.us-cert.gov/daisy/bsi/events/521.html CERT Secure Coding Standards, https://www.securecoding.cert.org/confluence/display/seccode/CERT+Secure+Coding+Standards CRA Conference on "Grand Research Challenges in Information Security & Assurance“, http://www.cra.org/reports/trustworthy.computing.pdf OWASP AppSec Seattle 2006 17 Resources – General Software Measures & Metrics Measures and Metrics Web Sites, http://www.stsc.hill.af.mil/crosstalk/1999/06/measuresites.asp Software Process Metrics Organizations: http://www.totalmetrics.com/cms/servlet/main2?Subject=List&ID=3 http://www.swmetrics.org/ Software Metrics Symposium Capability Maturity Model Integration (CMMI) Tenth ANNUAL PSM USERS' GROUP CONFERENCE Performance and Decision Analysis, http://www.psmsc.com/UsersGroup2006.asp History of Software Measurement (Horst Zuse), http://irb.cs.tuberlin.de/~zuse/metrics/History_00.html NASA Software Engineering Laboratory, Experience Factory: http://sel.gsfc.nasa.gov/website/exp-factory.htm ISO/IEC 15939, Software Engineering - Software Measurement Process Software Metrics Glossary, http://www.totalmetrics.com/cms/servlet/main2?Subject=List&ID=12 2006 State of Software Measurement Practice Survey, http://www.sei.cmu.edu/sema/presentations/stateof-survey.pdf OWASP AppSec Seattle 2006 18 Really Bad Metrics Advice According to my data, roughly 122.45 percent of this journal's 347,583,712 readers need some sharpening up on how to effectively collect and use metrics. There is less than a 0.0345 percent chance that this column will help. Q: I'm a manager who believes in keeping metrics simple, which is why I've limited the number we collect to 62. But I also want to simplify their collection—do you know where I can find timecard readers designed for bathroom stalls? A: Try voice print-activated stalls with timed door locks. But first, are you really trying to collect 62 metrics? 62? [snicker snort chortle] You're obviously clueless about the "KISS" principle: Keep It Stupefyingly Strenuous. You can collect a lot more than 62 different metrics. The accepted rule of thumb for the number of metrics you can reasonably work with is this: "Seven, plus or minus the square of the number of door knobs in your home." Remember, if something can be measured, it must be measured, and all metrics are equally critical. Q: I feel vindicated. Now I can introduce additional metrics for every obscure area of our process improvement model. Naturally, I plan to drop the whole wad as an enforced edict and then make myself unavailable for a few weeks. A: Bravo! But be sure you don't overcomplicate things by defining every minute detail, such as data integrity standards or what you plan to do with the data. People learn nothing from constant handholding. Your job is to sit back and wait for those reliable numbers to start pouring in. Q: Great! What do you suggest I do with all that data? A: What should you do with the data? Do? That question implies that metrics are a means to some end. Don't waste resources—time spent analyzing metrics is time that could have been spent collecting even more metrics. Q: My boss keeps asking for data on stuff I don't think can be quantified—and it's often common sense stuff he could just ask us! Aren't metrics just a big sham? A: Shhh! You're right, metrics are actually an extensive conspiracy—but an extremely helpful one. When people want to make decisions based on "facts" rather than "opinions," you need metrics to push your personal agenda under the guise of unassailable objectivity. Perception is everything: Politicized emotional drivel: "Let's try my approach. Her plan isn't working." Objective insight: "A consumptive analysis of my plan projects a 84.67 percent increased density of pro-active rationals within six months. However, her key preambulatory vindicators are creating a 24.38 percent downward sloping polymorphic trend. Plus, she wears really cheesy business suits." Source: http://www.stsc.hill.af.mil/crosstalk/1998/08/backtalk.asp OWASP AppSec Seattle 2006 19