Transcript Server Virtualization Assessment – Tools and Techniques
Server Virtualization Assessment – Tools and Techniques
Chicago ISACA Chapter 8/11/2011
Michael Hoesing CISA,
CISSP, CCP, ACDA, CIA, CFSA, CMA, CPA
[email protected]
Anything discussed herein should be tested thoroughly in a lab environment before use in production. Opinions are those of the author and not conference sponsors, employers, clients, past, present or future. Don’t sue me; I have no money.
Slide 1 of 41
www.isaca.org
Server Virtualization Assessment Objectives
• Virtualization Definitions, Background, Scope • Risks and Controls • Assessment Approaches and Tools:
• Assessment Examples
– VM (Guest) Sprawl – ESX Console Operating System (COS)Configuration
• Notes for vSphere 5
Slide 2 of 41
Background, Scope
Slide 3 of 41
BACKGROUND
• • • • •
2004 ++ Virtualization Spreads 2007 Gartner declares virtualization security important 2007 to Today risk and security/control techniques and products related to virtualization evolve Now , and before now, we should evaluate how effective are those security techniques and controls (assessment) Business can’t live without speed to deployment
Slide 4 of 41
SCOPE
• • • • Virtualization Scope – ESX servers hosting guests Not Included variants) – (only so much can be done in 1.5 hours) VDI, Hyper-V, Xen (Citrix & other Some risk topics reach beyond ESX (policy, process, procedure) if you are going to secure an ESX environment you must think beyond the COS Some topics should be in scope but their complexity is best covered separately (storage, backups) Slide 5 of 41
Risks & Controls
Slide 6 of 41
RISKS & CONTROLS – a list of 10
1. VM/Guest Sprawl 2. Host Mis Configuration
3. Network Segmentation 4. Remote Access Slide 7 of 41 • Policies, Procedures, Inventory Practices, Reporting, Assessment • Standards, Monitoring, Assessment • Deploy Segregated Management, Production and IP Storage Networks • SSH , SSL, access & account controls
RISKS & CONTROLS – a list of 10 (cont)
5. User Account Access & Roles 6. Single Point of Failure 7. Integration 8. Staff Skills 9. Architecture (Blue Pill) 10. Software Licensing 11. I lied # 11 Appliances 12. #12 Guest Escape VMSA-2009-0006 • Policies, Procedures, Least Privilege • Backups, Continuity Planning • Strategic Architecture, Capacity Planning • Training • Physical Security • Policy, Monitoring • QA, Certification Processes, Vendor Mgmt • Patch Process Slide 8 of 41
Assessment Approaches and Tools
Slide 9 of 41
ASSESSMENT APPROACH
• • The Approach - 1.) a standard 2.) gather metrics 3.)compare metrics to the standard and cite variances Standard – • • a.) yours, if you have created a document, congratulations b.) VMware Hardening Guide(s) http://www.vmware.com/files/pdf/vi35_security_hardening_wp.pd
f http://communities.vmware.com/docs/DOC-15413 • c.) CIS – ESX 3.0, 3.5, 4.x Xen http://cisecurity.org/benchmarks.html
has an XCCDF assessment tool (CIS-CAT) for members for 3.5 and 4.x) (also • d.) DISA STIG http://iase.disa.mil/stigs/stig/esx_server_stig_v1r1_final.pdf
Slide 10 of 41
ASSESSMENT APPROACH (cont)
• More Standards – • e.) NIST 800-125 Jan 2011 http://csrc.nist.gov/publications/nistpubs/800 125/SP800-125-final.pdf
• f.) PCI/DSS June 2011 https://www.pcisecuritystandards.org/documents/Virt ualization_InfoSupp_v2.pdf
• g.) NSA http://www.nsa.gov/ia/_files/support/I733 009R-2008.pdf
• h.) Vendors (HyTrust), consultants, books (Ed Haletky, Scott Lowe, Siebert …) Slide 11 of 41
ASSESSMENT APPROACH – Audit Programs
• • • ISACA – Whitepaper – issued Oct 2010 risks, audit approaches http://www.isaca.org/Knowledge Center/Research/ResearchDeliverables/Pages/Virtualizat ion-Benefits-and-Challenges.aspx
Audit program issued Jan 2011, GRC level http://www.isaca.org/Knowledge Center/Research/ResearchDeliverables/Pages/VMware Server-Virtualization-Audit-Assurance-Program.aspx
SANS talk through http://www.sans.org/reading_room/analysts_program/VM ware_ITAudit_Sep09.pdf
Mine (come to the hands-on class), mix of process/procedure and detailed metrics Slide 12 of 41
GATHERING METRICS – SOME THOUGHTS
• In 50 mins all I can do is name-drop, you do the research in your environment/strategy/risk appetite • Not a Bake-off, not a Best-of, I can only relate what worked in my lab (see bullet one above), any product not mentioned just means I have not installed it yet • Good News, lots of products to chose from , list grows almost daily (bad news, that expands due diligence time) • Some tools Work in a Virtual Appliance, some tools have both a physical and virtual appliance • Key – does ingress and egress to/from the Guest allow the product to do its Job (patching, AV, config assessment) Slide 13 of 41
GATHERING METRICS – SOME THOUGHTS (cont)
• Free Tools – great price, don’t scale well • Some tools inventory the Virtual Center database, some tools enumerate raw data • No one tool does everything, run multiple tools for corroboration and completeness • Tools that use a RHEL baseline, take care in reviewing, but maybe 80-90% correct • In a lab, build an ESX server (and vCenter) with the vendor defaults, and build a second ESX server with your organization ’s standard build, for education purposes and to calibrate tools Slide 14 of 41
METRIC GATHERING TOOLS
• • Interviewing and Document Review for policies, standards, procedures, training
!Free!
Tools – • console CLI, and vSphere remote CLI •
CIS-CAT 2.2.7 June/30/2011 (for members) ESX 3.5 and 4.x benchmark XCCDF test scripts
• VIToolkit & Powershell, (now called vSphere PowerCLI 4.1 U1) • esxcfg-xxx commands various (i.e. esxcfg-firewall and search –q) • esxcfg-info – dump of everything, load into ACL Slide 15 of 41
METRIC GATHERING TOOLS (cont)
• More Free Tools: • vmware-vim-cmd hostsvc/ = grep /net/info or grep /storage/info (careful, many of these commands change settings, stick with the ones with the word ‘info’) • Configuresoft (Ionix) ComplianceChecker, Tripwire configcheck, (ESX 3) • From VMware - VI API, VIX API ( allows files xfer from guest ) , Perl API, CIM API (risks of rolling your own = script storage security, stored passwords, change management, version management) Slide 16 of 41
METRIC GATHERING TOOLS (cont)
• More Free Tools: • Bastille – remember to run in the –assess mode, not the harden mode (3.0.9-1.0) • DISA – SRR (security readiness review evaluation script) watch these, they may harden if not run correctly • LSAT – works on 3.5 and before, but the MD5 process will try to analyze the very large vmdk disk files, this is time consuming and could crash running guests (note : does not work in vSphere, C compiler is removed) Slide 17 of 41
METRIC GATHERING TOOLS (cont)
• Existing Management Tools - (vCenter, Update Mgr, Lifecycle Mgr, Veeam & others) • Security Tools (Reflex, Catbird, BlueLane & others) • Commercial Tools – (Configuresoft [Ionix], Ecora, Tripwire, & others) • Hy-Trust - won a bunch at VMworld 2009, access control and enhanced logging • Vkernel Optimization Pack – inventories, finds underutilization • VMsafe vendor tools, Host Profiles(if using Enterprise Plus) Slide 18 of 41
Assessment Examples VM (Guest) Sprawl
Slide 19 of 41
SPRAWL - CLI
• • • • • • • • • • • • Free Tools – Command Line Interface (CLI) ls –lR /vmfs/volumes/* | grep vmx • Or the ‘find’ command (does not follow sym links) -rwxrwxrwx 1 root root 4831838208 Jul 7 2007 BLVS-flat.vmdk
-rwxrwxrwx 1 root root 331 Jul 7 2007 BLVS.vmdk
-rwxrwxrwx 1 root root 8589934592 Jul 7 2007 BLVSMgr-flat.vmdk
-rwxrwxrwx 1 root root -rw------- 1 root root 336 Jul 7 2007 BLVSMgr.vmdk
872415232 Sep 23 10:10 Reflex-VSA-Template-flat.vmdk
-rw------- 1 root root -rw------- 1 root root -rw------- 1 root root -rw------- 1 root root -rw------- 1 root root -rw------- 1 root root 480 Sep 23 10:10 Reflex-VSA-Template.vmdk
4294967296 Oct 8 11:37 Reflex-vsc-flat.vmdk
499 Oct 8 00:50 Reflex-vsc.vmdk
6442450944 Sep 29 01:59 RHEL-4-4-ES-flat.vmdk
339 Sep 29 01:57 RHEL-4-4-ES.vmdk
16791552 Mar 17 2008 SLES10-SP1-000001-delta.vmdk
Slide 20 of 41
SPRAWL – CIS-CAT
Free Tools CIS-CAT (if a member) will list VM ’s with non-compliant vmx config files (not a complete inventory but a good start on what needs correction) Slide 21 of 41
SPRAWL – PowerCLI 4.0
•
VI Tools for Windows & Powershell now named vSphere PowerCLI 4.1 (partial script)
• $VC = Connect-VIServer 192.168.1.21 -User XXXXXX -Password XXXXXX • $VMs = Get-VM | format-table -property name • $Datastores = Get-Datastore | Format-Table -property Name • $VMXlist = " " • $i = 1; while ($i -le $Datastores.length-4) $Datastore = Read-Host "Enter Data Store Name, like storage1* from the list above " • get-childitem -recurse -include >> c:\vmxlist *.vmx
| format-table -property name • $i +=1 • Then compare the two files (VM list and vmx list) with diff, ACL, or manually Slide 22 of 41
SPRAWL – vCenter
• Existing Management Tools - Virtual Center Slide 23 of 41
SPRAWL – Reflex
• Third Party Security Tools – Reflex Slide 24 of 41
SPRAWL – Configuresoft (IONIX)
• Commercial Configuration Assessment Tools – Configuresoft (Ionix) Slide 25 of 41
SPRAWL – Ecora
• Commercial Assessment Tools – Ecora Slide 26 of 41
SPRAWL – Honorable Mention
• • • • • • • • • Anything that Monitors Usually has an Inventory Component Akorri Balance Point CA ASM (Unicenter) BMC Performance Manager eG Innovations Enterprise Suite Embotics V-Commander IBM Tivoli Monitoring for Virtual Servers HP Operations Orchestration ManageIQ EVM Suite Quest vFoglight Tideway Foundation SPI for VMware Netuitive SI for VMware Symantec Altiris Veeam Monitor vmInformer Slide 27 of 41
Assessment Examples Host Configuration
Slide 28 of 41
HOST CONFIGURATION – CIS-CAT Categories
• CIS-CAT 9 Categories (3.5) Slide 29 of 41
HOST CONFIGURATION – CIS-CAT Benchmark Items
• CIS-CAT 29 Benchmark Items (3.5) Slide 30 of 41
HOST CONFIGURATION – CIS-CAT Detail Assessment Test and Results
• 1.2.3 Recommended Boot services (3.5) Slide 31 of 41
HOST CONFIGURATION – CIS-CAT Categories
• CIS-CAT 12 Categories (4.1) Slide 32 of 41
HOST CONFIGURATION – CIS-CAT Benchmark Items
• CIS-CAT Benchmark 65 Items (4.x) (partial) Slide 33 of 41
HOST CONFIGURATION – CIS-CAT Detail Assessment Test and Results
• 9.1 Recommended Boot services (4.x) Slide 34 of 41
HOST CONFIGURATION – Tripwire
• Commercial Assessment Tools – Tripwire Slide 35 of 41
vSphere 5
Slide 36 of 41
vSphere 5
• Released July 2011 • Memory based pricing is new, and not popular • ESX COS is gone, ESXi the only choice • ESXi has hypervisor and console all on the same partition, faster (vendor says) • ESXi 5 has a firewall (iptables) ESXi 1-4 did not • No (if configured as suggested) console access, all access is remote • Use vMA, remote CLI, and PowerCLI for audit metric gathering or vCenter Slide 37 of 41
vSphere 5 (cont)
• TPM (Trusted Processing Module) recognition available (Intel ’s TXT or AMD’s SEM , soon) • Hope they Fixed These in 5 (ESXi 4.1 issues) Logs removed upon reboot during installation root password not set Tech Support Mode (from console) Remote Tech Support Mode (SSH), accesses
Single User Mode
(root without any password if not set at default, even with password root SSH is enabled) Reset System Configuration
– resets an empty root password
(watch iLO and iDRAC) Slide 38 of 41
Conclusion
Slide 39 of 41
SUMMARY
• Virtualized Infrastructure is Important to the Organization and worthy of secure configuration and periodic assessment of that state • Standards are available for a starting point to create/edit your organization's policy • Tools are available, in all price ranges, to gather metrics from an ESX environment • Get the tools, gather the metrics, compare to the policy/standard, cite the differences, improve your security posture Slide 40 of 41
– If the question comes to you later [email protected]
– ?
– ?
– ?
– ?
– ?
– ?
Slide 41 of 41