Transcript Document

DCM 7.1
Hidden Security Vulnerabilities
Are your Servers Safe?
Richard Dominach
Senior Product Manager
1
True or False?
I don’t have to worry about anyone hacking my servers if they are
powered down
My security software can detect any and all viruses on my servers
My servers are behind a firewall; I don’t have to worry about them
I’ve purchased my servers from leading manufacturers so I know
they are secure
All the passwords on my servers are encrypted. There is no way
to login without a password
No one would be dumb enough to connect their servers directly to
the Internet
I can completely wipe clean any server before I sell or dispose of it
2
Is this Possible?
Without a warning, your servers stop running and cannot be restarted
Many of your servers stop their normal operation and start
running an unknown application
A large percentage of your servers unexpectedly power down
and can’t be rebooted
A rogue application has been silently loaded on your servers
and is waiting for a particular date to spring into action
Your administrators are unable to log in to any of your servers
A previously inactive server starts running and starts
communicating over the internet
3
Agenda
1. Intro to remote server management
2. IPMI & BMC overview
3. Recent research on IPMI/BMC security issues
4. Classes of vulnerabilities
5. How real is all of this?
6. What can/should/must you do about it?
7. For more information
4
Intro to Remote Server Management
• IT & Lab Managers use remote access systems to
manage thousands of servers and other devices
• Many different tools are available:
•
Hardware vs Software based
•
In-band vs out-of-band
• Tools must be:
• Powerful
• Secure
• Dependable
• Manageable
5
Types of Remote Management Tools
Software based tools like RDP or VNC
External hardware solutions - KVM-over-IP switches
Internal hardware solutions
Embedded Service Processors (ESP)
Baseboard Management Controllers (BMC)
Remote Management Protocols
Intelligent Platform Management Interface (IPMI)
You need to carefully consider the security of these tools
6
IPMI & BMC Overview
BMC’s are independent hardware processors and firmware
embedded inside virtually all servers
Remotely controllable, they have direct access to the server’s
motherboard
They monitor, boot, power and can even reinstall the server
Many provide KVM-over-IP access and the connection of
remote media
External access to the BMC provides “virtually unlimited” remote
control of the server
IPMI is a protocol used by the BMC for remote server
information & management
7
BMC Example #1
http://openipmi.sourceforge.net/IPMI.pdf
8
BMC Example #2
Source: http://electronicdesign.com/dsps/trying-out-remote-server-management-options
9
BMC Example #2
http://fish2.com/ipmi/itrain.pdf
10
Recent Research on IPMI/BMC Security Issues
Recent research has shown significant and widespread security
issues with the IPMI protocol used by BMC’s
•
Dan Farmer - security researcher working for DARPA
•
H.D. Moore - Metasploit founder and security expert
•
University of Michigan Research Team - Bonkoski, Bielawski, Halderman
BMC/IPMI used in the remote management of most server manufacturers
These vulnerabilities, if exploited, could lead to very serious consequences
These security issues are not well known by most server administrators and
security professionals
Security tools and policies not targeted at these vulnerabilities
11
IPMI Protocol Security Issues
Supports “Cipher0” - bypasses the entire authentication process
Allows IPMI commands from any source
Many BMC manufacturers enable this method by default.
Will send a hash of the requested user’s password
Can determine password unless password is very strong
May support anonymous logins
Some vendors ship with anonymous login configured by default
Will freely respond with the types of authentication supported
The BMCs freely tell whether an anonymous login has been configured,
May support Universal Plug and Play protocol which can provide root access
Some enable the Universal Plug and Play (UPnP) protocol by default and provide no way for the user to
disable this functionality
IPMI passwords stored unencrypted on the service processor
IPMI passwords must be stored unencrypted on the BMC
Orgs place servers into large managed IPMI groups with same password
Source: Dan Farmer, http://fish2.com/ipmi/itrain.pdf
12
Department of Homeland Security (US-CERT) Risks
Passwords for IPMI authentication are saved in clear text.
Knowledge of one IPMI password gives you the password for all computers in the IPMI
managed group.
Root access on an IPMI system grants complete control over hardware, software, firmware
on the system.
BMCs often run excess and older network services that may be vulnerable.
IPMI access may also grant remote console access to the system, resulting in access to the
BIOS.
There are few, if any, monitoring tools available to detect if the BMC is compromised.
Certain types of traffic to and from the BMC are not encrypted.
Unclear documentation on how to sanitize IPMI passwords without destruction of the
motherboard.
Source: https://www.us-cert.gov/ncas/alerts/TA13-207A
13
Classes of Vulnerabilities
1. IPMI Specification Vulnerabilities
2. Vendor Implementation Risks
3. End User Deployment Vulnerabilities
4. Architectural Risks
14
IPMI Specification Vulnerabilities
Uses Insecure IPMI Protocol
Cipher0 Authentication - access without a password
Will Send Hash of User Password
BMC Responds with Authentication Types
Unencrypted Passwords Stored
Static Encryption Keys - may exist
15
Vendor Implementation Risks
Default Passwords
Anonymous Logins
Universal Plug and Play protocol
Firmware updates - infrequent or not available
Backdoor Accounts - may exist
16
End User Deployment Vulnerabilities
Users unaware IPMI is enabled
Group passwords commonly used
Weak Passwords allowed
Passwords remain un-changed for long time
Connect servers to Internet
Connect to standard network
Don’t connect to management network
Lack of cross-vendor security tools
17
Architectural Risks
BMC Embedded in Server
BMC has Unlimited Server
Control
No visibility into BMC
activities
Survives re-install of the OS
BMC Can Power On/Off
Server
Provides Access to Powered
Down Servers
Multiple Attack Targets - one
per server
Malware & viruses difficult to
detect
BMC Can Infect Host System
BMC De-Provisioning
difficult
Host System can Infect BMC
18
How real is all of this?
19
Researchers Views
Moore: “In addition to vulnerabilities in the IPMI protocol itself, most BMCs seem to suffer
from issues common across all embedded devices, namely default passwords, outdated
open source software, and, in some cases, backdoor accounts and static encryption
keys.”
Moore: “The world of BMCs is a mess that is not likely to get better anytime soon, and we
need to be crystal clear about the risk these devices pose to our networks.”
Farmer: “Imagine trying to secure a computer with a small but powerful parasitic server on
its motherboard; a bloodsucker that can’t be turned off and has no documentation; you
can’t login, patch, or fix problems on it; server-based defensive, audit, or anti-malware
software can’t be used. Its design is secret and implementation old.”
Farmer: “It’s also the perfect spy platform: nearly invisible to its host, it can fully control the
computer’s hardware and software, and it was designed for remote control and
monitoring.”
Farmer: In sum, you may not know it, but your goose may already be cooked and you’re
simply asking for the orange sauce. There is no easy fix, but I’d suggest a dialogue
between customers, vendors, and the security community for starters.... In any case, good
luck. We may all need it.
20
US CERT Alert TA13-207A
https://www.us-cert.gov/ncas/alerts/TA13-207A
21
Headlines
“Widespread Vulnerabilities in BMCs and the IPMI Protocol”
“New Gaping Security Holes Found Exposing Servers”
“IPMI: The most dangerous protocol you've never heard of”
“IPMI Protocol, BMC Vulnerabilities Expose Thousands of
Servers to Attack”
“Hackers can wipe or steal data from security holes in 300,000
servers”
“IPMI: Hacking servers that are turned off”
22
What can you do about it?
1. Do nothing and hope and/or pray...
2. Continue to use, understand issues and secure
3. Use alternative strategies
23
Department of Homeland Security (US-CERT) Solutions
►Restrict IPMI to Internal Networks
Restrict IPMI traffic to trusted internal networks. Traffic from IPMI (usually UDP port 623) should be restricted to a management
VLAN segment with strong network controls. Scan for IPMI usage outside of the trusted network and monitor the trusted network
for abnormal activity.
►Utilize Strong Passwords
Devices running IPMI should have strong, unique passwords set for the IPMI service. See US-CERT Security Tip ST04-002 and
Password Security, Protection, and Management for more information on password security.
►Encrypt Traffic
Enable encryption on IPMI interfaces, if possible. Check your manufacturer manual for details on how to set up encryption.
►Require Authentication
"cipher 0" is an option enabled by default on many IPMI enabled devices that allows authentication to be bypassed. Disable
"cipher 0" to prevent attackers from bypassing authentication and sending arbitrary IPMI commands. Anonymous logins should
also be disabled.
►Sanitize Flash Memory at End of Life
Follow manufacturer recommendations for sanitizing passwords. If none exists, destroy the flash chip, motherboard, or other
areas the IPMI password may be stored.
►Identify Affected Products
• Most server products
https://www.us-cert.gov/ncas/alerts/TA13-207A
24
Select Recommendations from Farmer
•
If possible keep all IPMI network interfaces on their own segregated network. No other
computers should be using this managed network.
•
Severely restrict any network access to any BMC as well as the BMC’s capability for
communication. Monitor the traffic on the management network.
•
Restrict and alarm outbound network traffic and access for the BMCs
•
Add a layer of by placing a very secure computer to serve as a bastion host between the
management network and the unwashed masses of computers at large
•
Two-factor authentication with at least one factors unique to the management network
•
Build a set of best practices and policies around BMC and IPMI security.
•
Treat the BMCs as real servers (they are!): ensure that they’re monitored, scanned for
vulnerabilities, have their logs go to logging servers, etc.
•
Keep up-to-date with the most recent firmware for your BMCs as you can
•
Ensure that the BMC storage is wiped or reset, including passwords, when de-provisioning
Dan Farmer: IPMI++ Security Best Practices, http://fish2.com/ipmi/bp.pdf
25
Additional Recommendations
Setup a dedicated management network, and limit IPMI to the network card connected to
the management network.
Review the BIOS configuration option for IPMI. If you can't have a physical management
network, at least try to use a VLAN
Keep IPMI firmware up to date. Do not use default passwords
Eliminate IPMI access over insecure protocols. Use HTTPS with certificates, or SSH
Integrate IPMI authentication with existing authentication systems like RADIUS and AD.
Review hardening options your IPMI implementation provides.
Limit access from IP addresses, or turn off various features you do not need
Inventory servers with IPMI capability
Source: https://isc.sans.edu/diary/IPMI%3A+Hacking+servers+that+are+turned+%22off%22/13399
26
Alternative Strategies
1. Software Based Remote Access
1.
2.
3.
Remote Desktop, Terminal Services, RDP
VNC
SSH
2. Hardware Based Remote Access
1.
2.
3.
KVM-over-IP Switches
Serial Console Servers
Intelligent Rack Based PDU’s
27
Example: KVM-over-IP Access
LAN Users
WAN
Management Network
Server Rack(s)
Computer
Interface
Modules
(CIM)
KVM-over-IP
Switch
GigE
WAN Users
Cat5
VPN
At-the-Rack
Access
Home, on-the-road,
etc.
Intelligent Rack PDUs
28
28
For More Information
Original research by Farmer & Moore and
Other References (below)
Press Articles (below)
IPMI Articles on the Raritan Blog
Server Manufacturer Information
29
For More Information
Dan Farmer: IPMI | Trouble | Security, A paper on IPMI & BMC Security
http://www.fish2.com/ipmi/
HD Moore - a penetration testers guide to IPMI and BMC’s
https://community.rapid7.com/community/metasploit/blog/2013/06/23/a-penetration-testers-guide-to-ipmi
Widespread Vulnerabilities in BMCs and the IPMI Protocol - Frequently Asked Questions
https://community.rapid7.com/docs/DOC-2344
IPMI: Hacking servers that are turned "off“
https://isc.sans.edu/forums/diary/IPMI+Hacking+servers+that+are+turned+off+/13399
University of Michigan: Illuminating the Security Issues Surrounding Lights-Out Server Management
https://jhalderm.com/pub/papers/ipmi-woot13.pdf
Dark Reading - New Gaping Security Holes Found Exposing Servers
http://www.darkreading.com/management/new-gaping-security-holes-found-exposing/240157724
“Illuminating the Security Issues Surrounding Lights-Out Server Management”
https://www.usenix.org/system/files/conference/woot13/woot13-bonkoski_0.pdf
IPMI: The most dangerous protocol you've never heard of
http://www.itworld.com/security/369507/ipmi-new-four-letter-word-security
IPMI Protocol, BMC Vulnerabilities Expose Thousands of Servers to Attack
http://threatpost.com/ipmi-protocol-bmc-vulnerabilities-expose-thousands-of-servers-to-attack
Hackers can wipe or steal data from security holes in 300,000 servers
http://www.networkworld.com/community/blog/hackers-can-wipe-or-steal-data-gaping-security-holes-300000-servers
US-CERT: Risks of Using the Intelligent Platform Management Interface (IPMI)
http://www.us-cert.gov/ncas/alerts/TA13-207A
Supermicro IPMI based on ATEN firmware contain multiple vulnerabilities
http://www.kb.cert.org/vuls/id/648646
30
Conclusion
Remote management is absolutely necessary for server
administrators
But, existing BMC/IPMI tools have serious security issues
BMC/IPMI provides “virtually unlimited” remote control of the
server
If compromised, could lead to very serious consequences
You should understand the risks, solutions & alternatives
Take the appropriate actions to safeguard your infrastructure
31
Thank You ... and For More Information
Thank you for your time!
Contact me:
[email protected]
Web Site: www.raritan.com
Facebook: www.facebook.com/RaritanInc
Blog: http://blog.raritan.com/
Search for ‘Raritan You Tube” for videos
Stop by Raritan Booth # 1112
32