Transcript Document

DCM 7.1
Hidden Security Vulnerabilities
Are your Servers Safe?
Richard Dominach
Senior Product Manager
True or False?
I don’t have to worry about anyone hacking my servers if they are
powered down
My security software can detect any and all viruses on my servers
My servers are behind a firewall; I don’t have to worry about them
I’ve purchased my servers from leading manufacturers so I know
they are secure
All the passwords on my servers are encrypted. There is no way
to login without a password
No one would be dumb enough to connect their servers directly to
the Internet
I can completely wipe clean any server before I sell or dispose of it
Is this Possible?
Without a warning, your servers stop running and cannot be restarted
Many of your servers stop their normal operation and start
running an unknown application
A large percentage of your servers unexpectedly power down
and can’t be rebooted
A rogue application has been silently loaded on your servers
and is waiting for a particular date to spring into action
Your administrators are unable to log in to any of your servers
A previously inactive server starts running and starts
communicating over the internet
1. Intro to remote server management
2. IPMI & BMC overview
3. Recent research on IPMI/BMC security issues
4. Classes of vulnerabilities
5. How real is all of this?
6. What can/should/must you do about it?
7. For more information
Intro to Remote Server Management
• IT & Lab Managers use remote access systems to
manage thousands of servers and other devices
• Many different tools are available:
Hardware vs Software based
In-band vs out-of-band
• Tools must be:
• Powerful
• Secure
• Dependable
• Manageable
Types of Remote Management Tools
Software based tools like RDP or VNC
External hardware solutions - KVM-over-IP switches
Internal hardware solutions
Embedded Service Processors (ESP)
Baseboard Management Controllers (BMC)
Remote Management Protocols
Intelligent Platform Management Interface (IPMI)
You need to carefully consider the security of these tools
IPMI & BMC Overview
BMC’s are independent hardware processors and firmware
embedded inside virtually all servers
Remotely controllable, they have direct access to the server’s
They monitor, boot, power and can even reinstall the server
Many provide KVM-over-IP access and the connection of
remote media
External access to the BMC provides “virtually unlimited” remote
control of the server
IPMI is a protocol used by the BMC for remote server
information & management
BMC Example #1
BMC Example #2
BMC Example #2
Recent Research on IPMI/BMC Security Issues
Recent research has shown significant and widespread security
issues with the IPMI protocol used by BMC’s
Dan Farmer - security researcher working for DARPA
H.D. Moore - Metasploit founder and security expert
University of Michigan Research Team - Bonkoski, Bielawski, Halderman
BMC/IPMI used in the remote management of most server manufacturers
These vulnerabilities, if exploited, could lead to very serious consequences
These security issues are not well known by most server administrators and
security professionals
Security tools and policies not targeted at these vulnerabilities
IPMI Protocol Security Issues
Supports “Cipher0” - bypasses the entire authentication process
Allows IPMI commands from any source
Many BMC manufacturers enable this method by default.
Will send a hash of the requested user’s password
Can determine password unless password is very strong
May support anonymous logins
Some vendors ship with anonymous login configured by default
Will freely respond with the types of authentication supported
The BMCs freely tell whether an anonymous login has been configured,
May support Universal Plug and Play protocol which can provide root access
Some enable the Universal Plug and Play (UPnP) protocol by default and provide no way for the user to
disable this functionality
IPMI passwords stored unencrypted on the service processor
IPMI passwords must be stored unencrypted on the BMC
Orgs place servers into large managed IPMI groups with same password
Source: Dan Farmer,
Department of Homeland Security (US-CERT) Risks
Passwords for IPMI authentication are saved in clear text.
Knowledge of one IPMI password gives you the password for all computers in the IPMI
managed group.
Root access on an IPMI system grants complete control over hardware, software, firmware
on the system.
BMCs often run excess and older network services that may be vulnerable.
IPMI access may also grant remote console access to the system, resulting in access to the
There are few, if any, monitoring tools available to detect if the BMC is compromised.
Certain types of traffic to and from the BMC are not encrypted.
Unclear documentation on how to sanitize IPMI passwords without destruction of the
Classes of Vulnerabilities
1. IPMI Specification Vulnerabilities
2. Vendor Implementation Risks
3. End User Deployment Vulnerabilities
4. Architectural Risks
IPMI Specification Vulnerabilities
Uses Insecure IPMI Protocol
Cipher0 Authentication - access without a password
Will Send Hash of User Password
BMC Responds with Authentication Types
Unencrypted Passwords Stored
Static Encryption Keys - may exist
Vendor Implementation Risks
Default Passwords
Anonymous Logins
Universal Plug and Play protocol
Firmware updates - infrequent or not available
Backdoor Accounts - may exist
End User Deployment Vulnerabilities
Users unaware IPMI is enabled
Group passwords commonly used
Weak Passwords allowed
Passwords remain un-changed for long time
Connect servers to Internet
Connect to standard network
Don’t connect to management network
Lack of cross-vendor security tools
Architectural Risks
BMC Embedded in Server
BMC has Unlimited Server
No visibility into BMC
Survives re-install of the OS
BMC Can Power On/Off
Provides Access to Powered
Down Servers
Multiple Attack Targets - one
per server
Malware & viruses difficult to
BMC Can Infect Host System
BMC De-Provisioning
Host System can Infect BMC
How real is all of this?
Researchers Views
Moore: “In addition to vulnerabilities in the IPMI protocol itself, most BMCs seem to suffer
from issues common across all embedded devices, namely default passwords, outdated
open source software, and, in some cases, backdoor accounts and static encryption
Moore: “The world of BMCs is a mess that is not likely to get better anytime soon, and we
need to be crystal clear about the risk these devices pose to our networks.”
Farmer: “Imagine trying to secure a computer with a small but powerful parasitic server on
its motherboard; a bloodsucker that can’t be turned off and has no documentation; you
can’t login, patch, or fix problems on it; server-based defensive, audit, or anti-malware
software can’t be used. Its design is secret and implementation old.”
Farmer: “It’s also the perfect spy platform: nearly invisible to its host, it can fully control the
computer’s hardware and software, and it was designed for remote control and
Farmer: In sum, you may not know it, but your goose may already be cooked and you’re
simply asking for the orange sauce. There is no easy fix, but I’d suggest a dialogue
between customers, vendors, and the security community for starters.... In any case, good
luck. We may all need it.
US CERT Alert TA13-207A
“Widespread Vulnerabilities in BMCs and the IPMI Protocol”
“New Gaping Security Holes Found Exposing Servers”
“IPMI: The most dangerous protocol you've never heard of”
“IPMI Protocol, BMC Vulnerabilities Expose Thousands of
Servers to Attack”
“Hackers can wipe or steal data from security holes in 300,000
“IPMI: Hacking servers that are turned off”
What can you do about it?
1. Do nothing and hope and/or pray...
2. Continue to use, understand issues and secure
3. Use alternative strategies
Department of Homeland Security (US-CERT) Solutions
►Restrict IPMI to Internal Networks
Restrict IPMI traffic to trusted internal networks. Traffic from IPMI (usually UDP port 623) should be restricted to a management
VLAN segment with strong network controls. Scan for IPMI usage outside of the trusted network and monitor the trusted network
for abnormal activity.
►Utilize Strong Passwords
Devices running IPMI should have strong, unique passwords set for the IPMI service. See US-CERT Security Tip ST04-002 and
Password Security, Protection, and Management for more information on password security.
►Encrypt Traffic
Enable encryption on IPMI interfaces, if possible. Check your manufacturer manual for details on how to set up encryption.
►Require Authentication
"cipher 0" is an option enabled by default on many IPMI enabled devices that allows authentication to be bypassed. Disable
"cipher 0" to prevent attackers from bypassing authentication and sending arbitrary IPMI commands. Anonymous logins should
also be disabled.
►Sanitize Flash Memory at End of Life
Follow manufacturer recommendations for sanitizing passwords. If none exists, destroy the flash chip, motherboard, or other
areas the IPMI password may be stored.
►Identify Affected Products
• Most server products
Select Recommendations from Farmer
If possible keep all IPMI network interfaces on their own segregated network. No other
computers should be using this managed network.
Severely restrict any network access to any BMC as well as the BMC’s capability for
communication. Monitor the traffic on the management network.
Restrict and alarm outbound network traffic and access for the BMCs
Add a layer of by placing a very secure computer to serve as a bastion host between the
management network and the unwashed masses of computers at large
Two-factor authentication with at least one factors unique to the management network
Build a set of best practices and policies around BMC and IPMI security.
Treat the BMCs as real servers (they are!): ensure that they’re monitored, scanned for
vulnerabilities, have their logs go to logging servers, etc.
Keep up-to-date with the most recent firmware for your BMCs as you can
Ensure that the BMC storage is wiped or reset, including passwords, when de-provisioning
Dan Farmer: IPMI++ Security Best Practices,
Additional Recommendations
Setup a dedicated management network, and limit IPMI to the network card connected to
the management network.
Review the BIOS configuration option for IPMI. If you can't have a physical management
network, at least try to use a VLAN
Keep IPMI firmware up to date. Do not use default passwords
Eliminate IPMI access over insecure protocols. Use HTTPS with certificates, or SSH
Integrate IPMI authentication with existing authentication systems like RADIUS and AD.
Review hardening options your IPMI implementation provides.
Limit access from IP addresses, or turn off various features you do not need
Inventory servers with IPMI capability
Alternative Strategies
1. Software Based Remote Access
Remote Desktop, Terminal Services, RDP
2. Hardware Based Remote Access
KVM-over-IP Switches
Serial Console Servers
Intelligent Rack Based PDU’s
Example: KVM-over-IP Access
LAN Users
Management Network
Server Rack(s)
WAN Users
Home, on-the-road,
Intelligent Rack PDUs
For More Information
Original research by Farmer & Moore and
Other References (below)
Press Articles (below)
IPMI Articles on the Raritan Blog
Server Manufacturer Information
For More Information
Dan Farmer: IPMI | Trouble | Security, A paper on IPMI & BMC Security
HD Moore - a penetration testers guide to IPMI and BMC’s
Widespread Vulnerabilities in BMCs and the IPMI Protocol - Frequently Asked Questions
IPMI: Hacking servers that are turned "off“
University of Michigan: Illuminating the Security Issues Surrounding Lights-Out Server Management
Dark Reading - New Gaping Security Holes Found Exposing Servers
“Illuminating the Security Issues Surrounding Lights-Out Server Management”
IPMI: The most dangerous protocol you've never heard of
IPMI Protocol, BMC Vulnerabilities Expose Thousands of Servers to Attack
Hackers can wipe or steal data from security holes in 300,000 servers
US-CERT: Risks of Using the Intelligent Platform Management Interface (IPMI)
Supermicro IPMI based on ATEN firmware contain multiple vulnerabilities
Remote management is absolutely necessary for server
But, existing BMC/IPMI tools have serious security issues
BMC/IPMI provides “virtually unlimited” remote control of the
If compromised, could lead to very serious consequences
You should understand the risks, solutions & alternatives
Take the appropriate actions to safeguard your infrastructure
Thank You ... and For More Information
Thank you for your time!
Contact me:
[email protected]
Web Site:
Search for ‘Raritan You Tube” for videos
Stop by Raritan Booth # 1112