Transcript Linux Kernel Crash Dumps

Linux Kernel Crash Dumps
Matt D. Robinson and Tom Morano
Silicon Graphics Computer Systems
Linux Kernel Crash Dumps
Contents
• Objectives
• LKCD Components
• Kernel Design Considerations
• Kernel Initiating Dumps
• Kernel Dumping Hooks/Execution
• Dump Initiation Code/Layout
• Dump Tunables
• Introduction to LCRASH
Linux Kernel Crash Dumps
Objectives
• LKCD created for Linux customers, support
personnel and Linux kernel engineers
• LKCD reduces MTBF and MTTR statistics
• Kernel problems are resolved more quickly
• As the Linux kernel becomes more complex,
the need for LKCD increases
Linux Kernel Crash Dumps
LKCD Components
LKCD Components
• Kernel changes to configure, catch kernel
failures, and save crash dumps
• User level scripts to save and configure
system memory to a crash dump
• LCRASH, the kernel crash dump analyzer
Linux Kernel Crash Dumps
Kernel Design Considerations
The biggest design considerations were:
• Dump Save Mechanism
• Raw I/O vs. Buffer Cache I/O
• Kernel Code Location
• Dump Storage
NOTE: Other crash dump products available
for Linux may use different dumping methods
than those described here
Linux Kernel Crash Dumps
Kernel Design Considerations
Dump Save Mechanism
Crash
Reset System
Kernel
Disk
Save Memory to
Swap Space in
Kernel
PROM
Disk
Save Memory to
Swap Space from
PROM/BIOS
Reset System
Linux Kernel Crash Dumps
Kernel Design Considerations
Kernel save method chosen because:
• PROM/BIOS is too architecture-specific
• reset/power-off may clear memory
• kernel disk driver restrictions
• no disk to filesystem validation at PROM
• code can be modified in kernel; PROM code
is difficult to make changes for (backwards
compatibility issues)
Linux Kernel Crash Dumps
Kernel Design Considerations
Raw I/O vs. Buffer Cache I/O
• Buffer cache locking prevents handling dump
workaround without major performance hit on
basic I/O
• Re-entry interrupt locking problem
• Raw I/O is not fully supported in Linux yet (in
the kernel) - kiobuf code needs more work
• IDE, RAID, etc., drivers need raw I/O hooks
(current plan is to create driver layer above to
avoid necessary locking)
Linux Kernel Crash Dumps
Kernel Design Considerations
Kernel Code Location
• Code changes are separated into generic and
architecture-specific files
– kernel/vmdump.c
– arch/<arch>/kernel/vmdump.c
• Additional modifications made to
linux/include/sysctl.h,
kernel/sysctl.c, and kernel crash hook
functions
Linux Kernel Crash Dumps
Kernel Design Considerations
Dump Storage
• Memory dumps are saved to swap space
• Swapping during boot-up is an issue
• Disk partition tables in memory -- could this
cause a data corruption problem?
• Cannot assume filesystem layer will be
available during crash
Linux Kernel Crash Dumps
Kernel Initiating Dumps
Initiating Dump Process
• Change to /proc/sys/vmdump/dumpdev
calls dump_open() in kernel
•dump_open() checks to ensure the
– device specified is a block device
– device points to a valid swap partition
– device has valid character device
file_operations table (currently SCSI
only, due to lack of raw I/O capability for
IDE disks)
Linux Kernel Crash Dumps
Kernel Initiating Dumps
• Errors in dump_open() are logged to system
log buffer
• Changes needed for 2.3 (without devfs) due
to mismatch between block and character
major/minor pairs for the same disk device
• Success of dump_open() displays:
dump_open(): dump device opened: 0x803 [sd(8,3)]
Linux Kernel Crash Dumps
Kernel Dumping Hooks
Kernel Hooks for Executing Crash Dump
•panic() was modified to perform SMP
freeze and to call dump_execute()
•die_if_kernel() or die() calls
dump_execute() after KDB, GDB, and
show_registers() are done
• NMI (Non-Maskable Interrupt) hooks still
needed for systems that support the
capability in hardware
Linux Kernel Crash Dumps
Kernel Dumping Hooks
Kernel Hooks and Parameters
•panic(): register state is not saved, panic
string is saved
•die_if_kernel() or die(): registers are
saved, panic string is generic (for now)
• Interrupt handlers vs. non I/O request lock
dumping needs to be differentiated
Linux Kernel Crash Dumps
Kernel Dump Execution
Kernel Dump Execution
•dump_execute() checks to see if dumping
is turned on
• If DUMP_NONE is set, it returns immediately
•__dump_execute(), which is architecturespecific, is called to save the dump
• Within __dump_execute(), dump header
values are saved, memory pages are saved,
and the function returns when complete
Linux Kernel Crash Dumps
Kernel Dump Layout
Kernel Dump Layout
Dump Header
Dump Page Headers
Dump Pages
Linux Kernel Crash Dumps
Kernel Dump Layout
• Dump header is written out first; it contains
basic information about dump
• Memory pages are written next, each with a
page header containing
–virtual address of the page in memory
–size of page (important if compressed)
–page flags (compressed, raw, dump end)
• The last step is a re-write of the dump header
which updates the total number of pages
written
Linux Kernel Crash Dumps
Kernel Dump Limitations
Kernel Dump Limitations
• Current interrupt crashes will lock up with reentry to disk driver function
• Dump header needs to be written out more
often
• Raw I/O capabilities need to be added in
kernel for more disk drivers (using kiobufs,
scatter-gather lists, etc.)
• Page typing needed for ordered dumps
• More architectures need to be supported
Linux Kernel Crash Dumps
Kernel Recovery of Crash Dump
Kernel Reboot After Crash
• During early boot-up, the system runs the
/etc/rc.d/rc.sysinit script, which in turn
runs /sbin/vmdump
•/sbin/vmdump runs with either the config
or save option
config
sets all dump tunables and attempts to
open the dump device
save
looks for a crash dump in dump device
and saves it to disk (if requested)
Linux Kernel Crash Dumps
Kernel /proc Tunables
Kernel Tunables
•/proc/sys/vmdump contains all LKCD
kernel tunables
•/proc/sys/kernel/panic is modified so
that the system reboots after LKCD creates a
crash dump
•dumpdev holds the name of the dump device
•dump_compress_pages determines if the
memory pages should be compressed
•dump_level indicates which pages to dump
to disk (only three levels currently supported)
Linux Kernel Crash Dumps
Kernel Dump Tunables
•/etc/sysconfig/vmdump holds all LKCD
tunables (the /proc tunables are changed
automatically):
DUMP_ACTIVE=1
DUMPDEV=/dev/vmdump
DUMPDIR=/var/log/vmdump
DUMP_SAVE=1
DUMP_LEVEL=4
DUMP_COMPRESS_PAGES=1
PANIC_TIMEOUT=5
Linux Kernel Crash Dumps
Kernel Dump Tunables
DUMP_ACTIVE
Determines if the crash dump scripts should
perform any actions; the default value is 1
(active). Set to 0 to not save or configure
system for crash dumps
Linux Kernel Crash Dumps
Kernel Dump Tunables
DUMPDEV
The name of the dump device; this typically is
/dev/vmdump. NOTE: It is recommended to
change what device /dev/vmdump points to
rather than to change this value directly, as
/dev/vmdump is normally a symbolic link.
Linux Kernel Crash Dumps
Kernel Dump Tunables
DUMPDIR
The name of the directory to save dumps to;
this typically is /var/log/vmdump.
DUMP_SAVE
Whether to save the crash dump to disk or
not. The system will still be configured to
save crash dumps regardless of the value of
DUMP_SAVE.
Linux Kernel Crash Dumps
Kernel Dump Tunables
DUMP_LEVEL
Determines how much memory (or not)
should be saved in the crash dump. Default
value is 4 (DUMP_ALL), although other values
such as 0 (DUMP_NONE) and 1
(DUMP_HEADER) are also valid. This sets
/proc/sys/vmdump/dump_level to the
same value (/sbin/vmdump config).
Linux Kernel Crash Dumps
Kernel Dump Tunables
DUMP_COMPRESS_PAGES
Determines whether to compress memory
pages when saving memory image to disk.
Defaults to 1 (compress). This sets
/proc/sys/vmdump/dump_compress_pages
to the same value (/sbin/vmdump config).
Linux Kernel Crash Dumps
Kernel Dump Tunables
PANIC_TIMEOUT
Changes the amount of time to sleep before
resetting the system after a software failure.
Changes /proc/sys/kernel/panic to the
same value (/sbin/vmdump config).
NOTE: This value should always be non-zero;
if zero, the system will spin indefinitely until it
is reset by hand.
Linux Kernel Crash Dumps
Kernel Dump Files
Kernel Dump Files
•vmdump.N holds the crash dump data saved
from DUMPDEV; it is a copy of the memory
image at the time of the system crash
•map.N is a copy of /boot/System.map
• Both files needed to perform crash analysis;
addresses in map.N point to values in
vmdump.N; if the files do not come from the
same kernel build, crash analysis may be
inaccurate
Linux Kernel Crash Dumps
Introduction to LCRASH
Overview of LCRASH
• Linux system crash dump analysis tool
• Provides access to kernel data in LKCD crash
dumps or live system memory
• Displays detailed information about a system
crash
• Can be used interactively or to generate
system crash dump reports
Linux Kernel Crash Dumps
Introduction to LCRASH
LCRASH Crash Dump Report
• General system information
• Type of crash
• Dump of the system log_buf
• List of active tasks
• Kernel stack trace showing the function calls
leading up to the point of the crash
Linux Kernel Crash Dumps
Introduction to LCRASH
LCRASH Interactive Commands
• For a more detailed examination of the
elements of a crash
• Kernel data displayed in a clear, easy-to-read
manner
• Invoked via an ASCII command line user
interface featuring command line editing and
command history
• Command output can be piped to utilities
such as more and grep
Linux Kernel Crash Dumps
Introduction to LCRASH
Examples of LCRASH commands
stat
vtop
ptype
symbol
dump
task
trace
dis
Displays pertinent system information and the
contents of the log_buf array
Displays virtual to physical address mappings
Displays arbitrary kernel structures from the crash
dump
Displays kernel symbol information
Displays the contents of system memory in a variety
of bases and data sizes
Displays key information from selected tasks or all
tasks running on the system
Displays a kernel stack trace for one or more task
Disassembles one or more machine instructions
Linux Kernel Crash Dumps
Introduction to LCRASH
The libklib Library
Library of low-level functions providing access
to the system dump and kernel symbol table
• Translate virtual addresses into physical
addresses
• Determine the address of kernel symbols
• Access memory pages in the dump or live
system memory
• Read in blocks of kernel data
• Access kernel data type information
Linux Kernel Crash Dumps
Introduction to LCRASH
Accessing Kernel Symbol Information
• The System.map file contains the virtual
address of all kernel symbols (variables,
functions, etc.)
• LCRASH parses the System.map file at
startup and builds an internal table of kernel
symbols
• Functions determine the address of a kernel
symbol, or locate a symbol matching a
particular address
Linux Kernel Crash Dumps
Introduction to LCRASH
Reading in Blocks of Data from a Dump
• LCRASH can’t access data in a system dump
directly
• Functions read in blocks of data from a
system dump or live system memory
– Kernel virtual addresses are translated into
physical address
– Memory pages in the dump are uncompressed
automatically
– The desired data is then copied into an LCRASH
buffer
Linux Kernel Crash Dumps
Introduction to LCRASH
Accessing Kernel Type Information
• Facilities provided for accessing extended
information in the kernel symbol table (when
built using the -gstabs compiler option)
– Kernel data type definitions, including type and
size of kernel structure members
– Data types of global variables
– Function parameters
– Source code line numbers of kernel functions
• Most production systems are not built with
the -gstabs flag
Linux Kernel Crash Dumps
Introduction to LCRASH
Generating Kernel Stack Traces
• LCRASH is able to generate kernel stack
traces without using frame pointers
– Various heuristics are applied to each stack frame
to determine what the PC, RA, SP, and frame
pointer should be
– Derived values are sanity checked to ensure they
are at least reasonable
– The entire stack trace is constructed before it is
displayed
• Most x86 kernels do not use frame pointers
Linux Kernel Crash Dumps
Introduction to LCRASH
• LCRASH commands for displaying kernel
stack traces
trace
displays a stack trace for one or more
active tasks
strace
displays an arbitrary stack trace using a
PC, RA, and SP provided on the
command line; or finds all valid stack
trace fragments in a stack
mktrace
manually constructs a stack trace, frameby-frame, using PC, RA, and SP values
supplied on the command line
Linux Kernel Crash Dumps
Introduction to LCRASH
Location of LCRASH Source
LCRASH source code was located in the
kernel source tree to ensure that
• LCRASH gets built along with the kernel
• LCRASH uses the same configuration options
and header files as the kernel
• an LCRASH exists that can analyze crash
dumps from a newly built kernel
• any changes to kernel header files that break
the LCRASH build get resolved quickly
Linux Kernel Crash Dumps
Introduction to LCRASH
Support for Multiple System Architectures
• LCRASH impacted by differences in system
architecture
• Functionality and source code organized
much like the Linux kernel
• There are architecture dependent and
architecture independent sections in both
LCRASH and libklib
• At the present time, i386 is fully supported
(alpha and ia64 are under development)
Linux Kernel Crash Dumps
Introduction to LCRASH
Adding New LCRASH Commands
• LCRASH was designed to make it easy to
add new commands
• Access to raw data in the crash dump is
made through calls to libklib API functions
• Provisions made for both generic and
architecture specific commands
Linux Kernel Crash Dumps
Introduction to LCRASH
For more information about LKCD,
review the web site at:
http://oss.sgi.com/projects/lkcd