Daniel Pearson David Solomon Expert Seminars SVR311 Daniel Pearson Started working with Windows NT 3.51 Three years at Digital Equipment Corporation Supporting Intel and Alpha.
Download ReportTranscript Daniel Pearson David Solomon Expert Seminars SVR311 Daniel Pearson Started working with Windows NT 3.51 Three years at Digital Equipment Corporation Supporting Intel and Alpha.
Daniel Pearson David Solomon Expert Seminars SVR311
Daniel Pearson
Started working with Windows NT 3.51
Three years at Digital Equipment Corporation Supporting Intel and Alpha systems running Windows NT Seven years at Microsoft Senior Escalation Lead in Windows base team Worked in the Mobile Internet sustained engineering team Instructor for David Solomon, co-author of the Windows Internals book series
Agenda
Components of performance analysis Understanding the tools for troubleshooting and analyzing performance issues Troubleshooting CPU, memory and disk I/O issues using various Windows tools * Portions of this session are based on material developed by Mark Russinovich and David Solomon
Components of Performance Analysis
Event Tracing for Windows Core component of the operating system Kernel mode data structures Used to store information about the system and system objects that can be read by various tools e.g. dt nt!_KTHREAD KernelTime CPU performance monitoring events Refer to the Intel 64 and IA-32 Architectures Software Developer’s Manual
Event Tracing for Windows
Built in to the system High performance, low overhead and scalable 2.5% CPU usage for a sustained rate of 10,000 events/sec on a 2 GHz CPU 1 Operations throughout the system that are of interest to performance are fully instrumented e.g. process and thread activity, registry I/O, disk I/O 1. Milirud, Michael. 2008. Windows Performance Analysis: Using Windows Performance Tools. Presented at Microsoft's WinHEC conference, November 5-7, Los Angeles, CA.
Event Tracing for Windows
Uses a buffering and logging mechanism implemented in the kernel Per-processor buffers that are written to disk by an asynchronous writer thread Ability to enable and disable tracing dynamically Supports a managed code provider
Sysinternals Utilities
Sysinternals Utilities
Process Explorer Useful for displaying which files, registry keys and other objects processes have open and which DLLs they have loaded Process Monitor Useful for showing real-time file system, registry and process & thread activity Available for download from the TechNet site http://technet.microsoft.com/sysinternals
Resource Monitor
Resource Monitor
Included with Windows Vista and greatly enhanced in Windows 7 and Windows Server 2008 R2 Allows the viewing of CPU, memory, disk and network resources as well as handles and modules in real time Ability to end, suspend and resume processes as well as to start, stop and restart Windows services Useful for identifying the highest resource consumers by individual resource type, e.g. CPU Able to list the wait chain tree of a process to determine if a process is waiting on another
Using Resource Monitor
Performance Monitor
Performance Monitor
Queries performance counters that measure system state or activity Current values are read at specific intervals Performance counters are included in the operating system and can be included as part of applications Able to collect event trace data from trace providers that report actions or events Can combine multiple trace providers into a single session Configuration information can be collected from registry keys at a specific time or interval
Using Performance Monitor
Windows Performance Analyzer
Windows Performance Analyzer
Part of the Windows Performance Toolkit Support for both x86, x64, and IA64 architectures Consists of three primary programs xperf.exe
Used for controlling tracing and processing trace data xbootmgr.exe
Automates on and off state transitions and captures traces during those transitions xperfview.exe
A graphical trace visualization tool to represent data in the form of interactive graphs and summary tables
Windows Performance Analyzer
Primarily uses the Event Tracing for Windows infrastructure built in to the system Can be enabled or disabled at any time without requiring a system or process restart Supports symbol decoding, sample profiling, and recording of call stacks on kernel events Designed to be used during automation All the functions of the tools are available via the command line tool xperf.exe
Support for Earlier Systems
The Windows Performance Toolkit will fail to install on Windows XP and on Windows Server 2003 although data collection is supported Copy xperf.exe and perfctrl.dll
Trace analysis is only supported on Windows Vista and later systems
Capturing a Performance Trace
Kernel options divided into two parts Kernel Flags Identified by the use of uppercase characters e.g. PROC_THREAD, LOADER, PROFILE Kernel Groups Indentified by the use of title case characters e.g. Base, Diag, Latency, FileIO Kernel Groups are made up of a collection of Kernel Flags e.g. SysProf = PROC_THREAD+LOADER+PROFILE Flags and groups are separated by the ‘+’ token e.g. xperf.exe -on FileIO+DISK_IO_INIT Can use a mixture of upper and/or lower case text e.g. xperf.exe -on sysprof+interrupt+dpc
Merging of Performance Trace Data
Traces can be copied to another system for analysis The trace file should be “merged” on the collection system before analysis to include additional system information xperf -d trace.etl
System and symbol information Trace Merged trace Kernel trace XPerf
Using the Windows Performance Toolkit
Understanding CPU Activity
Windows uses 32 priority levels The system implements a preemptive, priority driven scheduler Priority adjustments can be applied to threads in the “dynamic” range At least one runnable thread with the highest priority will be running 31 16 15 Real time Dynamic 0 Thread – Priority Current Thread – Thread State
Context Switching
A switch from one thread to another is known as a context switch Switching involves saving the hardware state of a thread and restoring the state of another When a thread is scheduled, that thread’s context switch count is also incremented The context switch count represents how often a thread begins running, not how long it ran System – Context Switches/sec Thread – Context Switches/sec
Time Accounting Quirks
Looking at total CPU time for each process may not reveal where the system has spent its time CPU time accounting is driven by an interrupt timer which is set by the Hardware Abstraction Layer Usually at either 10 or 15 msec intervals Thread execution and context switches that happen between clock intervals are not accounted for e.g. a thread runs and enters a wait before the clock fires Thus threads may run but never get charged Process – % Privileged Time Process – % User Time
Time Accounting Prior to Windows Vista
Windows accounted for CPU time based on the interval clock timer Thread quantum expiration was not always fair A thread might get almost no turn Threads were also charged for interrupts that occurred while they were running Idle Idle T1 T2 T2
Time Accounting Since Windows Vista
Windows Vista and later reads the Time Stamp Counter during every context switch The actual CPU cycles consumed are charged to a thread Any interrupt time is not charged to the interrupted thread Allows for more accurate quantum accounting A thread gets at least one turn and at most will be given one turn plus an additional tick Idle Idle T1 T1 T2
Troubleshooting High CPU Utilization
Understanding Memory Management
Windows provides two system memory pools Nonpaged Pool and Paged Pool Used for system wide persistent data Prior to Windows Vista, pool sizes were a function of memory size and whether or not the system was configured as a server or a workstation Windows Vista introduced the concept of a dynamic system address space Memory – Pool Nonpaged Bytes Memory – Pool Paged Bytes
Dynamic System Address Space
In 32-bit Windows Vista and later, virtual memory is assigned as needed Permits larger paged, nonpaged, and session pools Components still cannot exceed 2 GB on 32-bit systems On 64-bit systems, address space regions are configured to their current maximum limits for all memory sizes
Troubleshooting Memory Leaks
Understanding Disk I/O
Background I/O, e.g. indexing, disk defragmenting, interferes with foreground interactive tasks, e.g. reading e-mail In earlier systems, the only way to prioritize work was based on thread CPU priority Windows Vista introduced I/O priority and I/O bandwidth reservation Physical Disk – Current Disk Queue Length Physical Disk – Disk Transfers/sec
I/O Priority Levels
Critical – Used by the Memory Manager High – Not currently implemented Normal – Default priority, e.g. Microsoft Outlook Low – Task priority, e.g. application prefetching Very Low – Background activity, e.g. antivirus Critical High Normal Low Very Low MM WMP Outlook Prefetch Defrag Indexer AV Hierarchy Idle
Troubleshooting Disk I/O
Additional Information
Windows Internals 5 th edition Windows Performance Analysis Developer Center http://msdn.microsoft.com/performance Windows Server Performance Team Blog http://blogs.technet.com/winserverperformance Ask the Performance Team Blog http://blogs.technet.com/askperf
Additional Information
David Solomon Expert Seminars offers training on Windows Internals both as public and private workshops and public webinars via the Internet Currently scheduled up and coming classes Public workshop in London scheduled March, 2010 Public webinar scheduled for January, 2010 Visit http://www.solsem.com
for further course descriptions and up to date information
Resources
www.microsoft.com/teched Sessions On-Demand & Community www.microsoft.com/learning Microsoft Certification & Training Resources http://microsoft.com/technet Resources for IT Professionals http://microsoft.com/msdn Resources for Developers
Complete an evaluation on CommNet and enter to win an Xbox 360 Elite!
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.