Transcript Mark Russinovich Technical Fellow Microsoft Corporation Session Code: CLI402 *Portions derived from David Solomon’s Windows Internals Seminar.

Mark Russinovich
Technical Fellow
Microsoft Corporation
Session Code: CLI402
*Portions derived from David Solomon’s Windows Internals Seminar
About Me
Technical Fellow, Microsoft
Co-founder and Chief Software
Architect of Winternals Software
Co-author of Windows Internals 4th and 5th
Edition and Inside Windows 2000 3rd Edition
with David Solomon
Author of TechNet Sysinternals
Home of blog and forums
Contributing Editor TechNet Magazine,
Windows IT Pro Magazine
Ph.D. in Computer Engineering
Pushing the Limits
Resource exhaustion is obviously not a good thing
At the minimum it causes service outage
At worst it can cause data loss or even corruption
As a Windows systems administrator you need to know
the limits
For capacity planning
For systems monitoring
This session explains core Windows kernel limits, where they
come from, and how to monitor them
I’ll use Sysinternals Testlimit and Notmyfault to stress resources:
http://technet.microsoft.com/enus/sysinternals/bb963901.aspx
Outline
Virtual Memory
Physical Memory
Paged and Nonpaged Pool
Processes and Threads
Handles
Address Space Limits
One virtual memory limit is address space
Process can run out of space to allocate
virtual memory
Typically only an issue for 32-bit processes
Use Testlimit -r (reserve memory) to see address
space exhaustion:
Process Explorer Virtual Size shows amount of
address space used
32-bit x86 Address Space
32-bits = 2^32 = 4 GB
Default
2 GB
User
process
space
2 GB
System
Space
4GT Address Space
(boot.ini: /3GB or /USERVA)
(BCD: increasuserva)
3 GB
User
process
space
1 GB
System Space
64-bit Address Spaces
64-bits = 2^64 = 17,179,869,184 GB
32-bit process on 64-bit Windows
x64
(AMD64
& Intel 64)
= 4 GB
8192 GB
(8 TB)
User
process
space
7152 GB
(7 TB)
User
process
space
~8192 GB
(8 TB)
System
Space
~8192 GB
(8 TB)
System
Space
IA-64
Virtual Memory Types
Address space breakdown
Shareable (e.g. EXE, DLL, shared memory, other
memory mapped files)
Private (e.g. process heap)
Reserved (not yet committed)
Free (not yet defined)
Performance counters available:
Private Bytes – private virtual memory
Virtual Bytes – total of shareable+private+reserved
No separate counters for Shareable or Reserved
or Free
Looking at a Process Address Space
Sysinternals VMMap utility shows addressspace usage
Tracking Process Commit Usage
Most virtual memory problems are due to a process
leaking private committed memory
Heap, GC heap, language heaps (CRT)
Testlimit -m
Viewing process commit usage:
Private Bytes perf counter
Process Explorer’s Private Bytes:
Before Vista: Task Manager’s Virtual Size
Vista and later: Task Manager’s Commit Size
The System Commit Limit
Another virtual memory limit is total virtual
memory: system commit limit
Committed virtual memory must be backed either
by physical memory or stored in the paging file
Sum of (most of) physical memory and paging files
The system commit limit can change
Default paging file (System Managed):
Minimum: 1.5x RAM if RAM < 1 GB; RAM otherwise
Maximum: 3x RAM or 4 GB, whichever is larger
Viewing the System Commit Limit
Process Explorer tracks the current commit limit in the
System Information window:
Prior to Vista, Task Manager
shows it as “PF Usage”:
Task Manager on Vista and
higher doesn’t show a graph:
Exhausting the System Commit Limit
64-bit Testlimit -m (committed memory) will
exhaust the system commit limit before its
address space:
You can increase the System commit limit by
adding RAM or increasing the pagefile size
Sizing the Paging File
Many recommendations use a formula based on RAM
(1.5x, 2x, etc.)
Actually, the more RAM, the smaller the paging file needed
Should be based on workload usage of committed
virtual memory
Look at commit after workload
has run
Pre-Vista: Task Manager
Vista+: Process Explorer
Apply a formula to that to
give buffer (1.5x or 2x)
Make sure it’s big enough
to hold a crash dump
Outline
Virtual Memory
Physical Memory
Paged and Nonpaged Pool
Processes and Threads
Handles
Windows Physical Memory Limits
The amount of physical memory Windows will
use depends on:
SKU licensing
32-bit vs 64-bit
Tested systems
Kernel address space
See “Memory Limits for Windows Releases” for
a complete list:
http://msdn.microsoft.com/enus/library/aa366778(VS.85).aspx
Physical Memory on 64-bit Systems
Kernel address space limits it to
2 PB (Petabytes)
x64 page table structures limit physical memory
to 48-bits physical addressing (256 TB)
Current x64 hardware supports up to 1 TB
64-Bit Server
Current Windows Server limits:
Windows Server 2003: 16 GB (Standard) – 2 TB (Datacenter)
Windows Server 2008: 32 GB (Standard) – 2 TB (Datacenter)
Windows Server 2008R2: 8 GB (Foundation) – 2 TB (Datacenter)
For Datacenter Edition, the limit is based on SKU and
hardware testing
We won’t support what we can’t test
Limits have gone up even for a particular release of Windows
because of access to larger machines
Datacenter Edition maxes out all licensing,
including memory
2 Terabytes
The Windows architecture can support much
more, but as of now, the largest system we’ve
tested is 2 TB (Itanium):
32-Bit Server
Windows 32-bit Server limits:
Windows Server 2003: 2 GB (Web) – 128 GB (Datacenter)
Windows Server 2008: 4 GB (Standard) – 64 GB (Datacenter)
Windows Server 2008 R2: n/a
The 32-bit Server limit is based on the kernel
address space
PFN entry is 28-bytes so 128 GB
requires 890 MB
Limits other system code and data
Also why limit is lower when booted 4GT
User
Address
Space
PFN Database
System
Client Systems
64-bit client limits solely based on licensing:
512 MB in Windows XP Starter
128 GB in Windows Vista Ultimate
192 GB in Windows 7 Ultimate
32-bit client limit is 4 GB
Limited by potential driver incompatibility
And usable amount might be less than 4 GB…
Drivers and Physical Memory
To be safe, client limits
physical memory addresses
to 4 GB
Some client device drivers
still truncate addresses > 4
GB
Client systems with > 4 GB
are recent
0
0.2 GB
DMA
Destination
Servers are safe with > 4
GB
Have had more than 4 GB
for longer
Devices are more
mainstream
Policy allows only WHQLsigned drivers
0x00032000
Device Data
4 GB
4.2 GB
6 GB
Target
0x00000001 0x00032000
Device Memory
0
Device memory can push physical
memory above 4 GB
Firmware maps all device memory
below 4GB
For compatibility with older operating
systems
Biggest device memory consumer is
video
Result is that some physical memory
may lie above 4 GB
Watch out for Shared Video Memory
(UMA)
Video RAM
4 GB
Inaccessible
RAM
Physical Memory Management
Working set: all the physical pages “owned” by a process
Pre-Vista: Task Manager’s “Mem Usage”
Vista+: Task Manager’s “Memory – Working Set”
Process Explorer’s “Working Set”
System has its own working set (broken into multiple in Windows 7)
System keeps physical pages on one of several lists
Working sets (process and system)
Available: Free page list, Standby page list (file cache), Zero page list
Modified page list: pages that must be saved before being repurposed
Lists are implemented by entries in the “PFN database”
Each page of physical memory has an entry in the database
The Memory Manager watches paging activity to determine
how to assign memory
Notepad
Word
Explorer
System
Available
Working Set Lists
newer pages
older pages
A process always starts with just enough working set to describe
the address space
It then incurs page faults when referencing a page that isn’t in its
working set
Many page faults may be resolved from memory (from other working
sets, the standby or Modified page lists)
The Memory Manager can take away pages and reassign
them (trimming)
Note: the amount of physical memory assigned to a process is
invisible to the process
It simply affects the performance of the process
Do You Have Enough Memory?
There’s no sure-fire rule or counter to tell if you if you have
enough memory
The general rule: available memory remains generally low
Use Perfmon to monitor available memory over time
Use Process Explorer, or on Vista and later Task Manager, to monitor
physical memory usage
Use Process Explorer or Task Manager to see instantaneous value
Watch in Process Monitor for excessive reads from paging file
Outline
Virtual Memory
Physical Memory
Paged and Nonpaged Pool
Processes and Threads
Handles
Paged and Nonpaged Pool
Nonpaged pool:
Backed by physical memory
Kernel and device driver heap used by code paths that can’t
take interrupts (ISRs and DPCs) or that hold spin locks
Page fault in those paths results in IRQL_NOT_LESS_OR_EQUAL
Paged pool:
Backed by virtual memory (can be saved to paging file)
Largest consumer is typically Registry
Drivers use ExAllocatePoolWithTag to allocate both
pool types
Parameter specifies pool type
Tag is 4-byte value that should be human readable and is
used to help track pool usage
Viewing Pool Usage
There are three performance counters that indicate
pool usage:
Pool nonpaged bytes
Pool paged bytes (virtual size of paged pool – some may
be paged out)
Pool paged resident bytes (physical size of paged pool)
No performance counter for viewing maximums
Use Process Explorer to view
current and maximums
Must configure symbols to
see maximum
System Information dialog shows usage
Pool Limits
Pool maximums are based on system address space
and physical memory
Nonpaged Pool
XP, Server 2003
Vista, Server 2008,
Windows 7, Server 2008 R2
32-bit
64-bit
up to 1.2GB RAM: 32-256 min( ~400K/MB of RAM,
MB
128GB)
> 1.2GB RAM: 256MB
min( ~75% of RAM, 2GB) min(~75% of RAM, 128GB)
Paged Pool
XP, Server 2003
32-bit
XP: up to 491MB
Server 2003: up to 650MB
64-bit
min( 4 * nonpaged pool
limit, 128GB)
Vista, Server 2008,
Windows 7, Server 2008 R2
min( system commit limit, min( system commit limit,
2GB)
128GB)
Testing Pool Limits
Sysinternals Notmyfault can
leak paged and nonpaged pool
Exhausting either pool type
leads to erratic behavior and
likely data loss or corruption
Tracking Pool Leaks
Poolmon from the Windows Driver Kit shows pool
usage by tag
http://www.microsoft.com/whdc/devtools/WDK/
Search strings of drivers
for pool tag:
strings * | findstr <tag>
Pool Leaks and Crashes
If a system crashes and you suspect a pool leak, use
!vm to confirm:
Use !poolused to view pool usage by tag
Use 1 for nonpaged pool and 4 for paged
Outline
Virtual Memory
Physical Memory
Paged and Nonpaged Pool
Processes and Threads
Handles
Thread Limits
User-mode stack area is reserved and
committed as needed
Stacks grow down in memory
“Guard” pages trigger stack expansion
Committed
Thread
Stack
Guard
Reserved
Before Expansion
Stack
Grows
Down
Committed
Guard
Reserved
Before Expansion
32-Bit Thread Limits
Default 2 GB address space and stack reserve:
2 GB / 1 MB = ~2,000 threads
32-bit thread on 64-bit Windows has additional
overhead of 256 KB 64-bit stack
64-Bit Thread Limits
On 64-bit Windows, the system commit limit
will be hit before address space exhaustion
Even with default 2 MB stacks
Process Limits
Minimum process working set
also charged against “resident
available memory”
Reserved physical memory so that
process can run
Default minimum is 200 KB, which
Process Explorer shows
Every active process has one
thread, so process limit will
be lower than thread limit
Outline
Virtual Memory
Physical Memory
Paged and Nonpaged Pool
Processes and Threads
Handles
Handles and Objects
Windows defines objects to represent
operating system resources
There are 42 in Windows 7
Use Sysinternals Winobj to view
defined objects
When a process wants to interact
with a resource, it opens it
That creates a handle in the process’ handle table
Handle entry points at the object and records granted access
Subsequent operations reference the handle
Handle
Table
Entry
Pointer
Accesses
Object
Handle Limits
A process handle table can grow to about
16 million handles
Handle tables are stored in paged pool, so that
can also be a limiter
32-bit Windows: 8 bytes per entry
64-bit Windows: 16 bytes per entry
Tracking Handle Leaks
A handle leaker will show a growing handle count
over time
Task Manager and Process Explorer show handle
count columns
Process Explorer also shows it in process properties dialog
Process Explorer uses difference highlighting to show
newly opened and closed handles
Tracking Handle Call Stacks
Windows can capture call stacks at the time of handle
open and close
Use Application Verifier to track from process start
Free download from Microsoft
Use Windbg to enable and disable dynamically
!htrace –enable, !htrace –disable
Viewing Handle Call Stacks
Use Windbg to view stacks
Use !htrace (no parameters) to show all recorded stacks
Use !htrace –snapshot and !htrace –diff to see differences
Conclusion
Knowing system limits can help you avoid
resource exhaustion
By knowing how to track resource usage you can
identify potential problems and root cause them
Visit my blog for posts on the limits I’ve covered
and others
Come to my other sessions:
SIA301 Windows and Malware: Which Features Are Security and
Which Aren't
Tomorrow at 9am
CLI301 Case of the Unexplained... Windows Troubleshooting
Tomorrow at 1pm
Complete an evaluation
on CommNet and enter to
win an Xbox 360 Elite!
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should
not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.