Transcript Session Title - Andy Pedisich

The Complete
Handbook for Domino
Domain Monitoring
Andy Pedisich
Technotics
© 2009 Wellesley Information Services. All rights reserved.
In This Session
• Domino Domain Monitoring (DDM) is a powerful, yet complex tool
that is often overlooked by administrators
• If you are using Domino 6, 7, or 8 you are already a proud owner of
Domino Domain Monitoring Database and could already be using
its powerful functionality
• This Jumpstart will lay out all of the parts to help you see the
connections you need to successfully use this feature in your
R7/R8 domain
1
What We’ll Cover …
• Discovering the basics of Domino Domain Monitoring
• Understanding how DDM fits into your environment
• Crafting a perfect DDM data collection hierarchy
• Looking at simple but important DDM events
• Developing proactive probes
• Mastering enhanced events and corrective actions
• Working with the new Release 8 modular documents
• Testing and debugging Domain Monitoring
• Wrap-up
2
What Is Domino Domain Monitoring?
• Domino Domain Monitoring is part of a suite of tools that can help
you watch over your domain
 They are built into Notes/Domino
 Notes Administrator Client
 Domino server monitor
 Remote server console
 Event generators and handlers
 Statistics collection
 Activity logging
 LOG.NSF
 Domino Domain Monitoring
3
DDM Details
• DDM provides a single location where administrators
can access issues that are affecting multiple servers
and databases
 This is important because it lets you discover and resolve
problems quickly before they cause serious issues
• This helps to reduce the risk of unplanned downtime
4
Can DDM Be Used with All Releases of Domino?
• DDM became available in Notes/Domino Release 7
 Some of its probe features work on R6 servers
We’ll discuss this shortly
 But it is best suited for Releases 7 and 8
• DDM has been improved in Release 8

5
The Domino Domain Monitoring Database
• The DDM database is the central repository of all monitoring data
 This can be data collected by probes that you can configure
• The data in the DDM database can also include
 Result messages from event generators that you configured in
previous releases of Notes
 Results for routine checks that run as part of specific server
tasks, such as the router or replicator
6
Do All Administrators Use DDM?
• Many administrators don’t use the potential of DDM as much as
they should
 They are already overwhelmed by the monitoring features
of Domino
 They don’t understand how DDM fits into the architecture
7
Some Administrators Have Probe-Aphobia
• And some have become intimidated by the configuration of probes
 Probes are not a required part of DDM
 They are nice to have and fun to use, but DDM functions
without them
 You can get started without probes
 Then add them into the configuration when you become
more familiar with how DDM works
• If your Domain is running Release 7 or 8, DDM is working for you
right now
8
What We’ll Cover …
• Discovering the basics of Domino Domain Monitoring
• Understanding how DDM fits into your environment
• Crafting a perfect DDM data collection hierarchy
• Looking at simple but important DDM events
• Developing proactive probes
• Mastering enhanced events and corrective actions
• Working with the new Release 8 modular documents
• Testing and debugging Domain Monitoring
• Wrap-up
9
The Big Relationships in Monitoring
• EVENTS4.NSF – the Monitoring Configuration database is a key
file in your monitoring infrastructure
 It contains all of the specifics for your DDM monitoring
configuration
 For DDM probes
 For the DDM collection hierarchy (more about this later)
• But it also contains all the configuration specifics for:
 Statistic collection
 Event monitoring, such as tracking disk space or ACL changes
 Event notification, such as sending emails when problems
occur or logging issues into the Monitoring Results database
10
Simple Events
• When DDM was introduced, Lotus began to refer to events that
were generated automatically by servers as “simple events”
 Most events logged to the Domino console are simple events
 You’ve seen simple events logged to STATREP.NSF in the past,
and you’ll still find them there in R7 and R8
• You will also find simple events along with the results of probes
you create in DDM.NSF
 These are called “enhanced events”
11
This Pair of Databases Is Created Automatically
• EVENTS4.NSF was automatically built when your first server was
created
 DDM.NSF is created the first time you start an R7 or R8 server
• All EVENTS4.NSFs in your domain must have the same replica ID
 And all DDM.NSFs in your domain must be have the same
replica ID
12
This Configuration Is Extremely Important
• If the replica ID of these databases are not correct, then your
configuration for monitoring won’t be distributed consistently
throughout your domain
 And you won’t be able to collect DDM data centrally
• This business of having EVENTS4.NSFs with different replica IDs
is a very common issue
 Especially among domains that have been around for a while
• And we’ve seen many domains where the DDM database has a
different replica ID on some servers
13
Some System Databases Have Special Replica IDs
• The EVENTS4.NSF and DDM.NSF databases have very specific
replica IDs
 They are a hash of the replica ID of your domain’s NAMES.NSF
14
We Know What the Replica ID Should Be for EVENTS4
• The replica ID of system databases, such as EVENTS4 and
DDM.NSF, is derived from the replica ID of your domain’s
address book
Database
Replica ID
NAMES.NSF
852564AC:004EBCCF
CATALOG.NSF
852564AC:014EBCCF
EVENTS4.NSF
852564AC:024EBCCF
ADMIN4.NSF
852564AC:034EBCCF
DDM.NSF
852564AC:0A4EBCCF
 Notice that the first two numbers after the colon for the
EVENTS4.NSF replica are 02 and OA for DDM.NSF
15
This Configuration Must Be Correct
• Make sure that EVENTS4.NSF and DDM.NSF have the correct
replica ID throughout the domain by opening each and putting it
on your desktop
 The icons should stack, showing they are the same replica
 Check the replica ID by looking at the properties of
the databases
16
Errors You Might See If DDM.NSF Is Not Right
• If there is a DDM.NSF on every server but they aren’t all the same
replica ID, you’ll see the following error on the console every
couple of minutes:
 Unable to replicate with server Server2: None of the selected
databases have a replica on the server
 You’ll get this even if you have a much longer replication
interval scheduled
• To fix problems related to EVENTS4.NSF and DDM.NSF replica IDs,
you must delete the bad databases and restart the server
17
Want to Add Every EVENTS4.NSF to Your Desktop?
• Add this code to a button on your toolbar
 This is courtesy of Thomas Bahn

www.assono.de/blog
 It’s a great tool that I use almost every day
_names := @Subset(@MailDbName; 1) : "names.nsf";
_servers := @PickList([Custom]; _names; "Servers"; "Select
servers"; "Select servers to add database from"; 3);
_db := @Prompt([OkCancelEdit]; "Enter database"; "Enter the file
name and path of the database to add."; "log.nsf");
@For( n := 1; n <= @Elements(_servers); n := n + 1;
@Command([AddDatabase]; _servers[n] : _db) )
18
Add a Database to the Desktop
• This code will prompt you to pick the servers that have the
database you want on your desktop
 Then it will prompt for the name of the database
 And open it on all the servers you’ve selected
• Use it to make sure all the EVENTS4.NSF are the same replica in
your domain
19
Demo – Looking at Simple and Enhanced Events
20
What We’ll Cover …
• Discovering the basics of Domino Domain Monitoring
• Understanding how DDM fits into your environment
• Crafting a perfect DDM data collection hierarchy
• Looking at simple but important DDM events
• Developing proactive probes
• Mastering enhanced events and corrective actions
• Working with the new Release 8 modular documents
• Testing and debugging Domain Monitoring
• Wrap-up
21
Configure DDM for Centralized Data Collection
• DDM.NSF has the most value when it is the central repository for
all issues
 It will contain all of the issues that come from all of the servers
• This does not happen on its own
 There is no collection hierarchy set up by default
 Each domain has different monitoring requirements
22
Collection Hierarchy Is a Must
• Without a collection hierarchy, DDM probes run on a server and
report events to DDM.NSF that are on that server
 Then they remain only on that server’s replica of DDM.NSF
 You have to check the DDM database on each server to
evaluate problems and discover potential issues
 This is time consuming
 It reduces the time you could be spending solving
problems
 And you might miss important issues
23
Aggregate Data Centrally
• A DDM server collection hierarchy lets you aggregate the data
onto a key server or servers
 This must be configured in the EVENTS4.NSF
• The simplest hierarchy is to configure one server to collect from
all servers in the domain
24
More Complex Scenarios Are Possible
• Perhaps as you become more familiar with DDM, you’ll want to roll
up some data regionally
 So that regional administrators receive only information that is
pertinent to the server they maintain
25
Rolling Up the Data
• DDM data rollup propagates the probe results up the DDM server
collection hierarchy
 Data rollup is accomplished using Domino’s selective
replication to transport the data
 The replication formulas are created automatically when
you define your DDM server collection hierarchy
26
Selective Replication Formulas
• Each selective replication formula is specific to each server in the
Domino Domain Monitor replica
• When they are fully populated, the selective replication formula
references the collection server
 And all of its monitored servers
 The selective replication formula filters out all documents
from servers that are not members of the collecting
server’s hierarchy
27
Hierarchy Collection Interval
• The DDM system sets up its own collection interval
 Collection replication occurs about every five minutes
This interval cannot be modified
 It is not controlled through connection documents
• Every five minutes, each collection server uses pull replication to
get updates from the DDM database on each of the monitored
servers

28
What We’ll Cover …
• Discovering the basics of Domino Domain Monitoring
• Understanding how DDM fits into your environment
• Crafting a perfect DDM data collection hierarchy
• Looking at simple but important DDM events
• Developing proactive probes
• Mastering enhanced events and corrective actions
• Working with the new Release 8 modular documents
• Testing and debugging Domain Monitoring
• Wrap-up
29
Severity Levels in DDM
• Each event arriving into DDM.NSF has been assigned a
severity level
 The severity indicates the importance of taking action
 Fatal – Imminent system crash
 Failure – Severe failure that does not cause a system crash
 Warning (high) – Loss of function requiring intervention
 Warning (low) – Performance degradation
 Normal – Status messages
30
Address Issues by Severity Level
• Looking at issues by severity gives you the chance to deal with
the most important issues first
 They are broken out by severity category
31
Looking at a Failure
• This simple event in DDM tells us what the problem was and when
it occurred
 And it offers a possible solution
 This example shows the basics of DDM
32
Not All Simple Events Make It to DDM.NSF
• You’ll notice that not all of the events you see on the console are
delivered to DDM
 This is because of a default filter in EVENTS4.NSF
33
Filtering Specifics
• This default filter is put in place to “turn down the noise”
 Only important Fatal and Failure simple events are let through
This lets you save time by focusing on the real issues
• This filter can be customized by you, based on the requirements
within your own domain

34
Complete Listing of Messages
• There is a complete listing of all simple events and
their severities
 You can find them in the EVENTS4.NSF Monitoring
Configuration database
 Look in the Advanced section
35
Another Helpful View
• Release 8 added a new view to DDM.NSF
 You can see issues by database name

This lets you determine whether a problem is happening
on just one server or on every copy of the database in
the domain
 Very handy information when problem solving
36
What We’ll Cover …
• Discovering the basics of Domino Domain Monitoring
• Understanding how DDM fits into your environment
• Crafting a perfect DDM data collection hierarchy
• Looking at simple but important DDM events
• Developing proactive probes
• Mastering enhanced events and corrective actions
• Working with the new Release 8 modular documents
• Testing and debugging Domain Monitoring
• Wrap-up
37
Proactive Probes
• Remember that there are two kinds of events:
 Simple
 Enhanced
• Enhanced events are:
 Generated by a DDM probe configuration document
 Generated by a Domino Event Generator
• These are both events with specific target information such as the
server, database, agent, or user checked appearing in the DDM
event report header
• It’s time to check out probes
38
Probes Defined
• A probe is the investigative component of DDM
• Probes:
 Need configuration to be useful
 Are configurable by administrators
• A probe is defined as:
 A discrete check, or set of checks, configured to run against
one or more servers, databases, and services
 A probe returns status and results to the Domino Domain
Monitoring Database – DDM.NSF
 The DDM.NSF is created automatically
when the server starts
39
Analysis Probes
• Configure DDM using probe documents in EVENTS4.NSF
 Otherwise known as the Monitoring Configuration database
• You can create multiple probes for each feature area
 And you can individually configure each probe to run:



Selective checks
Against specific servers and/or databases
At specific times
40
What’s in These Probe Documents?
• Probe type and probe subtype
 For example, Security is a probe type

One of its probe subtypes is Best Practices
 This combination of probe type and probe subtype
creates a Security probe
41
Extra Information About the Probe Is Provided
• These probe documents also contain a general description of the
probe, its purpose, and its intended use
42
Configuration Specifics, Too
• Documents can also specify configurable probe targets
 The server(s) that will run the probe
 And in some cases the servers, database, etc., that the probe
runs against
 Where it’s applicable, there is also configurable scheduling
information – but not for all probe types
43
There’s More Inside
• Probe documents can also hold configuration specifics
 What the probe monitors
 What it should report on
 Thresholds to watch for

And what type of severity those thresholds represent
44
Plenty of Cool Probes
• R8 gives us 58 default DDM probes to work with
 R7 gives us 48 – still plenty to get us started
• You can get probing as soon as R7/R8 is up
• Just plug in your server info to get DDM started
 You can also create new probe documents

Define and customize your own probes
45
Many Types of Probes
• There are ten major types of probes in R8, nine in R7
 These probes can run two different ways:
On a schedule that you specify
 As an active monitor of things that happen in the domain
• Some probes can run either way
 On a schedule or as a monitor
 It depends on what you ask them to do
 Some can only run as a monitor or on a schedule

46
Summary Table for Probes
• Here’s a handy table to show how probes can run:
Probes
Administration Probes
Application Probes
Database Probes
Directory Probes
Messaging Probes
Operating System Probes
Replication Probes
Security Probes
Server Probes
Web Probes
Scheduled
Yes
Yes
Yes
Yes
Yes
Yes
Monitor
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
47
Establishing a Schedule
• Scheduled probes can be controlled with great granularity
 Set the probe to run:

Daily, Weekly, Monthly
 Beyond that, specific schedule settings can vary from
probe type to probe type
48
Don’t Worry About Getting Off Schedule
• If a Weekly/Monthly probe is missed, you can specify how you
want the probe to be handled:
 Ignore it completely
 Run the missed probe on startup
 Run the missed probe at the next time range
49
Zeroing in on Probes
• We’re going to focus on four probes that have a high value in
almost every Domino domain:
 Application probes
 Web probes
 Security probes
 Messaging probes
50
Agents Are Tracked by Application Probes
• Application probes monitor agents
 None of these probes can be scheduled
They are all real-time monitors
• Application probes have the following subtypes:
 Agents behind schedule
 Agents ranked by CPU usage
 Agents ranked by memory usage
 Long-running agents

51
Agents Are Tracked by Application Probes (cont.)
• Agents behind schedule
 Monitors agents and detects when an agent starts after its
scheduled time
• Agents ranked by CPU usage
 Evaluates the CPU usage for agents executed by the specified
process (Agent Manager or HTTP)
 Event threshold and severity are adjustable based on
CPU usage
 These have a relatively high overhead
52
Agents Are Tracked by Application Probes (cont.)
• Agents ranked by memory usage
 Evaluates agents memory usage executed by the Agent
Manager or HTTP tasks
 Note that evaluation results for the same agent may differ when
the agent runs in Agent Manager/HTTP
 Also, results from this probe can depend on HTTP settings
• Long-running agents
 Detects agents that run longer than a time you specify
53
Web Probes
• Web probes provide results as a Domino Event document in the
DDM database
• There are two subtypes:
 Configuration probe
 Verifies the accuracy of a Web server’s configuration
 Best Practices probe
 Compares servers’ configurations against a known
good server
• The default schedule for these probes is once a month
54
Web Configuration Probe
• The Web Configuration probe provides feedback related to the
settings of a Web server or list of Web servers
 This feedback is based on a Web profile
 The Web profile you reference could be a real server or a
profile you create that is “perfect” for your environment
55
Web Configuration Probe Details
• If the values in the two profiles differ, the probe recommends
values to either correct the configuration or improve server
performance or security
 This probe reviews field values in the Server document, the
Internet Site document, or the Web Server Configurations
document
 Depending on which documents exist in your domain
56
Web Best Practices Probe
• The Web Best Practices probe monitors HTTP-specific fields in
domain compared to a recommended value
 The HTTP-specific fields are in:
 The Server document
 The Internet Site document (if one exists)
 The Web Server Configuration document (if one exists)
57
The Best Practice DDM Report
• A DDM document details the values that don’t match the Best
Practice values recommended by Lotus
 And provides information on how to resolve any issues
 Could be corrections to server functionality, server
performance, or server security
 This helps improve overall security and Web performance
58
Security Check Probes
• Security probes assess the overall security of servers and
databases in your domain
• There are five subtypes:
 Best Practices
 Configuration
 Database ACL
 Database Review
 Security Review
59
The Five Security Check Probes
• Best Practices
 Compares a set of baseline security configuration settings to
the same settings in a domain
 This probe is a “Best Practices” security audit of the domain
• Configuration
 Compares settings in a specific Server document to settings in
a specified “good” Server doc
 This doc can be real or built by you as an example
 Lotus recommends a daily schedule for these probes
60
The Five Security Check Probes (cont.)
• Database ACL
 Monitors the access control privileges that groups and
individuals have in specified databases
 You designate the acceptable access levels on the
Specifics tab
• Database Review
 Reviews the security
properties for a specified
database
 Generates a report on probe
findings
61
The Five Security Check Probes (cont.)
• Security Review
 Generates a report on the security settings specified in the
Specifics tab of the probe document
 You have the option of selecting the “Directory Profile Note”
and the “Security settings in the Server Configuration
Document”
 And a review of all security settings in a Server doc
 This can really help to tighten your domain’s security
62
Domino Domain Monitoring for Messaging
• DDM includes 10 probe types
for messaging
 Two types have subtypes
 That’s 12 probes in all
• These probes cause events to
be logged in the DDM database
 That means problem solving
can be tracked and
assigned to an
administration team
• First three probes shown here
deal with mail flow:
 Mail DSN
 Mail Flow Statistic Check
63
The Mail DSN Probe
• This uses the Delivery Status Notification (DSN) functionality to
test SMTP mail flow
 Good for determining if a site is reachable; however, domain
must support DSN (SMTP extensions)
 Domino supports these extensions, so if the target SMTP
domain is a Domino domain, you can easily use this probe
64
Preparation Is Required for a DSN Probe
• The probe’s configuration must include an actual email address to
probe and the message is delivered!
 Make sure you specify a test user
 Messages delivered by this probe to the mail recipient will
not be automatically deleted from the mail file
 And remember that Yahoo mail does not support
DSN extensions
65
Mail Flow Statistic Check Probe
• This monitors the quantity of mail
 Generates an event in the Domino Domain Monitor when
pending messages meet or exceed the configured values
• Slack percentage reports a large quantity of mail in the system or
when there is more mail in mail.box than the router can process
 Slack percentage = (Mail.TotalPending - Mail.Dead - Mail.Held Mail.Waiting) / Mail.Waiting
66
The Mail Reflector Probes
• Reflector tests mail routing to both Domino and
non-Domino systems
 You must specify a mail recipient
 The recipient must be configured so that messages
received from this probe are sent back to the originator
 The subject of the original message must be contained
in the subject of the returned message
 Messages delivered by this probe to the mail recipient will not
be automatically deleted from the mail file
 Be sure to use a test user account
67
Mail Reflector Target User Is Not Hard to Configure
• Configuration can be done easily in the Notes environment
 Create a rule to have the recipient autoforward the message to
the ISpy mail-in database on the server executing the probe
 The messages are not deleted automatically
68
Scheduled Messaging Probes
• Message retrieval process state
 Verifies health of IMAP and POP3 tasks on specified servers
• Message retrieval TCP Port Health
 Verifies IMAP and POP3 are processing protocol requests
• NRPC Routing Status
 Tests local NRPC calls to verify mail routing is working
69
More Scheduled Messaging Probes
• Router Process State
 Monitors status of
the router
• SMTP Process State
 Monitors status of SMTP
process
• SMTP TCP Port Health
 SMTP successfully
processes protocol
requests
• The default schedule is once
per minute for these
“Process State” and “Port
Health” probes
70
This Probe Monitors and Is Not Scheduled
• Transfer Queue Check
 Monitors mail flow to
individual destinations
 Checks all
destinations
 Or just specific
destinations that you
would like to
troubleshoot
71
More on Transfer Queue Checks
• Transfer queue checks generate an event based on a threshold
you set for message retry and message count
 “Message retry” is the number of attempts to send a message
 “Message count” is the quantity of mail pending delivery
• Transfer queue check is a real-time monitor – the schedule’s not
configurable
72
What We’ll Cover …
• Discovering the basics of Domino Domain Monitoring
• Understanding how DDM fits into your environment
• Crafting a perfect DDM data collection hierarchy
• Looking at simple but important DDM events
• Developing proactive probes
• Mastering enhanced events and corrective actions
• Working with the new Release 8 modular documents
• Testing and debugging Domain Monitoring
• Wrap-up
73
The DDM Database
• The Domino Domain Monitor database holds events reported by
the probes
 It also holds events reported by:
 Event generators
 Certain server tasks, like the router and replicator
• It’s all placed into the DDM database
 But that’s not all!
74
The DDM Database Has More Than What’s on the Surface
• The DDM DB lets you review probable causes and possible
solutions for reported DDM events
 And it lets you assign them to an administrator for resolution
 You can open a link to the appropriate database from which
you can resolve a reported event
75
Explaining the Issue
• In some cases, the event will give details on the problem and
provide a link to help you get to the place where the problem
occurred
 One document might represent several occurrences of the
same event
76
Resolving Problems
• In some cases a Choose Solution button is available
 Use it to help you automatically implement a particular solution
to resolve an event
77
Here Is a Security Case Result From a Best Practice Probe
• This security probe was
configured to check the ACLs
of two databases
 NAMES and EVENTS4
• On a specific server
 ADMIN/Servers/DomLab
• We wanted to make sure
certain problem individuals
did not have more privileges
than they were supposed to
78
More Granularity
• We wanted to make sure that
 Eric has no greater than Designer access
 Kyle has no greater than Editor access
 Kenny has no greater than Author access
• We want to run the security check every day at 18:00
79
DDM Detects a Violation
• The DDM document produced by the probe indicates when the
probe detected the violation
 The doclink takes me to the probe document
 The probable cause is given as
 The ACL was modified in violation of company security
guidelines
80
Get the Details
• Clicking on Details takes me to the explanation of the issue
 It is exactly what was asked for in the security probe document
81
Here’s a Nice Feature
• A possible solution is offered by DDM
 Modify the ACL
• The corrective action section presents a button
 Clicking on the button opens the ACL of the database

I can easily set Eric’s privileges back to the
corporate guideline
82
The Major Features Explored
• This demonstrated the power behind DDM
 We were effortlessly shown the issue based on the probe
we configured
 We were shown when it occurred, the probable cause, and
a possible solution
 And were given direct access to corrective action
83
Assigning DDM Messaging Events for Resolution
• Events reported back to the DDM application become part of the
workflow
 One of the best parts about DDM is that it lets you assign
someone to resolve the problem
84
Using the Assign Button
• You can assign the event to a team member and add comments
about the task using the “Assign” button
 Or you can simply assign the event to yourself
85
Changes Are Tracked
• All changes you make to the event are tracked in the Event
Change History for easy reference
 Finally there is an easy, built-in process for tracking problem
resolution in your environment
86
Closing the Event
• When the event has been resolved, or at least when you are sick of
looking at it, you can close the event
 Closing the event removes it from the Open Events View
87
ACLs on DDM
• Take care in setting the ACLs on the DDM DB
 Make sure the settings match the requirements of your team’s
workflow responsibilities
Action
Minimum access and required roles
Read event documents
Read access
Add comments to your own events
Author access
Add comments to another
administrator’s event
Editor access
Assign or reassign your own events
Author access and the [Assign Events] role
Assign or reassign events regardless Editor access and the [Assign Events] role
of owner
Change state of your own events
Author access and the [Change State] role
Change state of event regardless
of owner
Editor access and the [Change State] role
88
Demo
89
Common Actions Available in Release 8
• In R8, all Domino Domain Monitor (DDM) events now have a
Common Actions button
 Gives you access to the most commonly performed actions for
investigating events
 Plus it adds context-sensitive help when appropriate
90
Looking at a Server NOTES.INI
• The common action that lets you look at a server’s NOTES.INI is
very quick and valuable
91
Console Access on the Fly
• The remote console access gets you to the server console
 You can send commands or get a live console
92
What We’ll Cover …
• Discovering the basics of Domino Domain Monitoring
• Understanding how DDM fits into your environment
• Crafting a perfect DDM data collection hierarchy
• Looking at simple but important DDM events
• Developing proactive probes
• Mastering enhanced events and corrective actions
• Working with the new Release 8 modular documents
• Testing and debugging Domain Monitoring
• Wrap-up
93
Modular Documents Are New in Domino 8
• Modular documents are a welcome addition to DDM
 But to understand why, we must first go back in time to see how
similar functionality was accomplished in Release 7
• In R7 DDM, some events could have automated solutions
 These automated solutions were hard-coded into the Events
documents in the Monitoring Configuration application
EVENTS4.NSF
94
R8 DDM Lets You Re-Use Solutions
• Modular documents are new
reference documents for
Probable Cause, Possible
Solution, and Corrective
Actions
 Otherwise known as PC,
PS,
and CA
• When you create an Events
document, the PC, PS, and CA
statements that you choose to
include in the document are
referenced from modular
documents
95
Re-Use Modular Documents
• The big benefit is that you only need to define these statements
once, and can use them multiple times
 Domino 8 comes with over 1,000 modular documents
96
Add Your Own
• If you have a solution to some event specific to your
own environment, you can create a custom modular document
 You can include:
 Formula language code
 LotusScript code
 Or run an agent in a database
97
Actions Are Matched with Events
• Match up the modular document with the event in the Monitoring
Configuration application
98
Changes Might Take Time
• Events and modular documents are cached
 You might find that updates to events and modular documents
are not reflected in DDM.NSF right away
 If you’re impatient, restart the Event task to ensure
updates to Events and modular documents are reflected
in DDM.NSF immediately
99
Updated Events Documents
• You will notice that there is a new button in the upper right-hand
corner of the events documents
 It toggles from Display Stock Entries to Hide Stock Entries
 This represents a very important modification to the
process of applying action through DDM
100
Multiple Entry Types Mean Greater Flexibility
• Event documents have three categories of Probable
Cause/Possible Solution and Corrective Actions
• The first tab contains the Lotus Entries
 These are, of course, provided by Lotus
 Do not modify or delete these entries
 If you want to disassociate the entry with the event, simply
edit the document and uncheck the Enabled box
101
Your Custom Entries
• Add your own references to PC, PS, and CA on the Custom
Entries tab
 The interface looks similar to the Lotus Entries tab, but you can
only add up to two Probable Cause/Possible Solution actions
 And there is no Enabled setting as with the Lotus entries
• These settings will be retained as you move forward upgrading
from Domino 8.x
102
Stock Entries for Release 7
• Stock entries are pre-Domino 8 entries for PC, PS, and CA
 These are available for backward compatibility
Lotus entries have replaced the stock entries for Domino 8
and all subsequent releases
• If you are creating a custom PC, PS, and CA with the
Domino 8 Monitoring Configuration template (EVENTS4.NTF) on
Domino 7 servers, use the stock entries
 If customizing on Domino 8 servers, use custom entries

103
Merging Old into New
• If you have made modifications to Events documents on Domino
7, those modifications can be overwritten when you upgrade to
Domino 8
 Do you want to merge your old Domino 7 stock entries into
Domino 8 stock entries?
 Then follow the nine-step process detailed in the
Administrator Help application
 Do you want to merge your old Domino 7 stock entries into
Domino 8 modular documents?
 Then follow the nine-step process detailed in the
Administrator Help application
104
Where to Find the Help You Need
• You’ll find this information in the Monitoring section of
Administrator Help
 Domino Domain Monitoring subheading
 Event-related documents
105
New Role in DDM ACL
• Many events have corrective actions associated
with them
 Only users with the Execute CA role in the DDM ACL are able
to access the command actions and the corrective action
text and links
 This ensures that only qualified team members will be able
to make the changes
106
What We’ll Cover …
• Discovering the basics of Domino Domain Monitoring
• Understanding how DDM fits into your environment
• Crafting a perfect DDM data collection hierarchy
• Looking at simple but important DDM events
• Developing proactive probes
• Mastering enhanced events and corrective actions
• Working with the new Release 8 modular documents
• Testing and debugging Domain Monitoring
• Wrap-up
107
Why Debug DDM?
• You might want to test DDM to determine if you’ve got it set up
correctly
 But who wants to wait around for an error condition to
actually happen?
 In fact, no one wants to wait
 We’re engineers! We’re impatient!
• That’s why Lotus built testing and debugging into DDM
108
Get the Schedule for Your Probes
• Issue this command from the console to get a schedule for
your probes
 Show schedule –ddm
• You can shorten the command to this one
 Sh sch -ddm
109
Getting a Probe
• You can make the Event task tell you which probes have been
enabled using the following console command
 Tell Event dumpprobes
• The Event task will show which probes are enabled
 If a probe is the type that can be scheduled, it will also show
when the probe will run next
110
Running Probes On-Demand
• You can also run your probes whenever you’d like using
the following console command
 Tell Event Run Probe ####
Substitute your probe’s ID number for the #### above
 Use what was displayed with dumpprobes command

111
Getting the Probe IDs for Duplicate Probe Types
• If you have duplicate probe types, getting to the ID is a little
more work
• You must go to the probe document and look at the Note ID on the
document properties page
 Use a calculator to convert that hex number to a decimal
112
Getting a Probe Dump
• Want to know more about your events?
• Get the event task to just dump it all with this console command
 Tell event dump
113
A Helpful NOTES.INI Parameter
• Add the following parameter to the server’s NOTES.INI
 DEBUG_EVENT=1
• This debug variable can be used for all processes under the Event
task
 This includes probes, schedules, and other information that
you can use to test your DDM configuration
• This is a global debug variable for the Event task and includes
information about probes
114
You’ll See Results When You Restart the Server
• The parameter will cause the Event task to display what’s scheduled
right on the console without entering a console command
115
What We’ll Cover …
• Discovering the basics of Domino Domain Monitoring
• Understanding how DDM fits into your environment
• Crafting a perfect DDM data collection hierarchy
• Looking at simple but important DDM events
• Developing proactive probes
• Mastering enhanced events and corrective actions
• Working with the new Release 8 modular documents
• Testing and debugging Domain Monitoring
• Wrap-up
116
Resources
• Lotus Education on Demand: Domino Domain
Monitoring (DDM)
 www-1.ibm.com/support/docview.wss?uid=swg27007060
• Notes/Domino Best Practices: Domino Domain Monitoring (DDM)
 www-1.ibm.com/support/docview.wss?uid=swg27009312
• Redpaper: Lotus Domino Domain Monitoring
 www.redbooks.ibm.com/abstracts/REDP4089.html?Open
• What is Domino Domain Monitoring (DDM)?
 www-01.ibm.com/support/docview.wss?uid=swg21215726
117
7 Key Points to Take Home
• Make sure that EVENTS4.NSF and DDM.NSF each have their own
•
•
•
•
•
•
special replica IDs in your domain
Start with a flat data collection hierarchy and make it more
complex only if your requirements call for it
Addressing DDM documents with the highest severity level first
ensures that you’ll tackle the most critical issues daily
Adjust the DDM filter to keep the noise at a minimum
Use probes proactively to keep your domain running smoothly
Assigning issues to team members for follow up lets them grow
and lets you focus on the more difficult issues
Write your own corrective actions for issues that are unique to
your enterprise
118
Your Turn!
How to contact me:
Andy Pedisich
[email protected]
http://www.technotics.com
http://www.andypedisich.com
119