Transcript Slide 1
BP 401 - Admin Zero to Hero
in 60 Minutes
The question is no longer, "How can we?"
The question now is, "How should we?"
Andrew Pollack, President
Northern Collaborative Technologies
Language Note
I realize that for some of you, English is not your primary
language, and for others, my accent is not the same as yours.
If you are having trouble understanding me during this talk,
please raise your hand and I will try to slow down and speak
more clearly.
Thank you.
Wireless Devices
Wireless device noises are rude in any language. Please take a
moment to turn off any of the following:
Cell Phones
Scheduler Devices
Pagers
Alarm Clocks
Pacemaker low-battery warning alarms
Anything else you are carrying on or about your
which may make noise during this
presentation.
person
About this Presentation
A "best practices" session is different
This is not a list of product features.
This is a practical 'field guide' of which ones to use, and why.
Focused on What and Why, pointers to resources for how.
Designed for re-use
These are not empty bullet points.
The details you need are in this text.
The Goal of this Presentation
Provide an overview of what you should be thinking about as an
administrator
Provide a trail map for finding out more, and implementing the ones you
find of value
Help you start thinking in terms of the ‘big picture’ rather than being
constantly swamped by the details
Agenda
Who am I to be telling you
anything?
The Scenario Setup
Server Stability Management
Security Management
Mail Management
Database Management
Client Software Management
End User Support
Who am I To Tell You Anything?
Andrew Pollack
President, Northern Collaborative Technologies
2003 IBM Lotus Beacon Award Winner
1999 Lotus Beacon Award Finalist
Administrator & Developer since version 2.0
Member of the Penumbra Group
Firefighter – Cumberland, Maine!
Lieutenant of Engine 1, Ladder 7, Heavy Rescue, RIT, Special Operations
In firefighting, just like Server Administration it's all in the planning
Why We're Here
To learn and grow as human beings
The question has changed, now it isn't "How Can we," it's "How Should We"
Also, I'm here because it makes the phone ring more
A Typical Environment
Three Offices
Southeast – The Home Office
Mid Sized, easy to get to, excellent
net connection
Southwest – A Production Facility
Mid Sized, easy to get to
The Internet
(Very Scary)
Northeast R&D
Small Office
Extranet
Domino
Server
Border Router
Salt Lake Router
Tampa Router
Terrible Airport Access
Heavy Ground Traffic
Weather & Power Issues
Westford Router
Expensive Travel Costs
Then there’s you
The new Domino
Administrator
Domino
Server
Domino
Cluster
Domino
Cluster
Server Load & Hardware Choices
Clustering vs. Giant Boxes
Benchmarks are just statistics, and we know how much we
should trust those.
Would you really put 12,000 users on one server? 20,000?
More?
Domino clusters do not shared any hardware or part of the
same operating system. They are fully redundant.
Balance the load across all the servers in the cluster, but make
sure that if one goes down, the others can handle the load
without crashing.
A performance drop is acceptable for a brief period in most shops.
Clusters Provide High Availability, Low Cost
The Internet
(Very Scary)
Westford Router
Domino
Server
Extranet
Domino
Server
Border Router
Salt Lake Router
Tampa Router
Domino
Cluster
Domino
Cluster
Domino Clustering is REALLY Easy
Put databases on both servers
Make sure they replicate, and have proper access
Select the servers in the directory
Click "Add to Cluster"
Considering Peak Loads
We think of number of users – don’t do that.
Think number of concurrent users.
If you run three shifts, and only one shift is active at a time, you may be
able to use smaller hardware.
Think total disk space.
Disk usage is critical on the server, even if it isn't in use it costs the server
resources to keep indexes and run checks.
In some customer sites, mailbox size dictates server count because of
drive space limitations and the cost of massive storage networks.
For more information about clustering
JMP102 An Introduction To All Things IBM Lotus Domino Clustering --
Gabriella Davis
Software Version Management
Operating System Choice
Which operating system is the best?
Avoid politics, religion, and operating system preference discussions at the
dinner table
Either choose an OS that your staff knows well, or send them to school
All operating systems need to be patched and updated. Keeping up with
these is required for stability
Make a choice that is not unique in your company
Test, Test, Test
Watch out for case sensitivity when moving off Win32
Debugging can be very difficult because the initial hit to a resource is case
sensitive, but once the object is in the cache, it may not be.
BP403 Best Practices: IBM Lotus Domino for Linux -- Daniel Nashed
Remote Server Administration
No matter what tool you use, always use encryption
Many tasks you might think you need remote control software
for, can be done with the Web Administration Tool and the Lotus
Domino Administration Client
Editing the NOTES.INI on the server
Starting and Stopping Windows Services
Use the Server Controller and Java Console
These can restart even crashed servers remotely
Start the server with "–jc"
Stat the console from the Notes program directory "jconsole.exe"
Remote Control Software
Make sure it is set to lock the console automatically if your
connection drops
Make sure it requires encryption for connections
Keep up with the vendor’s patches and updates for the server
side
Security patches could be critical
These ports are scanned constantly
ADMINP is your best friend
Properly configured, this will do a lot of the hardest and most
tedious work for you
Distribution of new databases to multiple servers
User move, add, or change requests
This becomes more and more important with each new version
of the IBM Lotus Domino server
Each server should have a replica of the "ADMIN4.NSF" from
the administration server
For more information
ID113 Maximize the Power of AdminP in IBM Lotus Domino -- Kathleen
McGivney, Susan Bulloch
Local Staff
Nothing is better than local staff
Before doing any kind of remote access work, compile a list of local contact
staff with phone numbers and availability
Have someone check the cdrom trays – you do not want to reboot to a
setup disk
Nothing is worse than local staff
Control access to the sever
More on this topic when we talk security
Monitoring and Event Handling
Use Events – Be the First to Know
Easy to set up
Know about problems before your phone rings
Fix problems before the boss calls you
Make sure to log them, so he knows what you do
Event notices make great justification tools for new servers!
For more information –
BP407 What are Your Servers Trying to Tell You Now: The (Even) Easier
Route to IBM Lotus Domino Reporting & Logging -- Gabriella Davis
Power-off Recycle Devices
When all else fails, sometimes you need to power cycle a
machine – from 3000 miles away
Inexpensive power modules can be commanded to recycle
power with a 5 second power down pause
Controlled through serial port
Include "watchdog" software
Many devices on the market
Some include remote shell access
Some include Web browser control
Here’s what I use
http://www.cpscom.com/gprod/ipn.htm
Developer Management
Sir, please step away from
that Designer Client.
Deployment Policies
These are a good thing, and you should have some.
Questions to answer with your deployment policies:
Who decides when a database has been tested enough?
Who will be called when a problem is reported?
Do you have a contact number for this developer?
How will you know when the database is no longer in use?
More Deployment Policy Questions
How big is the database expected to get?
What servers does it need to be on?
Is external replication required?
How volatile is the access control going to be?
What kinds of agent code will be running at the server?
Server side java agents? Agents that call COM objects?
File System Access? ODBC or Connector LSX Use?
API Calls?
Do Not Modify the Domino Directory
Nothing impacts performance more than changes to the Domino
Directory
There are two critical view indexes in the Domino Directory
$ServerAccess
$Users
If the indexer is busy doing other things in that database, these
updates will take longer
If these indexes are not up to date, authentication and access
rights may not be granted to users
Java Agents Must Be Tested at Full Scale
Multi-threading is so powerful, you can shoot yourself in both
feet at once
Very easy and common mistakes in Java agents can kill
production servers easily
Unlike LotusScript, when writing Java agents programmers must call
"recycle()" on every object you instantiate, or their parent document
In test, it is frequently possible to get away with simply recycling the
"session" object when the agent terminates
In production, this kills servers when the agent handles a large number of
documents in a loop, among other things
Yes, I know this from bitter experience
Restricted vs. Unrestricted Agents
Unrestricted agents can do to things outside the scope of the
agent itself
Access the operating system
Access files on the server – important ones
Reboot or shutdown the server
If someone needs to run an unrestricted agent, you need to
understand why
Security Management
The Five Pillars of Security
Physical Server Security
Operating System / File System Security
Lotus Domino Server Access
Certificates & Cross Certification
Public / Private Key Certification
Cross Certification
Server Access Settings
Database Access – The ACL
Document Access – Reader Names
Notes Client Side Security
Guard Your Certifier
Dealing with a compromised certifier
Assume Users have Designer
It's easy to get
Obscurity is not Security
Encrypt Workstation Data
Escrow ID Files
Preventing Workstation Copies
Third Party Tool: dotNSF Tools noCopy – www.dotNSF.com
Client to Server Communication Encryption
Browser Access Security
Obscurity is not security!
This is the #1 issue on Web sites
URL Hacking
NoteID Crawling
Common Word Crawling
/database.nsf/knownViewName/<insert word here>
SSL – Preventing Man in the Middle Attacks
Creating an SSL Key Ring
Obtaining an SSL Certificate
An authority unto yourself – Are you trusted?
Buying an SSL Certificate
Deploying an SSL Key Ring to Domino
Securing the Other Protocols
Understand your ports
If your server faces the internet, put a firewall in front of it
Many of the server tasks listen on a port, understand them or don’t load
them. Particularly, LDAP and SMTP can give away a lot of valuable
information if improperly configured
If you don’t need a protocol, shut it down
If nothing is listening on a port, that port is secure. Well, mostly.
Password Guessing isn’t Just Browsers!
User's "Internet" passwords are frequently less complex than
their Notes ID Passwords – Use the tools to enforce complexity
It is now very common for hackers to "Name Guess" via POP3,
SMTP, and even "Harvest" names from Web sites, e-mail
addresses, and open LDAP ports
Once a name is guessed or harvested, POP3 or other protocols
are used to guess passwords
With a name and password, spammers can use your server
using an authenticated username
Mail Management
This is probably why many of you
came here in the first place.
Notes Mail Routing
Servers on the same Notes Named Network
Should be able to find each other "by name" without connection documents –
with TCPIP, this would be DNS
Servers on the same "named" network route mail automatically; no connection
document is needed
This is a "least cost" indicator to Domino's routing cost matrix
Use this to your advantage
Set up your named networks to reflect your network's faster and slower links.
Put only servers that have excellent connectivity on the same "Named Network"
Connection Documents
Connection documents tell servers which are not on the same
"Notes Named Network" how to find each other
They're also used for replication, but we'll get to that later
Internet Mail Routing -- Turning off SMTP inside
the Network
If you turn off the SMTP Inbound Listener, local Windows clients
which have been infected with a virus, worm, Trojan horse, or
spy-ware application cannot send mail through your servers.
This also eliminates accidental or deliberate use of your internal servers for
spam routing.
Even if you require password access for SMTP mail sending, password
guessing is now quite common.
If you disable SMTP Outbound on your servers, it will force the
mail to route through your single gateway. In many cases this is
a more secure method and provides greater traffic control on
your network.
Using a Single Internet Mail Gateway
Server Documents (all but the server that will route smtp):
Set "SMTP Listener" to Disabled
Set "Routing Tasks" to "Mail Routing" – but not "SMTP Mail Routing"
Create a "Foreign SMTP Domain" Domain Document
Route *.* to "OurFakeName"
Create a Connection Document
Type: SMTP
Source Server: The domino server with smtp
Destination Server: MAKE UP a name
Destination Domain: "OurFakeName"
Routing Task: SMTP Mail Routing
This method means you don’t even need TCPIP as a protocol
on your other Domino servers, because the routing all happens
using Notes RPC protocols to the one server with SMTP
capability.
Single Internet Mail Gateway -- What Really happens?
All the servers where SMTP Mail Routing is not a task, look for
a route to send the mail.
These servers see that *.* goes to the domain "OurFakeName"
That's the SMTP Domain Document's Job
The router task on the servers see that one Domino server has
a connection to the "OurFakeName" domain so they route the
messages to that server
That's the connection document's job
The server which is SMTP Mail Routing Enabled receives the
mail in its INBOX and knows how to send SMTP mail directly, so
it does.
Standardizing on a Mail Template
Beware of Customized Templates
Prevents Update & Bug Fix
Look at the update lists in each point release and note how many related to
small fixes in the mail templates.
Serious Performance Issues
More views means more view indexing work for the server.
Limiting Design Access to Mail Files
People are most likely to make "quick" (untested) updates to the design of their
mail file, considering it their own problem if they cause a problem. These people
can take down your server.
If you want additional features, look for "Packaged" alternative
mail templates which are properly supported.
openNTF.org has a very popular one, for example.
Managing Mail File Size – SCOS
Single Copy Object Store has been a feature for many years.
It DRASTICALLY reduces disk usage by keeping one copy of
each file no matter how many different people have it in their
mail files.
It's significantly better than it was, and with "Transaction
Logging" and Domino clustering can be much more reliable than
ever before.
It's still a single point of failure – if you do have a problem,
everyone is affected by the problem.
Managing Mail File Size (continued)
Take Advantage of Archiving
Archiving can be easily set up and managed through policies
Put Archives on different server, they're less frequently accessed and have
different load characteristics
Impose Realistic Limits with Quotas
Managing Unwanted Mail
Don't be a Relay
In the "Configuration" document for your server – not the Server document,
on the "Router/SMTP:Restrictions And Controls:SMTP Inbound Controls"
Tab
Deny messages from the following internet hosts to be sent to external
internet domains:(* means all) – Set to "*"
This is the Default on all recent Domino versions
Hold Undeliverable Mail
Don't send bounce messages – Frequently, the mail never even originated
on your site and you're only adding to the problem
Fighting unwanted mail is much more complex than this
BP405 Controlling Spam Mail In Your Organization
BOF509 Keeping Up with the Spammers with IBM Lotus Notes and
Domino
Don't Give Away Address Information
Verify that local domain recipients exist in the Domino Directory:
Pros:
Stops inbound SMTP messages send with dictionary style drops and name
guesses from clogging your router
Can make your site less attractive to spammers who get credit for "delivered"
messages – accepted by your server
Cons:
Makes it easy for spammers to test for valid names on your server
Consider using this if you have another tool that can detect
multiple failed attempts from the same source and ban those
sources at the firewall.
Other Message Filtering Considerations
Using Black Lists (aka Real-time Black Hole or RBL)
Many "black lists" exist that you can use
(e.g. bl.spamcop.net; sbl-xbl.spamhaus.org)
Not 100% accurate
Read the list’s website to understand their criteria for listing
Using White Lists (aka "Known Good" addresses)
Most mail you get, is from people you've communicated with already
New to version 7 of Lotus Domino, but part of several 3rd party tools for
some time
Mail Filtering Tools
Third Party Tools
User-Interactive Products like spamJam can be excellent because each
user decides individually what's wanted and what's not
Appliance Solutions can be inexpensive and effective, but less user-
specific
My Recommendations
spamJam – because users really like being able to interact with it
Barracuda – for simplicity and price, this device works very well
ASSP – Open source proxy, good but scale is uncertain
Signed Mail
Signed mail to Notes users
Your Public Key
Use "Files-Security-User Security" to get it or copy it from your Domino Directory
person document
Signed Mail to Internet users
X.509 Certificates – The modern standard for authentication
Self Certifying –
If you create your own certificate authority, everyone will always have to
decide accept it as trusted
Excellent alternative for internal company use
Buying Certificates or Certification Rights
Free Certification Network
Importing Your X.509 Certificate
If you obtain a personal x.509 certificate, you can import it into
your person document in the Domino Directory
Open your Person Document
Select "Actions Import Internet Certificates"
Once this is done, you can "sign" mail to be sent to users with
Internet addresses
Verifying Signed Mail
From Notes Users
The Lotus Notes Public Key
You must have their public key in your address book
Verifying Signed Mail from Internet Users
Accepting a Cross Certificate
Do this the first time you get signed mail from a user
Call the user, make sure its them sending the message
Adding a Sender's Public Key to Your Personal
Address Book
While viewing, use "Tools – Add sender to address book"
Advanced tab, check to add "x.509 certificate…"
Mail Encryption
The Recipient’s Public Key is required
The Public Key is used to create a one-way cipher that can only
be read with the private key – and only the user has the private
key, it's in their Notes ID file (or other file if a non-Notes user)
Obtaining a Recipient's Public Key
Notes Mail users in your domain already have it in their
"Person" document in the Domino Directory.
Notes Mail users in other domains must send it to you. They can
copy it from their record in their Domino directory, or use the
options in "Files – Security – User Security" to get it.
Users can also simply send you a "Signed" document, and you
can "Cross Certify" them when you receive the mail. (You'll be
prompted.)
Adding a Sender's Public Key to Your Personal
Address Book
While viewing, use "Tools – Add sender to address book"
Advanced tab, check to add "x.509 certificate…"
Database Management
Deployment Policies
Limit Designer & Manager Access
On the fly changes cause most problems
Use Database Access Groups to Delegate Control
Create Groups that a database owner can manage
Example: "SalesTools.NSF Editors"
Set the database owner to be the owner of that group
The Connection Document for Replication
A connection document is required for replication even on the
same "Notes Named Network"
A common error on the connection document is not changing the
schedule to work around the clock. Default is 8am-10pm.
Keep in mind that following replication, the indexer may be very
busy. Consider having replication occur prior to the start of the
normal business day.
Database Deployment Policies
Track Database Usage & Ownership
Every Database must have an Owner
Every Database must have a Review Date
Remove Outdated or Unused Databases
Even unused databases can load the server
Old data represents a security, accuracy, and legal risk
Replication Topologies
Avoid "Everyone Replicates with Everyone"
Map Network Choke Points
The Internet
(Very Scary)
Westford Router
Domino
Server
Extranet
Domino
Server
Border Router
Salt Lake Router
Tampa Router
Domino
Cluster
Domino
Cluster
Creating a Redundant Hub & Spoke
Two distinct local area networks or well
connected individual networks
v
One high bandwidth connection between the
two clustered hubs
Reduces traffic across the expensive long haul
network
Client Software Management
Common Policy Settings
Use policies to define ECL (Execution Control List) settings
Use policies to make sure users have the right replicas on the
local workstations
Policies in version 7 can be much more rigidly enforced
Client Version Update Rollout
Excellent for ROI – No more touching the desktop
Reduces support due to version/template incompatibility
BP404 Best Practices in IBM Lotus Notes Client Deployment -Steve Sterka, David Via
ID117 IBM Lotus Notes Deployment Made Easy -- Jeff Mitchell,
John Paganetti
Handling User Support
Delegating Admin Roles Safely?
Version 6.x added granularity to "Administrator" access
Allows you to delegate specific areas of responsibility without
giving complete control to junior administrators.
Using the administrator task, you can allow area managers to
register users without giving them a certifier.
Admin Roles in Version 6.x
Full Access administrators
Able to leap tall ACLs; impervious to Reader-Names
Administrators
Use all the power of the administrator tool, but subject to database and
document controls
Database Administrators
Manage databases, but not the server itself
Full Remote Console Administrators / View-only Administrators
System Administrators
No database controls, but plenty of server setup access
Restricted System Administrators
Restricted System Commands
Limit Use of Full Access Administration
Full Access Administration should only be used rarely, when a
need to override ACL or ReaderNames is required.
Grant this only to specific ID files. Make the administrator switch
to this ID file when needed.
Create an "Event" notification to notify management any time
this level of access is granted.
Use encryption on database you don’t want full access
administrators to read.
In summary
It's no longer a question of whether or not something can be
done, it's a question of which is the best way to do it and why.
This presentation serves as a guideline, not a bible.
This has been a high to medium high level look at the features
you should be using, with pointers to where to find more
detailed information.
Thank you for playing!
We’re all Lotus professionals here, please ask your questions
so others can here the answers. You may also contact me
directly if you like.
Please fill out your evaluations
The latest copy of this presentation will also be available at
my website: http://www.thenorth.com
For those playing the home game, direct questions & comments to:
Andrew Pollack
[email protected]
http://www.thenorth.com