Transcript Slide 1

BP 401 - Admin Zero to Hero
in 60 Minutes
The question is no longer, "How can we?"
The question now is, "How should we?"
Andrew Pollack, President
Northern Collaborative Technologies
Language Note
 I realize that for some of you, English is not your primary
language, and for others, my accent is not the same as yours.
 If you are having trouble understanding me during this talk,
please raise your hand and I will try to slow down and speak
more clearly.
 Thank you.
Wireless Devices
 Wireless device noises are rude in any language. Please take a
moment to turn off any of the following:
 Cell Phones
 Scheduler Devices
 Pagers
 Alarm Clocks
 Pacemaker low-battery warning alarms
 Anything else you are carrying on or about your
which may make noise during this
presentation.
person
About this Presentation
 A "best practices" session is different
 This is not a list of product features.
 This is a practical 'field guide' of which ones to use, and why.
 Focused on What and Why, pointers to resources for how.
 Designed for re-use
 These are not empty bullet points.
 The details you need are in this text.
 The Goal of this Presentation
 Provide an overview of what you should be thinking about as an
administrator
 Provide a trail map for finding out more, and implementing the ones you
find of value
 Help you start thinking in terms of the ‘big picture’ rather than being
constantly swamped by the details
Agenda
 Who am I to be telling you
anything?
 The Scenario Setup
 Server Stability Management
 Security Management
 Mail Management
 Database Management
 Client Software Management
 End User Support
Who am I To Tell You Anything?
 Andrew Pollack
 President, Northern Collaborative Technologies
 2003 IBM Lotus Beacon Award Winner
 1999 Lotus Beacon Award Finalist
 Administrator & Developer since version 2.0
 Member of the Penumbra Group
 Firefighter – Cumberland, Maine!
 Lieutenant of Engine 1, Ladder 7, Heavy Rescue, RIT, Special Operations
 In firefighting, just like Server Administration it's all in the planning
 Why We're Here
 To learn and grow as human beings
 The question has changed, now it isn't "How Can we," it's "How Should We"
 Also, I'm here because it makes the phone ring more
A Typical Environment
 Three Offices
Southeast – The Home Office
Mid Sized, easy to get to, excellent
net connection
Southwest – A Production Facility
Mid Sized, easy to get to
The Internet
(Very Scary)
Northeast R&D
Small Office
Extranet
Domino
Server
Border Router
Salt Lake Router
Tampa Router
Terrible Airport Access
Heavy Ground Traffic
Weather & Power Issues
Westford Router
Expensive Travel Costs
 Then there’s you
The new Domino
Administrator
Domino
Server
Domino
Cluster
Domino
Cluster
Server Load & Hardware Choices
Clustering vs. Giant Boxes
 Benchmarks are just statistics, and we know how much we
should trust those.
 Would you really put 12,000 users on one server? 20,000?
More?
 Domino clusters do not shared any hardware or part of the
same operating system. They are fully redundant.
 Balance the load across all the servers in the cluster, but make
sure that if one goes down, the others can handle the load
without crashing.
 A performance drop is acceptable for a brief period in most shops.
Clusters Provide High Availability, Low Cost
The Internet
(Very Scary)
Westford Router
Domino
Server
Extranet
Domino
Server
Border Router
Salt Lake Router
Tampa Router
Domino
Cluster
Domino
Cluster
Domino Clustering is REALLY Easy
 Put databases on both servers
 Make sure they replicate, and have proper access
 Select the servers in the directory
 Click "Add to Cluster"
Considering Peak Loads
 We think of number of users – don’t do that.
 Think number of concurrent users.
 If you run three shifts, and only one shift is active at a time, you may be
able to use smaller hardware.
 Think total disk space.
 Disk usage is critical on the server, even if it isn't in use it costs the server
resources to keep indexes and run checks.
 In some customer sites, mailbox size dictates server count because of
drive space limitations and the cost of massive storage networks.
 For more information about clustering
 JMP102 An Introduction To All Things IBM Lotus Domino Clustering --
Gabriella Davis
Software Version Management
Operating System Choice
 Which operating system is the best?
 Avoid politics, religion, and operating system preference discussions at the
dinner table
 Either choose an OS that your staff knows well, or send them to school
 All operating systems need to be patched and updated. Keeping up with
these is required for stability
 Make a choice that is not unique in your company
 Test, Test, Test
 Watch out for case sensitivity when moving off Win32
 Debugging can be very difficult because the initial hit to a resource is case
sensitive, but once the object is in the cache, it may not be.
 BP403 Best Practices: IBM Lotus Domino for Linux -- Daniel Nashed
Remote Server Administration
 No matter what tool you use, always use encryption
 Many tasks you might think you need remote control software
for, can be done with the Web Administration Tool and the Lotus
Domino Administration Client
 Editing the NOTES.INI on the server
 Starting and Stopping Windows Services
 Use the Server Controller and Java Console
 These can restart even crashed servers remotely
 Start the server with "–jc"
 Stat the console from the Notes program directory "jconsole.exe"
Remote Control Software
 Make sure it is set to lock the console automatically if your
connection drops
 Make sure it requires encryption for connections
 Keep up with the vendor’s patches and updates for the server
side
 Security patches could be critical
 These ports are scanned constantly
ADMINP is your best friend
 Properly configured, this will do a lot of the hardest and most
tedious work for you
 Distribution of new databases to multiple servers
 User move, add, or change requests
 This becomes more and more important with each new version
of the IBM Lotus Domino server
 Each server should have a replica of the "ADMIN4.NSF" from
the administration server
 For more information
 ID113 Maximize the Power of AdminP in IBM Lotus Domino -- Kathleen
McGivney, Susan Bulloch
Local Staff
 Nothing is better than local staff
 Before doing any kind of remote access work, compile a list of local contact
staff with phone numbers and availability
 Have someone check the cdrom trays – you do not want to reboot to a
setup disk
 Nothing is worse than local staff
 Control access to the sever
 More on this topic when we talk security
Monitoring and Event Handling
 Use Events – Be the First to Know
 Easy to set up
 Know about problems before your phone rings
 Fix problems before the boss calls you
 Make sure to log them, so he knows what you do
 Event notices make great justification tools for new servers!
 For more information –
 BP407 What are Your Servers Trying to Tell You Now: The (Even) Easier
Route to IBM Lotus Domino Reporting & Logging -- Gabriella Davis
Power-off Recycle Devices
 When all else fails, sometimes you need to power cycle a
machine – from 3000 miles away
 Inexpensive power modules can be commanded to recycle
power with a 5 second power down pause
 Controlled through serial port
 Include "watchdog" software
 Many devices on the market
 Some include remote shell access
 Some include Web browser control
Here’s what I use
http://www.cpscom.com/gprod/ipn.htm
Developer Management
Sir, please step away from
that Designer Client.
Deployment Policies
 These are a good thing, and you should have some.
 Questions to answer with your deployment policies:
 Who decides when a database has been tested enough?
 Who will be called when a problem is reported?
 Do you have a contact number for this developer?
 How will you know when the database is no longer in use?
More Deployment Policy Questions
 How big is the database expected to get?
 What servers does it need to be on?
 Is external replication required?
 How volatile is the access control going to be?
 What kinds of agent code will be running at the server?
 Server side java agents? Agents that call COM objects?
 File System Access? ODBC or Connector LSX Use?
 API Calls?
Do Not Modify the Domino Directory
 Nothing impacts performance more than changes to the Domino
Directory
 There are two critical view indexes in the Domino Directory
 $ServerAccess
 $Users
 If the indexer is busy doing other things in that database, these
updates will take longer
 If these indexes are not up to date, authentication and access
rights may not be granted to users
Java Agents Must Be Tested at Full Scale
 Multi-threading is so powerful, you can shoot yourself in both
feet at once
 Very easy and common mistakes in Java agents can kill
production servers easily
 Unlike LotusScript, when writing Java agents programmers must call
"recycle()" on every object you instantiate, or their parent document
 In test, it is frequently possible to get away with simply recycling the
"session" object when the agent terminates
 In production, this kills servers when the agent handles a large number of
documents in a loop, among other things
 Yes, I know this from bitter experience
Restricted vs. Unrestricted Agents
 Unrestricted agents can do to things outside the scope of the
agent itself
 Access the operating system
 Access files on the server – important ones
 Reboot or shutdown the server
 If someone needs to run an unrestricted agent, you need to
understand why
Security Management
The Five Pillars of Security
 Physical Server Security
 Operating System / File System Security
 Lotus Domino Server Access
 Certificates & Cross Certification
 Public / Private Key Certification
 Cross Certification
 Server Access Settings
 Database Access – The ACL
 Document Access – Reader Names
Notes Client Side Security
 Guard Your Certifier
 Dealing with a compromised certifier
 Assume Users have Designer
 It's easy to get
 Obscurity is not Security
 Encrypt Workstation Data
 Escrow ID Files
 Preventing Workstation Copies
 Third Party Tool: dotNSF Tools noCopy – www.dotNSF.com
 Client to Server Communication Encryption
Browser Access Security
 Obscurity is not security!
 This is the #1 issue on Web sites
 URL Hacking
 NoteID Crawling
 Common Word Crawling
/database.nsf/knownViewName/<insert word here>
 SSL – Preventing Man in the Middle Attacks
 Creating an SSL Key Ring
 Obtaining an SSL Certificate
 An authority unto yourself – Are you trusted?
 Buying an SSL Certificate
 Deploying an SSL Key Ring to Domino
Securing the Other Protocols
 Understand your ports
 If your server faces the internet, put a firewall in front of it
 Many of the server tasks listen on a port, understand them or don’t load
them. Particularly, LDAP and SMTP can give away a lot of valuable
information if improperly configured
 If you don’t need a protocol, shut it down
 If nothing is listening on a port, that port is secure. Well, mostly.
Password Guessing isn’t Just Browsers!
 User's "Internet" passwords are frequently less complex than
their Notes ID Passwords – Use the tools to enforce complexity
 It is now very common for hackers to "Name Guess" via POP3,
SMTP, and even "Harvest" names from Web sites, e-mail
addresses, and open LDAP ports
 Once a name is guessed or harvested, POP3 or other protocols
are used to guess passwords
 With a name and password, spammers can use your server
using an authenticated username
Mail Management
This is probably why many of you
came here in the first place.
Notes Mail Routing
 Servers on the same Notes Named Network
 Should be able to find each other "by name" without connection documents –
with TCPIP, this would be DNS
 Servers on the same "named" network route mail automatically; no connection
document is needed
 This is a "least cost" indicator to Domino's routing cost matrix
 Use this to your advantage
 Set up your named networks to reflect your network's faster and slower links.
Put only servers that have excellent connectivity on the same "Named Network"
Connection Documents
 Connection documents tell servers which are not on the same
"Notes Named Network" how to find each other
 They're also used for replication, but we'll get to that later
Internet Mail Routing -- Turning off SMTP inside
the Network
 If you turn off the SMTP Inbound Listener, local Windows clients
which have been infected with a virus, worm, Trojan horse, or
spy-ware application cannot send mail through your servers.
 This also eliminates accidental or deliberate use of your internal servers for
spam routing.
 Even if you require password access for SMTP mail sending, password
guessing is now quite common.
 If you disable SMTP Outbound on your servers, it will force the
mail to route through your single gateway. In many cases this is
a more secure method and provides greater traffic control on
your network.
Using a Single Internet Mail Gateway
 Server Documents (all but the server that will route smtp):
 Set "SMTP Listener" to Disabled
 Set "Routing Tasks" to "Mail Routing" – but not "SMTP Mail Routing"
 Create a "Foreign SMTP Domain" Domain Document
 Route *.* to "OurFakeName"
 Create a Connection Document
 Type: SMTP
 Source Server: The domino server with smtp
 Destination Server: MAKE UP a name
 Destination Domain: "OurFakeName"
 Routing Task: SMTP Mail Routing
 This method means you don’t even need TCPIP as a protocol
on your other Domino servers, because the routing all happens
using Notes RPC protocols to the one server with SMTP
capability.
Single Internet Mail Gateway -- What Really happens?
 All the servers where SMTP Mail Routing is not a task, look for
a route to send the mail.
 These servers see that *.* goes to the domain "OurFakeName"
 That's the SMTP Domain Document's Job
 The router task on the servers see that one Domino server has
a connection to the "OurFakeName" domain so they route the
messages to that server
 That's the connection document's job
 The server which is SMTP Mail Routing Enabled receives the
mail in its INBOX and knows how to send SMTP mail directly, so
it does.
Standardizing on a Mail Template
 Beware of Customized Templates
 Prevents Update & Bug Fix
 Look at the update lists in each point release and note how many related to
small fixes in the mail templates.
 Serious Performance Issues
 More views means more view indexing work for the server.
 Limiting Design Access to Mail Files
 People are most likely to make "quick" (untested) updates to the design of their
mail file, considering it their own problem if they cause a problem. These people
can take down your server.
 If you want additional features, look for "Packaged" alternative
mail templates which are properly supported.
 openNTF.org has a very popular one, for example.
Managing Mail File Size – SCOS
 Single Copy Object Store has been a feature for many years.
 It DRASTICALLY reduces disk usage by keeping one copy of
each file no matter how many different people have it in their
mail files.
 It's significantly better than it was, and with "Transaction
Logging" and Domino clustering can be much more reliable than
ever before.
 It's still a single point of failure – if you do have a problem,
everyone is affected by the problem.
Managing Mail File Size (continued)
 Take Advantage of Archiving
 Archiving can be easily set up and managed through policies
 Put Archives on different server, they're less frequently accessed and have
different load characteristics
 Impose Realistic Limits with Quotas
Managing Unwanted Mail
 Don't be a Relay
 In the "Configuration" document for your server – not the Server document,
on the "Router/SMTP:Restrictions And Controls:SMTP Inbound Controls"
Tab
 Deny messages from the following internet hosts to be sent to external
internet domains:(* means all) – Set to "*"
 This is the Default on all recent Domino versions
 Hold Undeliverable Mail
 Don't send bounce messages – Frequently, the mail never even originated
on your site and you're only adding to the problem
 Fighting unwanted mail is much more complex than this
 BP405 Controlling Spam Mail In Your Organization
 BOF509 Keeping Up with the Spammers with IBM Lotus Notes and
Domino
Don't Give Away Address Information
 Verify that local domain recipients exist in the Domino Directory:
 Pros:
 Stops inbound SMTP messages send with dictionary style drops and name
guesses from clogging your router
 Can make your site less attractive to spammers who get credit for "delivered"
messages – accepted by your server
 Cons:
 Makes it easy for spammers to test for valid names on your server
 Consider using this if you have another tool that can detect
multiple failed attempts from the same source and ban those
sources at the firewall.
Other Message Filtering Considerations
 Using Black Lists (aka Real-time Black Hole or RBL)
 Many "black lists" exist that you can use
 (e.g. bl.spamcop.net; sbl-xbl.spamhaus.org)
 Not 100% accurate
 Read the list’s website to understand their criteria for listing
 Using White Lists (aka "Known Good" addresses)
 Most mail you get, is from people you've communicated with already
 New to version 7 of Lotus Domino, but part of several 3rd party tools for
some time
Mail Filtering Tools
 Third Party Tools
 User-Interactive Products like spamJam can be excellent because each
user decides individually what's wanted and what's not
 Appliance Solutions can be inexpensive and effective, but less user-
specific
 My Recommendations
 spamJam – because users really like being able to interact with it
 Barracuda – for simplicity and price, this device works very well
 ASSP – Open source proxy, good but scale is uncertain
Signed Mail
 Signed mail to Notes users
 Your Public Key
 Use "Files-Security-User Security" to get it or copy it from your Domino Directory
person document
 Signed Mail to Internet users
 X.509 Certificates – The modern standard for authentication
 Self Certifying –
If you create your own certificate authority, everyone will always have to
decide accept it as trusted
Excellent alternative for internal company use
 Buying Certificates or Certification Rights
 Free Certification Network
Importing Your X.509 Certificate
 If you obtain a personal x.509 certificate, you can import it into
your person document in the Domino Directory
 Open your Person Document
 Select "Actions Import Internet Certificates"
 Once this is done, you can "sign" mail to be sent to users with
Internet addresses
Verifying Signed Mail
 From Notes Users
 The Lotus Notes Public Key
 You must have their public key in your address book
 Verifying Signed Mail from Internet Users
 Accepting a Cross Certificate
 Do this the first time you get signed mail from a user
 Call the user, make sure its them sending the message
Adding a Sender's Public Key to Your Personal
Address Book
 While viewing, use "Tools – Add sender to address book"
 Advanced tab, check to add "x.509 certificate…"
Mail Encryption
 The Recipient’s Public Key is required
 The Public Key is used to create a one-way cipher that can only
be read with the private key – and only the user has the private
key, it's in their Notes ID file (or other file if a non-Notes user)
Obtaining a Recipient's Public Key
 Notes Mail users in your domain already have it in their
"Person" document in the Domino Directory.
 Notes Mail users in other domains must send it to you. They can
copy it from their record in their Domino directory, or use the
options in "Files – Security – User Security" to get it.
 Users can also simply send you a "Signed" document, and you
can "Cross Certify" them when you receive the mail. (You'll be
prompted.)
Adding a Sender's Public Key to Your Personal
Address Book
 While viewing, use "Tools – Add sender to address book"
 Advanced tab, check to add "x.509 certificate…"
Database Management
Deployment Policies
 Limit Designer & Manager Access
 On the fly changes cause most problems
 Use Database Access Groups to Delegate Control
 Create Groups that a database owner can manage
 Example: "SalesTools.NSF Editors"
 Set the database owner to be the owner of that group
The Connection Document for Replication
 A connection document is required for replication even on the
same "Notes Named Network"
 A common error on the connection document is not changing the
schedule to work around the clock. Default is 8am-10pm.
 Keep in mind that following replication, the indexer may be very
busy. Consider having replication occur prior to the start of the
normal business day.
Database Deployment Policies
 Track Database Usage & Ownership
 Every Database must have an Owner
 Every Database must have a Review Date
 Remove Outdated or Unused Databases
 Even unused databases can load the server
 Old data represents a security, accuracy, and legal risk
Replication Topologies
 Avoid "Everyone Replicates with Everyone"
 Map Network Choke Points
The Internet
(Very Scary)
Westford Router
Domino
Server
Extranet
Domino
Server
Border Router
Salt Lake Router
Tampa Router
Domino
Cluster
Domino
Cluster
Creating a Redundant Hub & Spoke
 Two distinct local area networks or well
connected individual networks
v
 One high bandwidth connection between the
two clustered hubs
 Reduces traffic across the expensive long haul
network
Client Software Management
Common Policy Settings
 Use policies to define ECL (Execution Control List) settings
 Use policies to make sure users have the right replicas on the
local workstations
 Policies in version 7 can be much more rigidly enforced
Client Version Update Rollout
 Excellent for ROI – No more touching the desktop
 Reduces support due to version/template incompatibility
 BP404 Best Practices in IBM Lotus Notes Client Deployment -Steve Sterka, David Via
 ID117 IBM Lotus Notes Deployment Made Easy -- Jeff Mitchell,
John Paganetti
Handling User Support
Delegating Admin Roles Safely?
 Version 6.x added granularity to "Administrator" access
 Allows you to delegate specific areas of responsibility without
giving complete control to junior administrators.
 Using the administrator task, you can allow area managers to
register users without giving them a certifier.
Admin Roles in Version 6.x
 Full Access administrators
 Able to leap tall ACLs; impervious to Reader-Names
 Administrators
 Use all the power of the administrator tool, but subject to database and
document controls
 Database Administrators
 Manage databases, but not the server itself
 Full Remote Console Administrators / View-only Administrators
 System Administrators
 No database controls, but plenty of server setup access
 Restricted System Administrators
 Restricted System Commands
Limit Use of Full Access Administration
 Full Access Administration should only be used rarely, when a
need to override ACL or ReaderNames is required.
 Grant this only to specific ID files. Make the administrator switch
to this ID file when needed.
 Create an "Event" notification to notify management any time
this level of access is granted.
 Use encryption on database you don’t want full access
administrators to read.
In summary
 It's no longer a question of whether or not something can be
done, it's a question of which is the best way to do it and why.
 This presentation serves as a guideline, not a bible.
 This has been a high to medium high level look at the features
you should be using, with pointers to where to find more
detailed information.
Thank you for playing!
We’re all Lotus professionals here, please ask your questions
so others can here the answers. You may also contact me
directly if you like.
Please fill out your evaluations
The latest copy of this presentation will also be available at
my website: http://www.thenorth.com
For those playing the home game, direct questions & comments to:
Andrew Pollack
[email protected]
http://www.thenorth.com