Non Melanoma Skin Cancers

Transcript Non Melanoma Skin Cancers

Infrastructure Reliability
Common Systems Group
Experience @ UW Madison
Roger Hanson
5 Jan 2005
University of Wisconsin-Madison
2
Overview
• Basics – Redundant Hardware
• Test Environments
• Change Management
• Version control
• Testing processes
• Collaboration
• Service Management
3
Background
• MyUW
• WiscMail
Portal
campus Mail Service
• In Production in 2001
• New Complex Environments
– Layer 4 Switching
– Directory Enabled Systems
– ES Storage Area Networks
4
• Campus Portal
• Access to over 130 modules
• 1.8M Logins in Sept. 04
• 49K+ Unique Logins in Sept.
5
Hardware - Portal
Layer 4 Switch 1
Layer 4 Switch 2
Dell
Dell
Apache Web Server 1
Apache Web Server 2
WLS Plugin
WLS Plugin
Sun E280
WLS Node 1
EFS
My UW
Madison
WLS Administrative Server
Sun E280
WLS Node 2
EFS
My UW
Madison
Oracle
Application
Database
Sun E450
6
• Campus Mail system
• Nearly 90K accounts
• Daily Message Peak over 3M messages
• Service objective
– Never down
– Message delivery in less than 2 minutes
7
Hardware - Email
ry
l i ve
De s
r
d
o
r
f
tes orwa
bu
ttri x & F
A
o
&
th Mailb
Au
Stats
144.92.197.252 Priv
144.92.197.253 Priv
Private Network Mail
Delivery
Stats
WebMail SMTP
Steve Kohlbeck
12/16/04
Internet
ESS
150GB
Quota Stats
144.92.197.203 Priv
sunloan
--144.92.197.155
SMTP99
WiscMail SMTP
1. SMTP Inbound (MX)
2. SMTP Outbound
3. SMTP Inbound (POP/IMAP Client)
4. SMTP Inbound (WebMail)
5. SMTP Post Filtering Loop
6. SMTP Post Lists AV Scan
WiscMail Clients
POP & IMAP
Mhub/Lists/
ClassLists
AV Scanning
admin.wiscmail.wisc.edu
144.92.104.153
filters.wiscmail.wisc.edu
144.92.197.154
AVGATE/oceanus
144.92.104.17
LDAP & SMTP/AV
Lists
lists.services.wisc.edu
To mhub
SMTP/SMTP_AUTH from
Mail Clients
SPAM-6
stuka
144.92.197.167
Internet
Public Side Layer 4 Switch
smtp.wiscmail.wisc.edu - 144.92.197.138
smtpauth.wiscmail.wisc.edu - 144.92.197.134
SMTP Outbound
SPAM-5
hellcat
144.92.197.168
144.92.197.214 Priv
144.92.197.213 Priv
144.92.197.219 Priv
144.92.197.218 Priv
144.92.197.206 Priv
144.92.197.205 Priv
dasher
donner
blitzen
hermes
heimdall
----------144.92.197.145/xxx
144.92.197.184/xxx
144.92.197.183/142
144.92.197.190/141
144.92.197.159/140
SMTP1/SMTPAUTHx SMTP2/SMTPAUTHx SMTP3/SMTPAUTH3 SMTP4/SMTPAUTH2 SMTP5/SMTPAUTH1
SMTP Inbound (MX)
SPAM-4
corsair
144.92.197.166
Layer 4 Switch
spam.services.wisc.edu
128.104.1.199
SPAM-3
zero
128.104.1.227
Layer 4 Switch
newspam.services
.wisc.edu
144.92.197.133
SPAM-2
spitfire
128.104.1.226
144.92.197.146
MAILST4
--dancer
SMTP/AV Cluster
SPAM/Filtering Cluster
SPAM-1
mustang
128.104.1.225
ESS
450GB
144.92.197.154
MAILST3
--sneezy
Internet
Departmental
Mail & HAN
Auth for Wisc
144.92.197.251 Priv
144.92.197.153
MAILST2
--sleepy
ESS
600GB
WebMail SMTP
ESS
900GB
Store Cluster
144.92.197.152
MAILST1
---happy
ESS
to
SMTP Inbound
(MX)
144.92.197.222 Priv
ESS
144.92.197.221 Priv
144.92.197.202
LDAP4
144.92.197.155
WISCMAIL
bashful
LDAP, SMTP/AV
iDA & Mail
Ac
Stats
144.92.197.201
LDAP3
+
WM
on
144.92.197.237
144.92.197.148
MMP2
---vixen
144.92.197.216
LDAP2
aerate
144.92.197.157
WiscNet Clients
POP, IMAP, Web
ESS
150GB
mt
Mg
nt
u
o
c
To Ldap
Auth & Attributes
144.92,197.147
MMP1
---prancer
144.92.197.215
LDAP1
liquefy
144.92.197.156
ESS
r Login
Auth fo utes
& Attrib
Wiscmail.wisc.edu
Layer 4 Switch
144.92.197.133
SMTP_ Auth
admin.wiscmail.wisc.edu
144.92.197.163
filters.wiscmail.wisc.edu
144.92.197.165
Multiplexor Cluster
Mail SMTP &
WiscMail Plus Admin
Spam/Filtering Admin
Stats Admin
Private Side
Layer 4 Switch
ldap.doit.wisc.edu
144.92.197.196
WiscMail Clients
POP, IMAP, Web
ESS
LDAP Cluster
WiscMail Service Design
Spring 2004
kodos
144.92.104.60
kang
144.92.104.61
8
Basics – Redundant Hardware
• Clustered Server Environment
• Spares (Hot/Warm/Cold)
• Automated Load Balancing
• Automated fail over
9
Test Environments
• Test Cycle
– Test
– Development
– QA
– Production
• QA (also called Integrated
Test Environment)
10
Change Management
• Use of Change Information System
– Tracking
– Notification
• Use of Code Migration Request process
– Files promoted
– Configuration steps
– Test process
– Backout plans
11
Version Control
• Use CVS
– http://www.gnu.org/software/cvs/
– Develop in private or shared environments
– Code is published into repository
– Code is then copied to environment (dev, test,
qa, and prod)
12
Testing Process
• Unit testing
• Integrated Testing (QA)
• Log analysis from testing
• Written test plans
• Load Tests
• Testing tools (Empirix)
• System Monitoring (Wiley Introscope)
13
Collaboration
• Wiki
• Document Repository/Sharing
• Email Lists
• IM
• E-mail
14
Service Management
• Major direction at UW to
improve reliability
• CIO asking for 5 9s on key systems
• Consulting assistance
• Manage the service not the servers
• Adopt customer’s perspective
15
Service Management
• Models
– Information Technology Library
– Based on British Telecom agency processes
– Service Support processes
• Incident management
• Problem management
• Change management
• Release management
• Configuration management
16
Service Management
• Models
– Microsoft Operations Framework
• Combines ITIL processes with recommendations
for technical processes
• http://www.microsoft.com/mof
17
Next steps
• Define service level objectives for
key services
• Determine how to measure
service reliability
• Engage Data Center staff
18
Observations
• Infrastructure complexity
– Teams of specialists
• Funding for environments
• Staffing
• Process costs
19
Questions
Roger Hanson
Internet Infrastructure Applications
[email protected]
20