Transcript Document 7138906

AIS Infrastructure and Operations
Overview for Kevin Morooney
May 14, 2007
AIS Infrastructure and Operations Functional Overview










Operational support for central administrative systems
Production control
Network management and planning
AIS LAN management
Enterprise Server hardware, systems software, and middleware
MidTier Server hardware, systems software, and middleware
Database support
Software integration
AIS disaster recovery planning
Facilities planning and management
AIS Infrastructure and Operations Organizational Overview
65 Staff
Computer Operations & Facilities – Clyde LeFevre, Director
Network Infrastructure & Desktop Support – Todd Litzinger, Manager
Enterprise Infrastructure – Mike Kauffman, Manager
- Bill Cook, Manager of Enterprise Database Admin
- Phil Hawkins, Team Lead for Enterprise Services Integration
 MidTier Infrastructure – Pete Dawson, Manager
- Scott Neidigh, Team Lead for MidTier Server Mgmt
- Alex Pollock, Team Lead for MidTier Database Admin




AIS Infrastructure and Operations Challenges
 Broad customer base
 Managing competing priorities
 Communicating effectively with customers/internal staff
 Collaboration outside AIS
AIS Infrastructure and Operations Opportunities
 Work with new & exciting technologies
 Provide staff growth in conjunction with changing technologies
 Build on our partnerships with limited number of strategic vendors
 Investigate new communications/collaboration techniques
 Improve collaboration with other Penn State IT units
 Exploit strengths of ITS
Overview of AIS Computer Operations and
Facilities
Clyde LeFevre, Director
AIS Computer Operations and Facilities
Functional Area
Staff
Location
ES Operations (24x7)
Production Control
10
3
1
2
3
1
1
1
Computer Building
Shields Building
Computer Building
Shields Building
Shields Building
Shields Building
Rider Building
Shields Building
Scheduling
MT Operations (5x2)
CIDR Data Steward Asst
Data Entry
AIS Disaster Recovery
Total
22
Five Year Batch Workload Trend
Fiscal Year Ending
User Initiative
Aidaes
Testing
Degree Audits
Student Info
Production
Total Batch Jobs
01/02
05/06
9,304
27,921
111,290
379,474
375,534
285,497
7,251
6,314
81,723
156,576
354,429
352,683
1,189,020
958,976
* Eliminated 12/31/2006
** Transferred to Windows Based Application (DARWIN) November 2005
%Change
-22.0%
-77.4% *
-26.57%
-58.74% **
-5.6%
23.53%
-19.35%
FTP’s
 88,000+ executed in 2006
 VPN
 Mellon Bank
 PNC Bank
 PHEAA
 PGP
 Concur (ERS)
 SSL
 All ftp’s are initiated on the mainframe
FTP’s (cont)
 File Access
 ACF2 controls access to all files
 An access rule must exist for a file. (No rule, No access)
 Access rules based on file name
Tape Inventory
Shields Building:
3590
1,075 Robot Tapes (AIS Production/testing)
Computer Building:
3480
12,600
1. 3,000 History Tapes
2. 2,600 used exclusively by Budget Office
3. 7,000 used for AIS Production/testing
3590
1,000 Robot Tapes (AIS Production/testing)
Tape Inventory (cont)
Wagner Building:
3420
1,200 History Tapes
3590
14 Catastrophe Robot Tapes
Altoona:
Total Tapes in Inventory:
15,889
Five Year Report Output Trend
Fiscal Year Ending
Impact Printer
Remote Printers
(user Offices)
Microfiche
Laser Printers
(AIS)
eDDS
Total Pages
01/02
05/06
%Change
1,207,388
4,020,256
529,977
2,709,270
-56.1% *
-32.4%
8,115,918
8,265,160
-05,735,841
-100% *
-30.6% *
14,213,789
23,641,751
+66.3% **
35,812,511
32,616,839
-8.92%
* Output Distributed by Production Control to Customers
** There are 2 eDDS server applications. One contains only payroll reports and has very limited access.
Annual Printing Tidbits
29% of total printing:
 600,000 SRTE’s printed (3 runs)
 508,000 Deposit slips printed (38 runs)
 233,000 Account Payable checks printed (250 runs)
 101,000 1098T tax forms printed (202,000 pages, 1 run)
 550 Executive Deposit Slips (12 runs)
13 % of annual printing done in January
Projects:
 Microfiche Elimination
 Completed in 2003
 Estimated annual savings: $80,000
 RMDS Elimination
 Completed in 2006
 Estimated annual savings: $12,000
Projects: (cont)
 Laser Printing by AIS
 Completed January 31, 2007
 Estimated annual savings: $23,000
 3480 Tape Elimination
 Estimated completion: June 30, 2007
 Estimated annual savings: $10,000
Computer Operations & Facilities
Discussion and Questions
Overview of Network Infrastructure and
Desktop Support
Todd Litzinger, Manager
Network Infrastructure Team Members
Todd Litzinger, Manager
Barry Phelps, Network Specialist
Bob Miller, Network Specialist
Kevin Cramer, Desktop Support Lead
Frank Alterio, Desktop Support
Doug Swope, Desktop/Telecom
Char Wilusz, Special Projects and Audits
Network Architecture Overview
 16 Subnets at 3 Locations
 Computer Building, Shields, and Altoona
 Production Subnets for both MidTier and Enterprise Systems
 Redundancy:
 Firewalls
 Layer 3 Switches
 TNS Router
 Physical pathing
Previous Physical Pathing
Current Physical Pathing
LAN – Desktop Support
 Support 140 Staff, Teleworkers
 Windows Domain Based
 Hardware: Dell 95%, handful of IBM and Mac, Palm PDA
 Desktop hardware refreshed every 3-4 years via ITS Bulk Buy
 50-50 Laptop to Desktop Mix
 Office 2003 -> 2007
 Meeting Maker Calendar with an eye towards Oracle Calendar
Audits
 AIS is actively audited by the following areas:
 Internal Auditing
 269 recommendations made over the years
 More coming in 2007: ACH, DWH/FERPA, IBIS
 Deloitte and Touche (annually)
 Defense contract Audit Agency (DCAA)
 AIS must stay compliant with a number of areas such as FERPA, HIPAA, CISP, ACH.
AIS ACTIVE INTERNAL AUDITS SUMMARY
May 2006 - October 2006
Audit Description (In Priority Order)
Mainframe Supervisory Calls (SVC)
MVS System Review
Grade Collection & Reporting
Natural Security
APF- Authorized Libraries
CA-ACF2 Security Subsystem
Program Properties Table (PPT)
PSU Purchasing Cards
Business Continuity
Data Warehouse
E-Business – IBM Payment Server
SecurID Cards
Voice Response System
TOTALS:
Original Number of
Recommendations
Recommendations Still Open
16
13
19
52
17
47
10
10
11
17
15
26
16
6
5
9
42
1
10
0
0
0
0
0
0
0
0
0
3
3
0
0
0
0
0
0
0
0
0
3
1
2
22
1
6
0
0
0
0
0
0
0
3
4
4
17
0
4
0
0
0
0
0
0
0
269
73
6
35
32
Percentage Open
27%
Closed This 6 Month Cycle
16
Total Closed to Date
196
High Priority Medium Priority
Low Priority
Network Infrastructure & Desktop Support
Discussion and Questions
Overview of Enterprise Infrastructure – ES
Mike Kauffman, Manager, ES
Enterprise Group Discussion Points:
 Mike Kauffman: Overview of Infrastructure Group
 Bill Cook: Primary Production Responsibilities
 Phil Hawkins: New Projects: VM and Linux Using IBM Series “Z”
Enterprise Infrastructure Groups and Staff:
Operating System
Team
Brian France
Ed Micucci
Allen Cheung
Matt Gourley
Database Team
Bill Cook
Rusty Sodergren
Mark Mikowicz
Christine McCrum
Storage Team
Gary Gentzel
Karl Nordstrom
Al Dix*
Integration Team
Phil Hawkins
Karin Hilbert
Daryl Hoffman
Dom DePasquale
5/19/2016
Major Hardware Overview:

IBM – z/890 16G Memory, 1-CP/1IFL, 12-Ficon/16-Escon
Voyager – Shields Building (IPO1, IPO3,IPO5 – VM)
NCC1701 Computer Building (IPO2, IPO4, IPO6 – VM)
 IBM – ESS – F20 Shark Storage
2TB – Count/Key – IBM Mainframe Storage
7TB – Fixed Block – Wintel Storage
4TB – SAN Volume Controller (SVC) Controlled
Major Hardware Overview (cont):
 IBM -3490 Tape Libraries (2) Mainframe/WinTel Shared
 Miscellaneous – 3420/3480 Tape, Sysplex Timers, Visara
 Console Controllers, ESCON Directors, Network
Major Operating System Overview:
 z/OS – Base Level Sysplex with Shared SPOOL/Jes2:
IPO1-ISIS
IPO2-IBIS
IPO3/4/5/6 – Test – D/R - Maint
 z/OS – USS (Unix System Services)
 z/VM – IFL Hypervisor
 z/Linux – (Approximately 10 instances)
Major Software Overview:
 ACF2/CA7/WLM/MIM/Roscoe/TSO-ISPF/FDR/Upstream/FTP
 SMS/HSM/CA1/CA-Vtape/ESS-Expert
 Adabas Database/Complete/Natural/EntireX
 Websphere/DB2/iBPM/NetworkDeploy/EntireX/Tamino
Business Partners Overview:

Primary: IBM
Historic Business Partner
Consistent Migration Path from MFT to z/OS
Production Focus
Excellent Field Support
 Primary: Software AG
Relationship Built Over 25 Years
Excellent Performance and Consistency
Forward Looking Products – Tamino/EntireX
Business Partners Overview (cont):
 Secondary: Computer Associates
 Other: LRS, Syncsort, BIS, Innovation, SAS
Projects and Focus Points From
The Past Year:








VM/Linux Infrastructure
Hydra/GI Migration (DCE Removal)
Print Migration to Media and Print Center
Disaster and Catastrophe Recovery Planning
Tape: Library Upgrade and Virtualization (CA-Vtape)
Improvement To Hours of Availability
Secure Data Transfer – Travel Services Project
Security Issue Resolution – Natural, Grades
Peak Workloads and Challenges:
 Registration and Drop/Add
Day 2 – Spring 07 – 30 Million Adabas IBIS/ISIS commands
Determining Peak Growth Rate Difficult
Changing Architecture
Changing Calendars
Estimate 15 – 20%
 Grades – Becoming The Major Focus Point
Short Duration – But Heavy Impact
Required Before Commencement Exercises
Enterprise Infrastructure
Discussion and Questions
Overview of Database Administration for
Enterprise Systems – DBA-ES
Bill Cook, Manager, DBA-ES
Current Staff:
Bill Cook – Manager
Rusty Sodergren – Senior DBA
Mark Mikowicz – Intermediate DBA
Christine McCrum – Junior DBA
DBA Functions: Install and Manage Software
Build, Manage Databases
Provide Technical Support
Software Vendors: Software AG
Treehouse Software
Computer Associates
Primary Database Software Vendor:
Software AG
 Installed at PSU since 1982
 ISIS, IBIS, ADIS and eLion run on the Software AG suite of
products
Software AG Core Product Line: Database
Data Dictionary
Pgm Language
TP Monitor
Middleware





Database:
Data Dictionary:
Pgm Language:
TP Monitor:
Middleware:
Adabas
Predict
Natural
Com-plete
EntireX Broker, RPC Server, Net-work
Software AG Other Products: Remote Development
Code Analysis
Refactoring
 Remote Development:
Natural Development Server
Single Point of Development (SPoD)
Natural for Eclipse
 Code Analysis and Refactoring:
Natural Engineer
Software AG Advantages:
 Adabas: Extremely Fast and Reliable; Natural, SQL, XML Access




Predict: Full Documentation of Systems
Natural: Easy To Learn and Powerful
Com-plete: Reliable
Broker, RPC Server, Net-work: Fast, Powerful and Reliable
Software AG Advantages (cont):




Friendly, small-company feel
Technical Support is excellent
Products run on z/OS, Unix, Linux, zLinux and Windows
Builds a real partnership with customers
Some Database Numbers…
 ADA229 Test Database: 61 Million Records
 ADA227 ISIS Database: 151 Million Records
 ADA226 IBIS Database: 459 Million Records
Primary Tools Software Vendor:
Treehouse Software
 Installed at PSU since 2002
 Software used to convert files, model and propagate data
Treehouse Product Line: File Conversions
Data Modeling
Data Propagation
 File Conversions: ADAREORG
 Data Modeling:
tRelational
 Data Propagation: DPS (Data Propagation System)
Treehouse Software Advantages:
 ADAREORG: Significantly Cuts File Conversion Times (in one
instance shortened a file conversion from 24 hours to 2 hours)
 tRelational: Interactive Data Modeling from Adabas to RDBMS
 DPS: Efficiently Propagates Data From Adabas To Any Format
Secondary Tools Software Vendor:
Computer Associates
 Installed at PSU since 1985
 Software used to create performance reports, trace problems
CA Product Line: Performance Reports
Tracing Problems
 Performance Reports: APAS (Adabas Performance Analysis System)
 Tracing Problems: Insight
Enterprise Infrastructure – Database
Administration
Discussion and Questions
Overview of Enterprise Services Integration
for Enterprise Systems – ESI-ES
Phil Hawkins, Team Leader, ESI
Daryl Hoffman, Systems Administrator
Karin Hilbert, Systems Administrator
Dom DePasquale, Systems Administrator
Enterprise Services Integration:
Born of the need to build a Services Oriented Architecture in support of the AIS
modernization program on the enterprise server utilizing the zVM/zLinux O/S platform

> NEW and EXCITING Technologies, c0mplexity is integrating them into real solutions
 Workflow and PELL XML document processing are the first implementations of the AIS
J2EE modernization strategy
 2 New Operating Systems, New Database Technology, New Application Server Environment, New
Web Server Environment, Newly ported Workflow Engine, New Software AG XML software……BUT
on the familiar enterprise server iron
SO…..Where are We?
 Built 3 test architectures since 11/05 including a windows environment and multiple
zLinux environments
 Successful workflow engine port test to zLinux in 08/2006
 Completely abandoned the windows environment 09/2006
 Pilot test for UGTR workflow application 10/2006
SO…..Where are We (cont)?
 PELL ready for a parallel test on zLinux using all new XML document processing
software
 IBM health check completed and recommendations implemented
 Production environment created & ready for use on March 15th
 Leveraged the power of virtualization on a known hardware platform, incorporated vendor best
practices on the environment, and started to integrate into the organizations’ strengths in
production control and operations
New Technologies:
 XML
 XML orchestration using Software AG’s Mediator
 XML database using Software AG’s Tamino
 Software AG’s IBPM workflow engine
 IBM’s DB2 relational database
 IBM’s Websphere Application Server Network Deploy
 zVM and zlinux OS
 LDAP role based authorization security
 IBM’s Omegamon monitoring software
 CA’s operator software
What’s Next?







Continued emphasis on identity management and security architectures – corroboration with ASSET
Implement true SOA – Governance, Registry, Websphere ESB, CentraSite
Implement Web Server Load Balancer and Security Gateway Appliance
Cross Training
Migration of eCommerce to new architecture
Automating alerts and increased monitoring
High availability and DR documentation
Enterprise Infrastructure – ESI
Discussion and Questions
AIS MidTier Infrastructure
Pete Dawson, Manager,
Scott Neidigh, Lead Systems Administrator,
Alex Pollock, Lead Database Administrator
5/19/2016
AISMission,
MidTier
Infrastructure
Staffing,
Unit Organization
Mission, Staffing, Unit Organization
Server Admin Team
Database Team
Storage Team
Systems Pgm Team
Scott Neidigh
Alex Pollock
Lowell Smith
Dave Reed
Matt Henry
Todd Kos
Gordy Peck
Rick Rhoades
Darryl Griffin
Russ Mulkey
Jeff Davidson
Chris Grove
AIS Services
ADIS
ePay
Grads
AIMS
eStore
Hershey
AIS Web
eCommerce
Imaging
AIS RedDot
eDDS
LDRPS
ANGEL
eDDS Payroll
MeetingMaker
Anti-Virus
EIS
MyAdmissions
ASAPB
eISIS
PIA Workload
BSR
eLion
Registrar
College Cost
eLion2
SQL Consolidated
DARWin
EMA Imaging
SSN
Data Warehouse
Load Balancer
Talisma
Domain Controller
Fitness
VMWare
DFI
FDR Upstream
WebIBIS
eSteward
FM Pro
WebSphere
eAccount
FTCAP
WebStat
eBill
GI
World Campus
AISBusiness
MidTier
Infrastructure
partners,
hardware & software
Business partners, hardware & software
ANGEL USERS
ANGEL
Computer Building
SHIELDS
HTT
PS
S
HTTP
Load Balancer
S-F5
HT
TP
Load Balancer
C-F5
Angel-Web-P01
PE1850
Angel-Web-P06
PE1850
File DT Replication
ANGEL-FILE-P01
PE2850
ANGEL-FILE-P02
PE2850
SQL DT Replication
Angel-Web-P05
PE1850
ANGEL-SQL-P02
PE6850
ANGEL-SQL-P01
PE6850
Brocade 2
FcSw02
Brocade 1
FcSw01
CX700 (spA)
CX700 (spB)
Cx70x01
SN (CR5Qn81)
Brocade 3
FcSw03
Brocade 4
FcSw04
CX700 (spA)
CX700 (spB)
Cx70x01
SN (BR5QN81)
AIS MidTier Infrastructure
Systems Management, Job Monitoring, Logging
Systems Management, Job Monitoring, Logging
Dave
Reed,
RickRick
Rhoades
Dave
Reed,
Rhoades
• 3000 service requests annually
• 12,000 scheduled jobs executed
• 9,500 files accessed
AIS MidTier Infrastructure
Systems Administration
Systems Administration
Support
& Maintenance
Support
& Maintenance
• 278 total devices
• 77 ANGEL devices or 28%
• 133 production servers
AIS MidTier Infrastructure
Systems Administration
Systems Administration
Growing
size
complexity
Growing
inin
size
&&
complexity
• 50 TB storage currently and growing
• More prevalent Windows 2003 64-bit
• Dell OpenManage monitoring
AIS MidTier Infrastructure
Database Administration and System Programming
Database Administration and System Programming
Server
Oracle
on Windows
SQLSQL
Server
and and
Oracle
on Windows
• 509 databases
• 2 TB for just the db data
• 28 db servers
AIS MidTier Infrastructure
Database Administration and System Programming
Database Administration and System Programming
Server
Oracle
on Windows
SQLSQL
Server
and and
Oracle
on Windows
DWH, Alumni BSR,
DARWIN, Donor DSS
CA7 Agent and home grown
system to coordinate batch jobs
33 SQL databases for
Admissions/Student Aid
Double-Take software for quick
HA/DR failover
Angel db server growth
Data retention, rolling partitions
AISFuture
MidTier
Infrastructure
Directions
Future Direction
• Migration of services to VM
• Storage strategy and management
• Load testing for high demand apps
Load
Testing
Load Client
192.168.0.80
Load Client
192.168.0.81
Load Client
192.168.0.82
Load Client
192.168.0.83
Load Client
192.168.0.88
Load Client
192.168.0.89
Load Client
192.168.0.91
Start Test
2
1
Stop Test
4
1
Load Controller
192.168.0.90
1
Load Client
192.168.0.84
Load Client
192.168.0.85
Load Client
Load Client Load Client
192.168.0.86 192.168.0.87 192.168.0.93
Load Client
Load Client
192.168.0.94 192.168.0.92
3
1
Load Test Group
Start Performance Counters
&
SQL Trace
Generated Http Get/Post
Data Collection
Web Logs
SQL Trace
Perfmon Counters
SQL Server
192.168.0.110
5
1
Web Group
Data
Web Server
192.168.0.120
6
1
Data Collection Point
Web Server
192.168.0.121
Web Server
192.168.0.122
Web Server
192.168.0.123
Generate
Reports
Data Group
Test Process Diagram
SQL Server
192.168.0.100
Lowell Smith
4 Jan 2007
Data
Web Server
192.168.0.124
Web Server
192.168.0.125