Transcript QAD Monitor - Midwest User Group
FNST’s QAD Monitoring
One part of the monitoring/management initiative for FNST/FST
© Freudenberg-NOK General Partnership
1
Agenda
• Intro – About FNST • Background/History • QAD Implementation • Present State • Benefits Realized • Future Directions • Quick Review (screenshots) • Lessons Learned • Q&A
© Freudenberg-NOK General Partnership
2
Freudenberg-NOK Sealing Technologies Established 1989 Headquartered in Plymouth, Mich., USA
(Privately Held)
~23% Established 1849
Headquartered in Weinheim, Germany (Privately Held) © Freudenberg-NOK General Partnership
Established 1939
Headquartered in Tokyo, Japan (Publically Traded on the NSS)
3
Diverse and Established Partners
Headquarters Established 2008 Sales Ownership Products Plymouth, Michigan
1989 $1 Billion Privately Held Radial Shaft Seals O-Rings Gaskets Fluid Power Products Diaphragms Specialty Sealing Products Weinheim, Germany 1849 $6.5 Billion Privately Held Radial Shaft Seals O-Rings Gaskets Fluid Power Products Oil and Gas Products Diaphragms Specialty Sealing Products Vibration Control Devices Medical Products Tokyo, Japan 1939 $4.5 Billion Publicly Held Radial Shaft Seals O-Rings Gaskets Fluid Power Products Diaphragms Specialty Sealing Products Vibration Control Devices © Freudenberg-NOK General Partnership
4
Freudenberg Global locations
Production Sales Production and Sales © Freudenberg-NOK General Partnership
5
Production Locations in the Americas
Elgin, IL
Simrit Sales Office
Plymouth, MI
Technology Center
Morristown, IN
Custom Molded Rubber Products
Shelbyville, IN
Boots & Bellows
Necedah, WI
Gaskets
Santa Ana, CA
O-rings
Quer étaro, Mexico
O-rings
Cuautla, Mexico
Radial Shaft Seals and Custom Molded Rubber Products and Vibration Control Devices
New Hampshire Facilities: Ashland
–
Aerospace Radial Shaft Seals Industrial Radial Shaft Seals
Tillsonburg, ON
Silicone and Fluoro-silicone Products
Bristol
–
Mechanical Face Seals, Bonded Piston Seals
Northfield
–
Axle Seals
Manchester
–
Gaskets
Manchester
–
Microcellular Polyurethane
Cleveland, GA
Valve Stem Seals and Powertrain Seals
LaGrange, GA
O-rings
Milan, OH
Heavy Industry Lead Center, Imported Products
Findlay, OH
PTFE Products
Troy, OH
PTFE Products
Sao Paulo, Brazil
Radial Shaft Seals
© Freudenberg-NOK General Partnership
6
Freudenberg-NOK Sealing Technologies Today Responsible for sealing technology business in the Americas Total sales in 2010: US$ 798.7 million Workforce in 2010: 4,224 Locations in: U.S., Canada, Mexico, Brazil, Malaysia, China Sales exceeded market recovery in 13 of 15 principal market segments Sites: 35 production, sales and technical facilities
© Freudenberg-NOK General Partnership
7
• • • • • • •
Background/History
Agenda © Freudenberg Sealing Technologies GmbH & Co. KG
8
Background/History
HP OpenView implemented in 1994 Dedicated a great deal of effort to include Progress and QAD in our monitoring/management Versions not kept current (budget constraints) to support our Linux environment
© Freudenberg Sealing Technologies GmbH & Co. KG
9
• • • • • • •
QAD Implementation
Agenda © Freudenberg Sealing Technologies GmbH & Co. KG
10
QAD Implementation -
Contracted QAD Services.
“Canned” product, virtual Linux server (recover image and change IP), Icinga and Cacti for alerting and graphing.
“Base” configuration file created. Deployment simplified, copy base and change host info.
-
Scripts built for deploying: Database monitors based on our “/node/info/databases” file (list of databases per host with environment, used for many purposes including start-up).
Disk statistics.
Graphs too.
© Freudenberg Sealing Technologies GmbH & Co. KG
11
Our Environment
Centralized 2010SE Host (co-lo in NC) Supporting 30 sites from Canada to Mexico Distribution Site Linux eB2 (our facility in OH) Brazil Site Linux 2007
© Freudenberg Sealing Technologies GmbH & Co. KG
12
Our Monitoring Environment
Each host has a system “control” file containing all databases and environments This allows easy scripting of status checks, up/down, and query from other applications and scripts to invoke This file allowed us to automate our monitoring implementation
© Freudenberg Sealing Technologies GmbH & Co. KG
13
Database Control File
###################### /prod/prddbhe10/prddbhe10 DEFAULT hlpea PRODEA10 QAD_add-on /prod/prddbqxo17/prddbqxo17 DEFAULT qxodb PRODEA10 QAD_add-on ####################### /prod/prdcmbme10/prdcmbme10 DEFAULT mfgdb PRODEA10 Prod_QAD /prod/prdcmbae10/prdcmbae10 DEFAULT admea PRODEA10 /prod/prdcmbfe10/prdcmbfe10 DEFAULT fngpea PRODEA10 FNGP_QAD_add-on /prod/prdcmbte10/prdcmbte10 DEFAULT tpea PRODEA10 TailorPro /prod/prdcmbev17/prdcmbev17 DEFAULT qxevents PRODEA10 Qxtend_events ####################### ## Corteco Mexico ####################### /prod/prd73me10/prd73me10 DEFAULT mfg73 PRODEA10 Prod_QAD mexico /prod/prd73ae10/prd73ae10 DEFAULT admea PRODEA10 /prod/prd73fe10/prd73fe10 DEFAULT fngpea PRODEA10 FNGP_QAD_add-on /prod/prd73te10/prd73te10 DEFAULT tpea PRODEA10 TailorPro /prod/prd73ev17/prd73ev17 DEFAULT qxevents PRODEA10 Qxtend_events #######################################################
© Freudenberg Sealing Technologies GmbH & Co. KG
14
Site Monitoring
Ping a host in each supported site Good method of determining how a site sees performance
© Freudenberg Sealing Technologies GmbH & Co. KG
15
Scripts for Automation of Monitoring Services
./build_server_cfg.sh [remote host] [DLC] Builds a file with the Icinga components for Progress database monitors based on the control file. This generated file gets inserted into the host’s configuration file in Icinga.
Creates Icinga monitors for DB check, extents, log entries, last backup, buffer hits, read rates, and locks ./build_server_graph.sh [remote host] [DLC] Queries for host ID, then creates Cacti graphs for each database in the control file ./build_server_disk.sh [remote host] Ran a “df” on the remote host, builds Icinga space monitor for each local disk, insert into host’s configuration file.
© Freudenberg Sealing Technologies GmbH & Co. KG
16
Script to Build Configuration File
copy the template to [servername]-server.cfg
cp fnst-server.cfg.template [servername]-server.cfg
find and replace ral2lnxqad with the server name edit the host IP correct the webspeed entries correct DLC if needed build the databases section: ./build_server_cfg.sh [remote host] [DLC]
© Freudenberg Sealing Technologies GmbH & Co. KG
17
• • • • • • • • • Present State
Agenda © Freudenberg Sealing Technologies GmbH & Co. KG
18
Present State
All Linux hosts monitored (ping, OS) All sites monitored (ping to a host in all sites) All Linux QAD hosts monitored (Progress databases, NetUI performance)
© Freudenberg Sealing Technologies GmbH & Co. KG
19
• • • • • • • • • Benefits Realized
Agenda © Freudenberg Sealing Technologies GmbH & Co. KG
20
Benefits Realized
Network issues identified “stair step” graph noticed, turns out cable disconnect at site Qxtend performance issue Alert on CPU utilization, worked w/ EA Team Cache Hit Rate Alert, increased “Big B” DB Writes/Min Worked w/ ERP Team, identified code that was not “domainized”
© Freudenberg Sealing Technologies GmbH & Co. KG
21
• • • • • • • • • Future Directions
Agenda © Freudenberg Sealing Technologies GmbH & Co. KG
22
Future Direction
Regular internal review.
Regular review with ERP Team.
Review what else would benefit us.
Remote point(s) for monitoring. NetUI testing for one.
With the implementation of Vmware in remote sites we may just deploy VM Include HPUX (retire OpenView).
Visibility to others outside the Server Team.
Migrate additional legacy performance and monitoring scripts Review how to play with SCOM
© Freudenberg Sealing Technologies GmbH & Co. KG
23
Agenda
• • • • • • • • • Quick Review (screenshots)
© Freudenberg Sealing Technologies GmbH & Co. KG
24
Alerts Graphs
Quick Review © Freudenberg Sealing Technologies GmbH & Co. KG
25
Graphical, Emails
Alerts © Freudenberg Sealing Technologies GmbH & Co. KG
26
Status/Alerts by Location/Service © Freudenberg Sealing Technologies GmbH & Co. KG
27
Alerts by Service © Freudenberg Sealing Technologies GmbH & Co. KG
28
Alerts by Host © Freudenberg Sealing Technologies GmbH & Co. KG
29
Email Alerts © Freudenberg Sealing Technologies GmbH & Co. KG
30
Graphs
Hourly (min avg), Daily (5 min avg), Weekly (30 min avg), Monthly (2 hr avg), Yearly (1 day avg) Data points include: Ping Response Host Statistics QAD Users DB Reads Time to Read 10K Records Database Performance Statistics NetUI View
© Freudenberg Sealing Technologies GmbH & Co. KG
31
Site Overview
We included (at least) one host in each of the sites accessing our central QAD servers This gives us up/down and response times by site, assisting in troubleshooting issues
© Freudenberg Sealing Technologies GmbH & Co. KG
32
Site Overview © Freudenberg Sealing Technologies GmbH & Co. KG
33
Ping Response Time © Freudenberg Sealing Technologies GmbH & Co. KG
34
Statistics Specific to QAD Hosts
Processor statistics QAD users Database statistics NetUI performance
© Freudenberg Sealing Technologies GmbH & Co. KG
35
Host Performance Statistics © Freudenberg Sealing Technologies GmbH & Co. KG
36
QAD Users © Freudenberg Sealing Technologies GmbH & Co. KG
37
QAD DB Reads © Freudenberg Sealing Technologies GmbH & Co. KG
38
Sidecar DB Reads © Freudenberg Sealing Technologies GmbH & Co. KG
39
Admin DB Reads © Freudenberg Sealing Technologies GmbH & Co. KG
40
Time to Read 10,000 Records © Freudenberg Sealing Technologies GmbH & Co. KG
41
tr_hist Records Added (good indicator of activity) © Freudenberg Sealing Technologies GmbH & Co. KG
42
NetUI Response © Freudenberg Sealing Technologies GmbH & Co. KG
43
Point browser to http://ra2sqadmon
Live © Freudenberg Sealing Technologies GmbH & Co. KG
44
Questions, Suggestions, Comments,
Questions?
© Freudenberg Sealing Technologies GmbH & Co. KG
45