Transcript Slide 1

WW TSS-13
Wonderware MES
Support and Troubleshooting Technique
Presenters:
Dan Scott
Jim Stella
© 2012 Invensys. All Rights Reserved. The names, logos, and taglines identifying the products and services of Invensys are proprietary marks of
Invensys or its subsidiaries. All third party trademarks and service marks are the proprietary marks of their respective owners.
Session Agenda
• MES Architecture Overview
• Middleware Technology Changes 3.5 to 4.x
• Database Tables
• Tips and Common Mistakes
• Troubleshooting
Slide 3
MES Architecture
Overview
Slide 4
Wonderware MES Architecture
MES Client
Operator
MES Client
Supervisor
InTouch w/
.NET Controls
Supervisor
Reports
Supervisory (P)
Configurator
InTouch w/
MES API
ArchestrA IDE
Shop Floor (P)
MES Admin (P)
Galaxy
Repository
Field I/O
AOS Platform (M)
Factory
Connector
Server (M)
Middleware
Server (M,S)
Reports
Server (M or None)
Key: M = Full Middleware
Slide 5
Database
Server
P = Middleware Proxy
S = MES Service
Operations
Capability Obj
Utilization
Capability Obj
Sample
Recording Obj
App Objects
w/ MES API
Technology change in
MES Middleware
from 3.5 to 4.x
Slide 6
Wonderware MES Middleware 3.5
Slide 7
Wonderware MES Middleware 4.x WCF
(Windows Communication Foundation)
Slide 8
MES Database Tables
Slide 9
Common MES Database Tables Used in
Troubleshooting
Sessn – User sessions
– Who is logged into which session ID, or not in some cases
– Records with heartbeat older than 10 minutes can be purged
• These should be auto-purged by MES Service if it is running
– Only ever one session type 53 – Background User for MES Service
• Orphaned type 53 record can stop MES Service from starting
– Tech Note 699 covers this table and the client type in more detail
Ent_Logon – User logged on entities
– Which user is logged into which entities
– User must have privilege for those entities
Slide 10
Common MES Database Tables Used in
Troubleshooting
Item_Prod – Production records
– Distinct: Default selection
• Can report hourly or shift OEE
• Records stored for every production or consumption transaction
• Can result in large record count
– Shift:
• Can report shift OEE only
• Most compact record count
– Hourly: New for 3.5 p02, MES 4.0 SP2+ (replaces shift)
• Can report hourly or shift OEE
• Compromise between Distinct and Shift option
• Should be purged periodically to maintain high performance
Slide 11
Common MES Database Tables Used in
Troubleshooting
Util_Log – Utilization history
– Should be purged periodically to maintain high performance
TPM_Stat – OEE raw data
– Data only stored when a job is running
– Should be purged periodically to maintain high performance
Job – Jobs and related status
– Contains total production count, allowing item_prod details to be purged
Job_Exec – Active job information
– One placeholder record per allow Job Exec per entity
– Never delete records
Slide 12
Common MES Database Tables Used in
Troubleshooting
Util_Exec – Current utilization information
– Lookup current raw reason
OEE_Exec – Current OEE information
– See current Quality and Performance OEE subcomponents
– Quality % x Performance % x Availability % = OEE %
Ent – Entities
– View current shift ID and times
– View capabilities
Slide 13
Common MES Database Tables Used in
Troubleshooting
Shift_to_go – Upcoming shifts
– Can be purged, then restart MES Service to rebuild list
Tz_offset – Pre-calculated time zone offsets from UTC
– Only time zones of interest are calculated
– Can be purged, then restart MES Service to rebuild list
– Purge is required if the database is moved to another time zone
WO – Work Orders
– Should be purged at some point. (Archive/Purge has System parameters
that can be defined to automatically remove the data)
Item_inv – Inventory
Slide 14
MES Tips and
Common Mistakes
Slide 15
MES Tips and Common Mistakes:
Middleware
• Middleware proxy must be installed on IDE node if using Entity Model
Builder (EMB)
– You must open/close a capability object editor window before running EMB
• Full middleware should be installed App Object and Manager nodes
– License purchase required per full middleware server (multiple inc.)
– Middleware is not deployed, must be installed manually
• Use Proxy Configuration Editor to test the connection to MW server
• Use MES Client or Configurator to test client connectivity since you
will get better errors
• If middleware is remote from database, MES Client must be installed
Slide 16
MES Tips and Common Mistakes:
Load Management
Server / Platform Loading
Middleware Loading
• What else runs on this node?
• Capability Object count?
• Available system resources?
Engine Loading
• Engine Scan Rate?
– Many capability object functions result
in MES database transactions.
– Consider separate engines for MES
objects versus regular, higher speed,
App Objects
– No more than 200-300 per platform*
– No more than 50 SROs per platform*
– No more than 30-50 per engine*
– Full middleware on the platform hosting
MES App Objects
• Reporting?
Database Loading
• Dedicated MES Database server?
– MES has high transaction count
*Suggestions dependent on transaction volume and scan rate
Slide 17
MES Tips and Common Mistakes:
Maintain Distinct Production Records
• Enabled to report Hourly OEE
• Daily/Shift OEE can still be reported
with this disabled
• This setting can have a significant
impact on the MES database size
– If enabled (default), a new item_prod
record is created every time production
is recorded
– If disabled, the system will update
existing item_prod records
• New feature in 3.5 p02, 4.0 SP2
aggregated hourly prod records
Slide 18
MES Tips and Common Mistakes:
Multiple Job Execution
• How many Jobs can be run
concurrently on each entity
• More than 10 can cause problems
• OCO has a hard cap of 50, but not
recommended for entities with large
transaction numbers
Slide 19
MES Tips and Common Mistakes:
OCO/UCO Production Counters
Misconfiguration, especially with a rollover counter, can be disastrous
• NEVER set Deadband, Max Value, and Update Interval to 0
– Production may be recorded every engine scan
• Every time production is recorded, you are transacting with the MES Database
– If Maintain Distinct Production Records is enabled (default), you are creating a new
item_prod records every time production is recorded
• UCO rollover counter Max Value MUST be synced with PLC rollover
– OCO rollover counter can be reset on demand, or PLC sync (TN 710)
Slide 20
MES Tips and Common Mistakes:
Raw Reason Code Management (UCO)
Every Raw Reason Code change
results in multiple MES Database
transactions
• Changing reasons too fast can cause
reason codes to be missed (TN657)
– UCO can only handle current change
plus queue the next change
– Micro stops should be recorded to
Historian (events < 30-60s)*
• Future product plans in the
works to allow for faster
changes
*Suggestion dependent on transaction volume and scan rate
Slide 21
MES Tips and Common Mistakes:
Continuously Running Work Order
“But we’re only using Performance, we don’t care about Work Orders”
• The underlying MES database structure and table relationships makes
Work Order management important
– Problems with continuously running work orders:
• Database performance degrades over time
– Updating database records require the system to query large datasets
• Archiving capabilities are limited (ex: Archive WO where closed > 30 days)
• General recommendation is a new Work Order daily or at shift change
Slide 22
MES Tips and Common Mistakes:
Work Queue Grid Filtering
The more that you try to display on the Work Queue grid, the slower it
will load and refresh
• How many Jobs does an operator really need to see?
• Use New/Ready status to limit records
– Create jobs as New, have a supervisor set them to Ready when they assign
the work. Hide New jobs from the operator’s work queue.
• Eliminates operator confusion
Slide 23
MES Tips and Common Mistakes:
Utilization Grid Filtering
The more that you try to display on
the Utilization grid, the slower it will
load and refresh
• No filter will query the database for
ALL utilization records for this entity
• What does the operator really need
to see vs. what can just show up on
a report?
(right-click)
Slide 24
MES Tips and Common Mistakes:
Security
• If utilizing OS-based security, the
Wonderware Change Network User
account must be a domain user
• Special steps required to enable OS
Group Security - Tech Note 708
• Lost Database Connection Editor
password
– Delete C:\Program Files\Wonderware\
MES\Middleware\Middleware.cfg
Slide 25
MES Tips and Common Mistakes:
OEE Calculation
Slide 26
MES Tips and Common Mistakes:
Data Inconsistencies
• Moving database to a new timezone?
– Purge shift_to_go and tz_offset tables
– Restart MES Service
• Errors that Job exec limit exceeded, but Operator work queue does
not show that many jobs running
– Job state and Job_Exec current job are out of sync
Slide 27
MES Tips and Common Mistakes:
What does MES Service do?
• Only one MES Service can run per database
• Executes minutely, hourly, and daily housekeeping tasks
– Shift changes
– Session clean-up
– Auto-ready jobs
– Run scheduled Supply Chain Connector tasks
– Run scheduled archive and purge tasks
– Generate time-based Quality Samples
Slide 28
MES Tips and Common Mistakes:
System Maintenance
• Database maintenance
– Truncate transaction logs and shrink files and database (TN 664)
• Archive and/or Purge
– Large tables:
• item_prod (Production details)
• util_log (Utilization states)
• tpm_stat (OEE stats)
– Custom reports can union data from Prod and Archive databases
• Available from MES Services, not part of the standard product
– Tech Note 746 - Using the New MES 4.0 Archive, Purge and Restore
Functions
Slide 29
MES Tips and Common Mistakes:
Supply Chain Connector
File
Xml or
delimited
Table
Supply Chain
Connector
(MES Service)
Middleware
3rd party or
custom
Supply Chain Connector can have data brought in by either data file or
database Table.
Slide 30
MES Tips and Common Mistakes:
Supply Chain Connector
Questions to consider when deciding whether to use files or
tables:
•
How much data will I be importing?
•
How often will the data be imported?
•
How much of the importing data will be new and how much will be
updating existing records?
•
What will be creating the new data that will be imported?
Slide 31
MES Tips and Common Mistakes:
Supply Chain Connector
File Base imports:
Pros
•
Easy to implement
•
Good for smaller amounts of data
•
Easy one time configuration scenarios
Cons
•
Cannot track the processing of individual records
•
Higher chances of data integrity issues
•
Slower performance with larger amounts of data
Slide 32
MES Tips and Common Mistakes:
Supply Chain Connector
Table Base imports:
Pros
•
Able to track and keep information on individual processed records
•
Reprocessing of individual records is easier
•
Troubleshooting bad records is easier to find.
Cons
•
More difficult to implement
•
Better for larger amounts of data.
•
Needs more maintenance (cleaning out old records, backing up of
tables….)
Slide 33
MES Troubleshooting
Tools and Techniques
Slide 34
MES Troubleshooting:
Middleware
• Licensing
– MES Licenses must be installed on Middleware servers (restart MW)
– Performance-only or Quality-only license can use OCO for production counts
and basic job execution
– MES 4.0 license is valid for MES 4.5
• Database Connection Editor (restart Middleware if changed)
– Verify SQL connectivity
• Middleware Configuration Editor
– Enable XML Trace of all middleware/SP calls with parameters (restart MW)
• Middleware Proxy Configuration Editor
– Test connection to Middleware server
Slide 35
MES Troubleshooting:
Error Messages
• MES SMC Log Flags
– UCO – EquipmentStatusRuntime1
– OCO – JobExecutionRuntime1
– SRO – SampleCharacteristicRuntime1
– Entity Model Builder – EntityModelBuilder
– MES Service – FactMES.Server.Services
• MES ArchestrA Objects
– .ErrorMessage attribute
Slide 36
MES Troubleshooting:
MES Client Debug
• Modify C:\Program Files\Wonderware\MES\Client Framework
Modules\aaClient.exe.config
Change this section:
<!-- Uncomment to enable the file based trace listener.
<add name="LogTraceListener"
type="System.Diagnostics.TextWriterTraceListener“
initializeData="Trace.log" /> -->
<!-- Uncomment to enable the file based trace listener. -->
<add name="LogTraceListener“
type="System.Diagnostics.TextWriterTraceListener“
initializeData="Trace.log" />
Slide 37
Useful Tech Notes
• Tech Note 753 - Resolving MES Entity Model Builder Errors on a
Remote IDE Development Node
• Tech Note 746 - Using the New MES 4.0 Archive, Purge and Restore
Functions
• Tech Note 744 - Resolving Incorrect Shift Times When MESDB is
Moved to Another Time Zone
• Tech Note 723 - Solving Wonderware MES 4.0 Known Installation
Issues
• Tech Note 711 – Wonderware MES Operation Capability Object:
Configuring Multiple Job Counters
Slide 38
Useful Tech Notes
• Tech Note 710 - Wonderware MES Operations Capability Object:
Configuring Rolling Counter Reset
• Tech Note 708 - Enabling MES 4.0 Operating System (OS) Group
Security
• Tech Note 699 - Wonderware MES Sessn Table and Client Types
• Tech Note 664 - Truncating Operations & Performance Transaction
Logs
• Tech Note 657 - Wonderware Performance 3.5 Software:
Performance Tips
Slide 39
Other Useful Documents
• MES 4.0 Deployment Guide
• MES Database Diagrams
• MES 4.0 Development Guide
Slide 40
Useful Sessions
WW TSS-09:
MS SQL Server Performance & Tuning
for Wonderware Database
Slide 41
Questions?
THANK YOU
Slide 42