Transcript PolyAnalyst presentation - Webdienste FHNW — Zugang

Mining data with
PolyAnalyst
Your Knowledge Partner
© 2002 Megaputer intelligence, Inc.
www.megaputer.com
TM
Outline

Data Mining in BI chain

PolyAnalyst overview

Learning algorithms

Additional features

Future developments
© 2002 Megaputer intelligence, Inc.
Data Mining
in BI chain
Your Knowledge Partner
© 2002 Megaputer intelligence, Inc.
TM
DM in Decision Making
Consider a fragment of the BI chain:
Data
Knowledge
Decision
Action

Data - is what we can capture and store

Knowledge - is what provides for informed decisions

Problem: How to get from Data to Knowledge?

Solution: Data Mining (Machine Learning)
© 2002 Megaputer intelligence, Inc.
Data Mining

"Data Mining is the process of identifying valid,
novel, potentially useful, and ultimately
comprehensible knowledge from databases that is
used to make crucial business decisions."
-- G. Piatetsky-Shapiro, KDNuggets editor
www.kdnuggets.com
Valid
 Novel
 Actionable
 Comprehensible

© 2002 Megaputer intelligence, Inc.
Data Mining vs. OLAP

OLAP
- Helps prove or reject your hypotheses
by dissecting data along different dimensions
- But you have to guess the answer first !

Data Mining
- Automatically develops and tests numerous
hypotheses by learning from historical data
- Analyzes raw data
© 2002 Megaputer intelligence, Inc.
Business Intelligence Chain

Consider direct marketing automation


Analyze data
Integrate applications
X
© 2002 Megaputer intelligence, Inc.
Data Mining Tasks









© 2002 Megaputer intelligence, Inc.
Predicting
Classifying
Clustering
Segmenting
Explaining
Associating
Visualizing
Link Analysis
Text Mining
Fields of application
Database marketers
Insurance and HMO companies
Securities and currency traders
Hospitals and physicians
Government agencies
Retailers
© 2002 Megaputer intelligence, Inc.
q
Response prediction
q
Market segmentation
q
Customer valuation
q
Cross-sell analysis
q
Customer retention
q
Head-cost reduction
q
Premium sensitivity testing
q
Forecasting market behavior
q
Trading strategy optimization
q
Medical diagnostics
q
Selecting medical interference
q
Fraud detection
q
Revenue prediction
q
Market Basket Analysis
q
Recommendation systems
What makes DM hard?






Unfamiliar concept and lack of experience
Results require interpretation by an analyst
Poor integration in existing applications
Difficulty processing very large databases
Necessity to learn a new application
High cost
© 2002 Megaputer intelligence, Inc.
Megaputer response

Challenge: Unfamiliar concept and lack of experience
Response: Collaborative Appliance Program – combines
Megaputer analysts expertise in data mining and customer
knowledge of the business project

Challenge: Results require interpretation by an analyst
Response: Simple reporting and batch processing
capabilities

Challenge: Poor integration in existing applications
Response: Easy scoring of external data with a few mouse
clicks

Challenge: Difficulty processing very large databases
Response: In-Place Data Mining

Challenge: Necessity to learn a new application
Response: An SDK of easy-to-integrate PolyAnalyst COM
components

Challenge: High cost
Response: Flexible licensing mechanism
© 2002 Megaputer intelligence, Inc.
PolyAnalyst
overview
Your Knowledge Partner
© 2002 Megaputer intelligence, Inc.
TM
What is PolyAnalyst?

Multi-strategy data mining suite



Ease-of-use:


friendly data manipulation and visualization
Deep integration




The largest selection of ML algorithms for diverse
business tasks
Structured data and text processing tools
Applying models to external DB through the OLE DB
protocol
Exporting models to XML
COM components
Best Price/Performance ratio
© 2002 Megaputer intelligence, Inc.
Key differentiators of PolyAnalyst

Integrated analysis of structured (numeric and categorical)
and unstructured (text) data

Easy to learn and operate visual analytical interface

The largest selection of powerful machine learning algorithms

Mouse-driven application of predictive models to data in any
external system through a standard OLEDB link

Simple integration with external applications: SDK of COM
components

In-Place Data Mining capabilities for processing huge
databases

Step-by-step tutorials based on real-world case studies

Rich data manipulation and visualization tools

Reusable analytical scripts for batch process data mining

The best Price/Performance ratio
© 2002 Megaputer intelligence, Inc.
PolyAnalyst
Customer base: 300+ installations
Sample customers
Boeing (USA)
3M (USA)
Chase Manhattan Bank (USA)
McKinsey & Company (USA)
Siemens (Germany)
Lockheed Martin (USA)
Allstate Insurance (USA)
ICICI Bank (India)
Mars (USA)
Taco Bell (USA)
DuPont (USA)
Asea Skandia (Sweden)
France Telecom (France)
Cambridge Technology Partners (USA)
Carlson Marketing (USA)
Central Bank (Russia)
US Navy (USA)
KPN Research (Netherlands)
Alka Insurance (Denmark)
National Cancer Institute (USA)
© 2002 Megaputer intelligence, Inc.
PolyAnalyst workplace
Project
navigation tree
Control buttons
Data and Results
pane
Exploration engine
report fragment
Objects and Collections
represented by icons
PolyAnalyst
log
journal
© 2002
Megaputer
intelligence, Inc.
PolyAnalyst provides

Access to data held in a database or data
warehouse




Numerical
Categorical
Yes/no
Date

Data manipulation and visualization

14 machine learning algorithms

Convenient results reporting and outputing

Integration with external applications
© 2002 Megaputer intelligence, Inc.
PolyAnalyst
machine learning algorithms
Your Knowledge Partner
© 2002 Megaputer intelligence, Inc.
TM
“Probably one the most impressive
characteristic of PolyAnalyst is the sheer
number of data mining tasks it can tackle.”
Mario Apicella
Technology Analyst
InfoWorld Test Center
July 3, 2000
© 2002 Megaputer intelligence, Inc.
Learning algorithms













Find Laws (SKAT algorithm)
Cluster (Localization of anomalies)
Find Dependencies (n-dimensional distributions)
Classify (Fuzzy logic modeling)
Decision Tree (Information Gain criterion)
PolyNet Predictor (GMDH-Neural Net hybrid)
Market Basket Analysis (Association rules)
Memory Based Reasoning (k-NN + GA)
Linear Regression (Stepwise and rule-enriched)
Discriminate (Unsupervised classification)
Summary Statistics (Data summarization)
Link Analysis (Visual correlation analysis)
Text Mining (Semantic text analysis)
© 2002 Megaputer intelligence, Inc.
Cluster (FC)

Identifies clusters of similar records

Selects best variables for clustering

Suggests the number of clusters

Separates clusters of records in new data sets for
further investigation - preprocessing for other
algorithms
© 2002 Megaputer intelligence, Inc.
Cluster
© 2002 Megaputer intelligence, Inc.
(continued)
Groups of similar records
Cluster



(continued)
Based on analyzing distributions in hypercubes of
all variables rather than on measuring distances
between points
Hence, independent of rescaling of axes variable
Finds only clusters actually present in data, on the
background of uniformly distributed cases
© 2002 Megaputer intelligence, Inc.
Classify (CL)




Fuzzy-logic based classification
The function of belonging modeled by either Find
Laws, PolyNet Predictor, or LR
Provides record scoring with Lift and Gain charts
used for visualization
Assigns records to one of two classes and furnishes
utilized classification rule
© 2002 Megaputer intelligence, Inc.
Classify
(continued)
PolyAnalyst Lift chart
illustrates an increase in the
response to a campaign
based on the discovered
model - instead of random
mailing
PolyAnalyst Gain chart
helps optimize the profit
obtained in a direct
marketing campaign
Targeted mailing
Mass mailing
Targeted mailing
Mass mailing
© 2002 Megaputer intelligence, Inc.
Decision Tree (DT)

Intuitively classifies cases to selected categories

Based on Information Gain splitting criteria

The fastest algorithm in PolyAnalyst

Scales linearly with increasing number of records
© 2002 Megaputer intelligence, Inc.
Decision Tree (continued)
Node characteristics
Classification tree
© 2002 Megaputer intelligence, Inc.
Decision Forest (DF)

The most efficient classification
algorithm for tasks with multiple
target categories

Transforms the task of categorizing
data records to N classes into the
problem of solving N tasks of
categorizing records to two classes

Develops the best collection of N
classification trees, with leaves
containing probabilities of classifying
records in the corresponding classes

Scales linearly with increasing
number of records
© 2002 Megaputer intelligence, Inc.
Link Analysis (LK)

Reveals pairs of correlated objects

Used in Fraud Detection, Text Analysis and other
correlation analysis tasks
© 2002 Megaputer intelligence, Inc.
Text Analysis (TA)

Extracts key concepts from natural language notes

Tags individual records with the main encountered
concepts

Recognizes synonyms and othe semantic relations

Can perform user-focused or unsupervised analysis

Integrates the analysis of text with the power of other
machine learning algorithms of PolyAnalyst

Facilitates categorization of textual documents
© 2002 Megaputer intelligence, Inc.
Text Analysis
© 2002 Megaputer intelligence, Inc.
(continued)
Basket Analysis (BA)

Is used in Retailing, Fraud Detection and Medicine

Identifies in transactional data groups of products
sold together well

Finds directed association rules for each of these
groups

Groups baskets containing similar sets of products

Characterized by


Support

Confidence

Improvement
Based on new mathematics:

works 10 to 50 times faster than traditional algorithms
© 2002 Megaputer intelligence, Inc.
Basket Analysis
(continued)
Groups of products
sold together well
Directed
Association Rules
© 2002 Megaputer intelligence, Inc.
Basket Analysis


(continued)
Works with both transactional and flat data format
Easily finds many-to-one rules
“I would like to continue working together with
Megaputer on other CTP customers’ projects
(mainly Swedish and Danish Banks ).”
-- Olof Goransson
Senior Data Consultant
CTP Skandinavien AB
© 2002 Megaputer intelligence, Inc.
Find Laws (FL)

Models relationships hidden in data

Presents discovered knowledge explicitly

Searches the space of all possible hypotheses
“The unique Find Laws algorithm along with an easy to
use interface made PolyAnalyst the only choice for our
environment.”
-- James Farkas, Senior Navigation Engineer, The Boeing Company
© 2002 Megaputer intelligence, Inc.
Find Laws

(continued)
FL is based on the Megaputer’s unique
Symbolic Knowledge Acquisition Technology (SKAT)

A good introduction to SKAT: PCAI magazine, January 99, p. 48-52
© 2002 Megaputer intelligence, Inc.
Find Dependencies (FD)

Determines most influential variables

Detects multi-dimensional dependencies

Predicts target variable in a table format

Used as preprocessing for FL
© 2002 Megaputer intelligence, Inc.
Find Dependencies (continued)
© 2002 Megaputer intelligence, Inc.
Predicted Sales per Employee
PolyNet Predictor (PN)

Predicts values of continuous attributes

Hybrid GMDH-Neural Network method

Works well with large amounts of data

The best architecture network is built automatically
© 2002 Megaputer intelligence, Inc.
Memory Based Reasoning (MB)

Performs classification to multiple categories

Based on identifying similar cases in the previous
history

Uses Genetic Algorithms to find the most suitable
metric for the problem
© 2002 Megaputer intelligence, Inc.
Discriminate (DS)

Determines what features of a selected data set
distinguish it from the rest of the data

Requires no target variable

Can be powered by

Find Laws

PolyNet Predictor

Linear Regression
© 2002 Megaputer intelligence, Inc.
Linear Regression (LR)

Incorporates categorical and yes/no variables in the
analysis correctly

Stepwise Linear Regression: only influential
variables included

Can be used as a preprocessing and benchmarking
module
© 2002 Megaputer intelligence, Inc.
PolyAnalyst
features in more detail
Your Knowledge Partner
© 2002 Megaputer intelligence, Inc.
TM
Data Analysis Project Workflow





Access data
Understand, clean and transform data
Run machine learning analysis
Visualize, report and share results
Integrate results in existing business process
© 2002 Megaputer intelligence, Inc.
Data Access
 ODBC-compliant
databases:
Oracle, DB2, Informix, Sybase, MS SQL Server, etc.
 Dedicated
access

IBM Visual Warehouse

Oracle Express
 OLE
DB (can do In-Place Data Mining)
 CSV
or DBF files
 Data
can be appended to the project
when necessary
© 2002 Megaputer intelligence, Inc.
Data cleansing and manipulation

SQL querying through OLE DB

Records selection according to multiple
criteria

Union, intersection, or complement of data
sets

Categorical values aggregation

Visual Drill-through

Exceptional records filtering

Split into n-tile percentage intervals

Random sampling
© 2002 Megaputer intelligence, Inc.
Visualization







Histograms
Line and scatter plots with zoom and drillthrough capabilities
Snake charts
Interactive 3D-charts
Interactive Rule-graphs with sliders for
visualizing multi-variable relations
Frequency charts for categorical, integer,
or yes/no variables
Lift and Gain charts for marketing
applications
© 2002 Megaputer intelligence, Inc.
Histograms and Frequencies
Histogram displays
distribution of numerical
variables
© 2002 Megaputer intelligence, Inc.
Frequencies chart displays
distribution of categorical and
yes/no variables
2D charts and Rule-graphs
Sliders help visualize
effects of other variables
in more than twodimensional models
© 2002 Megaputer intelligence, Inc.
The Find Laws model (red
line) for a product market
share dependence on the
price predicts a dramatic
change in the formula when
the product goes on
promotion
Snake-charts

“High”
“Low”
© 2002 Megaputer intelligence, Inc.
Quickly compare qualitatively several
datasets on all their attributes
Compared data sets
All variables
Interactive 3D charts

© 2002 Megaputer intelligence, Inc.
You can use mouse to rotate the 3D-cube
PolyAnalyst
integration features
Your Knowledge Partner
© 2002 Megaputer intelligence, Inc.
TM
Integration objectives




Use models to simply score data in various
external databases
Deliver models to external applications in the
format they understand - XML
Be able to analyze very large databases in their
entirety
Integrate dedicated machine learning components
in existing decision support systems
© 2002 Megaputer intelligence, Inc.
Applying models externally

PolyAnalyst can readily apply predictive models
directly to data in any external source through a
standard OLE DB protocol

PolyAnalyst can export models to XML (PMML)
format for their incorporation in external decision
support applications
© 2002 Megaputer intelligence, Inc.
Analyzing large databases
Traditional Data Mining
In-Place Data Mining
© 2002 Megaputer intelligence, Inc.
PolyAnalyst COM

A kit of COM-based Data Mining components
See DMReview magazine, January 2000, p. 42 and PCAI magazine, March 99, p. 16

Benefits






Develop new applications quickly and effortlessly
Incorporate third party components
Choose best components from different vendors
Extend functionality by adding new components
Cross-platform applications
Integration with most simple tools (Visual Basic)
© 2002 Megaputer intelligence, Inc.
PolyAnalyst COM


(continued)
Offers individual machine learning engines
Integration with external applications
Users see only the
familiar interface
enhanced by a few
new buttons
Hard analytical work is
performed by integrated
PolyAnalyst machine
learning components
behind the scenes
© 2002 Megaputer intelligence, Inc.
The main
program instructs
PolyAnalyst on
how to access the
stored data
PolyAnalyst platforms

Standalone system:




PolyAnalyst - Windows 9x/NT/2000/XP
PolyAnalyst Pro - Windows NT/2000P/XP Pro
PolyAnalyst XL - Add-ins for MS Excel
Client/Server system:


PolyAnalyst Knowledge Server - Windows NT
Client - Windows 9x/NT/2000 or OS/2
© 2002 Megaputer intelligence, Inc.
Customer
quotes
Your Knowledge Partner
© 2002 Megaputer intelligence, Inc.
TM
PolyAnalyst supports medical
projects at 3M
Timothy Nagle
Consulting Scientist
3M Corporation
St. Paul, MN, USA
© 2002 Megaputer intelligence, Inc.
“Analytical engines do an
excellent job of finding
relations amongst many
fields without overfitting.”
PolyAnalyst helps improving flight
control system at Boeing
James Farkas
Senior Navigation
Engineer
The Boeing Company
Kent, WA, USA
© 2002 Megaputer intelligence, Inc.
“PolyAnalyst provides
quick and easy access
for inexperienced users
to powerful modeling
tools.
PolyAnalyst facilitates marketing
research at Indiana University
Raymond Burke
E.W. Kelley Professor of BA
Kelley Business School
Indiana University
Bloomington, IN, USA
© 2002 Megaputer intelligence, Inc.
“PolyAnalyst provides a
unique and powerful set
of tools for data mining
applications, including
promotion response
analysis, customer
segmentation and
profiling, and crossselling analysis.”
PolyAnalyst helps medical research at
the University of Wisconsin-Madison
Prof. Roger L. Brown
Director of RDSU
University of
Wisconsin
Madison, WI, USA
© 2002 Megaputer intelligence, Inc.
“PolyAnalyst suite
enabled our researchers
to search their data for
rules and structure
while providing a
symbolic knowledge of
the structure, the detail
they needed.”
PolyAnalyst provides efficient machine
learning algorithms
Mario Apicella
Technology Analyst
InfoWorld Test Center
© 2002 Megaputer intelligence, Inc.
“PolyAnalyst focuses
more effectively on data
discovery than its
competition.”
PolyAnalyst
future developments
Your Knowledge Partner
© 2002 Megaputer intelligence, Inc.
TM
Future developments

Further support for OLE DB for DM


New machine learning algorithms





Nested tables
Time series analysis
Kohonen maps
Enhanced data import and manipulation
Visual development of workflow scripts
New push-button vertical applications
© 2002 Megaputer intelligence, Inc.
PolyAnalyst -- WebAnalyst

PolyAnalyst supports support visual project
development when used on top of a new Megaputer
web-enabled enterprise server, WebAnalyst
© 2002 Megaputer intelligence, Inc.
PolyAnalyst evaluation
Download a FREE evaluation copy of PolyAnalyst from
www.megaputer.com
and enjoy using it hands-on following the provided stepby-step lessons, or exploring your own data.
© 2002 Megaputer intelligence, Inc.
Any Questions?
Call Megaputer at
(812) 330-0110
or write
[email protected]
120 W Seventh Street, Suite 310
Bloomington, IN 47404 USA
Your Knowledge Partner
© 2002 Megaputer intelligence, Inc.
TM
Case 1:
Asea Skandia (Sweden)
© 2002 Megaputer intelligence, Inc.
Asea Skandia

Established 1907

Largest Swedish distributor of electrical equipment

About 1,400 employees and a turnover of SEK 5.1
billion

About ten thousand product names

Not good at CRM and DB marketing yet

Had only transactional data in a database
© 2002 Megaputer intelligence, Inc.
Groups of products offered
Home Appliances
Telecommunications
90 Cookers, cooker fans, microwave ovens
91 Fridges/Chillers/Freezers
92 Washing machines, dishwashers, dryers
93 Sauna unit, fans
94 Small appliances
48 Low current cable
49 Data and optical fiber cable
50 Network material
51 Local data networks
52 Power Supply
53 Signalling equipment
55 Distress signal systems
57 Telephony
58 Internal communication systems
60 Aerial equipment
62 Sound and time distribution systems
63 Safety and Security Systems
64 Service Alarm Systems
Lightning
17 IR, RF and Bus control systems
19 Light reg.. timers, plugs, CCE-con., car heaters
70 Interior light fittings
72 Industrial light fittings
73 Emergency luminaires
74 Spotlights and downlights, lighting tracks
75 Decorative interior light fittings
77 Exterior light fittings
79 Accessories and spare parts
80 Fluorescent lamps and other discharge lamps
81 Incandescent filament and halogen lamps
82 Special lamps
Ventilation and sheet metal
15 Fastening and fixings, protective equipment
16 Tools, implements, protective equipment & clothin
66 Ventilation
67 Sheet Metal for Buildings
© 2002 Megaputer intelligence, Inc.
Electrical Equipment
1 Power and control cables
2 Electrical installation, wiring and flexible cable
6 Material kits, cable protection, lightning equipment
7 Terminations, joints, cabinets and electrical tape
8 Contact crimping
9 Electric meters
11 Cable ladders, trays, trunking, cable trolleys
14 Conduit, boxes, glands, fire protection
15 Fastening and fixings, protective equipment
16 Tools, implements, protective equipment & clothin
18 Switch systems
20 Fuses with accessories
21 Miniature circuit breaker systems
22 Distribution board systems IP20-IP43
23 Distribution board systems IP43-IP65
25 Equipment boxes, equipment cabinets
26 Distribution board accessories
28 Switchgear components, capacitors, busbar
trunking
29 Connection terminals and marking materials
31 Motor, safety, load and MCCB breakers
32 Contactors and starters
35 Motors
37 Push switches
38 Sensors, monitors and regulators
40 Relays, time relays
42 Metering instruments
43 Spare parts for consumer goods
45 Programmable control system
85 Radiators and thermostats
87 Fan heaters
88 Water heaters and electric boilers
89 Heating cable
(continued)




Predicting cross-sell opportunities was possible
Closer cooperation with the client was necessary
Megaputer teamed with Cambridge Technology Partners (Sweden)
Data was disguised prior to the analysis
Asea Skandia
 Identified new opportunity
 Hired a consultant
 Helped aggregating
products in groups
 Incorporated results in
marketing activities
CTP
 Identified most suitable
solution provider
 Determined business
potential of the data
 Worked with the client
 Developed data
exploration strategy
 Collected available data
 Aggregated data in
product categories
 Presented Megaputer
results to the client
© 2002 Megaputer intelligence, Inc.
Megaputer
 Carried out Market Basket
Analysis
 Provided actionable
results to CTP
PolyAnalyst MBA


Works 10-50 times faster than traditional
Easily finds many-to-one rules
“I would like to continue working together with
Megaputer on other CTP customers’ projects
(mainly Swedish and Danish Banks ).”
-- Olof Goransson
Senior Data Consultant
CTP Skandinavien AB
© 2002 Megaputer intelligence, Inc.