Transcript Intellectual Property - DAMA New York

Putting It All Together:
Trends in Business Intelligence
Claudia Imhoff, PhD
Intelligent Solutions, Inc.
[email protected]
www.intelsols.com
Blog: http://www.b-eye-network.com/blogs/imhoff/
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
Claudia Imhoff
President and Founder
Intelligent Solutions, Inc.
A thought leader, visionary, and practitioner in the
rapidly growing fields of business intelligence and
customer focused-strategy – Claudia Imhoff, Ph.D., is
an internationally recognized expert on analytical CRM,
business intelligence, and the infrastructure to support
these initiatives – the Corporate Information Factory
(CIF). Dr. Imhoff has co-authored five highly-regarded
and popular books on these subjects and writes
monthly columns (totaling more than 100) for technical
and business magazines.
Email: [email protected]
Phone: 303-444-6650
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
2
Putting It All Together
 Going Beyond Traditional BI – Operational BI Takes
the Stage
 Data Warehouse Appliances and Analytic
Databases – Making Life Simpler
 BI Software as a Service – Feeling SaaS-y?
 Open Source BI – Free Software Anyone?
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
3
The Three Levels of Business
Intelligence
Strategic BI
timeframe ~ months
Tactical BI
timeframe ~ days or weeks
Operational BI
timeframe is intra-day
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
4
The Three Levels of Business
Intelligence
Strategic BI
Tactical BI
Operational BI
Business
focus
Achieve long-term
business goals
Manage tactical initiatives
to
achieve strategic goals
Monitor & optimize
operational business
processes
Primary
users
Executives &
business analysts
Business analysts,
& LOB managers
LOB managers, operational
users &
operational processes
Timeframe
Months
to years
Days to weeks
to months
Intra-day
to daily
Data
Historical
data
Historical
data
Real-time, low-latency,
& historical data
Paradigm Shift
Mode of
operation
User driven
Data centric
User driven
Data centric
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
Event driven
Process centric
5
What is Operational BI*
A set of services, applications and technologies
for monitoring, reporting on, analyzing and
managing the business performance of an
organization’s daily business operations
*From research study. “Embedded BI”, written by Colin White and Judy Davis, www.B-EYE-Research.com
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
6
Operational BI – Answers to Day-toDay Business Questions
 picked
 packed
What is my customer’s order status? What can
I offer based on customer’s life-time value?
 shipped
 invoiced
What is my current inventory level world wide?
Is it sufficient to meet demands?
Yield
What is my production yield right now?
Am I at par with acceptable standards?
Can I afford to make this move
at current margin rates?
Operational BI
 Helps front-line workers
make immediate business decisions
to squeeze out inefficiencies.
New Data Needs
 Information on demand
 Real-time + historical data
 Access to SAP, Siebel, Oracle and
BI results
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
7
Real Time Decision-Making*
 Operational BI optimizes time latency between
when a business event occurs and when an
appropriate action is taken
 The goal – to “right-size” the decision-making cycle
 Compressing time lag between knowing what is
happening and taking action based on that knowledge
 Real-time must consider potential trade-off between timeto-action and business value of actions
* From “Right-Time Business Intelligence: Optimizing the Business Decision Cycle” By Judy Davis, www. BEYE-Network.com Research paper
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
8
Impact on BI Environment
 History of BI
 Extract usable information from operational systems
 Users, technologies, processes, procedures – all
independent of operations
 Now what?
 Impact on BI environment is significant
 Increase in number of users, volume of data, and faster
performance
 Operational BI – MUST be integrated into the operational
environment
 Requires understanding of operational systems, processes,
procedures, workflows, personnel
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
9
Impact on BI Environment
 Numbers of users increase significantly
 Traditional BI rarely supported a few hundred, maybe a
thousand or so users
 Opening BI up to operational personnel means ramping
up into tens of thousands of users
 These users have very different interface requirements
 Means BI implementers must rethink how BI is delivered to
business users
 Means tighter and faster connectivity of enterprise
decision support environment to rest of the company.
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
10
Impact on BI Environment
 Volumes of data increase substantially
 Detailed intraday snapshots of data are loaded or tricklefed into data warehouses
 Tens of terabytes to hundreds of terabytes are not
unusual storage requirements for operational BI
 Scalability now a mandatory requirement in any BI
technology
 Whether in processing and integration of data, storage of massive
volumes, or retrieval of query responses
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
11
Impact on BI Environment
 Faster performance
 Query performance must mimic or emulate response
times in operational systems
 Sub-second to just a few seconds to return data from a query.
 Ability to prioritize queries not only according to their
importance but also their response requirements is
mandatory success criterion
 This last feature has stumped many BI implementers and BI
vendors
 Must have ability to handle mixed work load gracefully
and simultaneously
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
12
Getting Started – Assess
Reality
 First step – perform honest assessment of existing
data delivery capabilities – available technologies,
maturity of the BI architecture, existing personnel,
etc.
 Combine these with solid understanding of business
requirements for operational BI data
 Important to understand which weaknesses
discovered in assessment will be exaggerated as
you speed up the enterprise
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
13
Operational BI Requirements
 Continuous availability of operational data and BI
results
 Current information from operational systems
 Integrated with BI data on demand
 Minimal impact on operational systems performance
 Presented in a proactive manner
 Make decisions – act on information presented
 Easy to understand and use
 Dynamic modeling
 Ability to change business rules on the fly
 Show different set of metrics depending on situation
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
14
Picking a Project
 Look for workflow activities that have significant
impact on costs or revenues
 Bottlenecks today that can be made more efficient
through use of operational BI
 Don’t make big changes to operational processes
 Just speed up or make more efficient processes you
already have in place
 You will have to retrain personnel and retool SOPs
 Project managers may not realize operational BI
application has ramifications beyond project’s
immediate boundaries
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
15
Putting It All Together
 Going Beyond Traditional BI – Operational BI Takes
the Stage
 Data Warehouse Appliances and Analytic
Databases – Making Life Simpler
 BI Software as a Service – Feeling SaaS-y?
 Open Source BI – Free Software Anyone?
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
16
Data Warehouse Appliances
 BI and data warehousing technologies continue to
evolve and innovate
 Produce more efficient & cost effective ways to deliver BI
 Latest innovations are DW and BI appliances
 Definition of an appliance*




One purpose
One package
One installation
One vendor
* From the B–EYE-Research.com paper titled “Data Warehouse Appliances: Evolution or Revolution?”
by Colin White, and Richard Hackathorn
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
17
Data Warehouse Appliances
 All-in-one box that provides a hardware server
preconfigured with all software components
 Designed for a specific purpose – supporting data
warehouse processing
 Offers ease of use, simplicity, and compatibility – tested,
ordered and delivered as a single system
 Simple to understand even though mechanism may be
complex
 Low cost in terms of TCO
 High performance in achieving its purpose
 Single point of service provided by single vendor
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
18
Data Warehouse Appliances
 Cost effective solution
 TCO of a data warehouse appliance is lower because
cost of hardware and software is cheaper
 Also because simplicity and ease of reduces installation,
administration and support cots
 Improved usability of a data warehouse appliance means
projects can be developed and deployed faster
 Includes popular BI capabilities
 Interactive dashboards, analysis, reports, alerting, and
data integration
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
19
Sweet Spot for Data Warehouse
Appliances
Size of Data
Multiple
Terabytes
Data Warehouse
Appliances
Mega- to
Gigabytes
Focused Purpose
Specialized
Databases
(e.g., Teradata,
IBM)
Any database
vendor
Complexity of Workload
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
Mixed Purpose
20
Data Warehouse Appliances
 Pros
 Cons
 Immediate visibility &
interaction into
business performance
 Non-disruptive to
existing infrastructure
 Faster deployment
 Low maintenance –
black box
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
 Still some opposition to
use of appliances by IT
departments
 Loss of “control” over
moving parts
 DW and BI appliance
scalability
 Customization to fit
each company’s needs
21
Sample Data Warehouse
Appliance Vendors








Netezza
Teradata
DATAllegro (now Microsoft)
Sun + Green Plum
Sun + Vertica
Sun + ParAccel
Sun + Kognitio
IBM InfoSphere Warehouse
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
22
Role of Appliances in BI SaaS
 Many data warehouse appliance and BI SaaS
vendors are forming partnerships
 Gives SaaS vendors scalability, reliability, performance
 Gives appliance vendors applications, new markets,
greater exposure
 Gives customers more confidence that solution is on solid
technological footing




Performance
Support for multi-tenancy
Scalability
Applications
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
23
Analytic Databases
 Many are Massive Parallel Processing (MPP)
 Can use commodity hardware
 Many have column-based data organization
 Limit I/O by putting similar data together – reduces reads
to only columns needed for query
 Single data type per column allows for significant
compression
 Data compression
 Compression can be optimized for particular data types
 CPU is not the bottleneck, only I/O is
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
24
Analytic Databases
 Built-in intelligence
 Allows decompression of only data that must be for query
resolution and ignore all others
 Is major factor in overall improved performance
 Load times remain constant regardless of table size
 Should also have query times that remain constant
regardless of table size
 Bottom line – technology must be seamlessly scalable
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
25
Analytic Databases
 Many new vendors on the market (sample):








Green Plum
Vertica (Michael Stonebraker*)
ParAccel (Barry Zane**)
Dataupia (Foster Hinshaw**)
InfoBright (Warsaw University)
Aster Data (Stanford University)
illuminate (Former Synerra Systems founders)
One well-established vendor: Sybase IQ since 1993
 Most are column based, MPP, shared nothing
architectures (not all though) * Ingres and Illustra founder
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
** Netezza founders
26
Analytic Databases – Really
Fast: TPC-H 1 TB
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
27
Analytic Databases – Really
Fast and Really Inexpensive
Solution
Pricing model
Price/unit
1 TB solution
Remarks
Vertica
Data Volume
(raw)
$ 100,000/TB
$ 200,000,-
Based on 5 nodes,
$ 20,000,- each
ParAccel
Node
$ 40,000,(+$10,000/TB)
$ 310,000,-
Based on 5 nodes,
$ 20,000,- each
ParAccel
Data Volume
(raw)
$ 1,000,-/GB
$ 1,250,000,-
From TPC-H
publication
InfoBright
Data Volume
(raw)
$ 40,000,-/TB
$ 140,000,-
Based on 5 nodes,
$ 20,000,- each
Dataupia
Node
$ 19,500/2TB
$ 19,500,-
You can not buy a
1 TB Satori server
ExaSol
Data Volume
(active)
$ 675 - $1,750
per GB
$ 940,000,-
From TPC-H
publication
Graph compliments of Jos van Dongen, Tholis Consulting, NL. Numbers are estimated.
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
28
Analytic Databases
 Pros




Excellent performance
Very cost-effective
Low maintenance
Partnering with
hardware vendors (DW
appliance)
 Cons
 Many are small
companies
 May not handle mixed
work load well
 New (unknown)
technology for IT
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
29
Putting It All Together
 Going Beyond Traditional BI – Operational BI Takes
the Stage
 Data Warehouse Appliances and Analytic
Databases – Making Life Simpler
 BI Software as a Service – Feeling SaaS-y?
 Open Source BI – Free Software Anyone?
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
30
BI Delivery Models
 There are two BI delivery models today
 On-premises – traditional model
 Software as a Service (SaaS)
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
31
On-premises – Traditional
Model
 Internal IT is responsible for entire environment
from first project
 Find excess capacity on machines
 Upgrade memory on existing machine for usage
 Leverage installed end user access tools
 Buy smaller platforms that can scale
 Migrate to bigger box when necessary
 Use smaller box for data mart(s)
 Look into data warehouse appliances for very large,
focused BI analytics
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
32
Software as a Service (SaaS)
 Characteristics*
 Secure, flexible, and efficient business processes &
workflows
 Service level agreements
 Value-added business services such as analytics & best
practices
 Extensive use of service-oriented architecture (SOA) to
enable scaling, configurability, and integration
 Subscription monitoring & usage-based billing
* From www.sandhill.com, “Get Ready for SaaS 2.0” by Bill McNee, Saugatuck Technology, May 8. 2006
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
33
Advantages for SaaS Vendors
 Vendors support only one platform and one version
of the application
 No need to support multiple operating systems, platforms,
and older versions of the software
 Decreases development costs significantly
 SaaS gives vendor great visibility into how their
customers are actually using their software
 See every move, every feature, every function used by
customers
 Gives vendor great intelligence on how to build a better
product based on actual usage
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
34
Advantages for SaaS Vendors
 SaaS model gives vendor a predictable cash flow
 Subscription model is reliable for cash flow estimation
 Improves start-up estimations and growth track
 Vendors don’t get trapped in “feature bloat”
 No need to keep adding feature after feature to get
customers to buy new versions
 Create only features that are needed based on actual
customer usage
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
35
Disadvantages for SaaS
Vendors
 SaaS produces lower revenues at first than
traditional vendor models
 Must attain critical mass of subscribing customers
 Vendor must have enough funding to tide them over
 More time is needed to ramp up to mature status
 Higher customer set up costs
 Traditional vendor model – send customer a CD
 SaaS vendors must allocate space, set up customer
support, etc.
 SaaS vendor becomes IT support for their customers
(higher costs for customer service?)
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
36
Disadvantages to SaaS
Vendors
 Customers still need ability to integrate SaaS
application data with other enterprise data
 Need mechanism to export data out of SaaS environment
 Who supplies integration of SaaS data with customer’s
other data?
 If customer is not SOA-compliant yet, what does this
mean to SaaS model?
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
37
Reasons for Adoption: Ease of
Deployment
 This is the SaaS model’s greatest advantage




No installation of hardware
No installation of software
No administration of new versions of either
No need for IT expertise in the tool or application
 Set up consists of getting a login and password set
up for the business users
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
38
Reasons for Adoption: More
Flexibility for Evolving Needs
 Perhaps…
 You can certainly change SaaS vendors quite easily
 If you are unhappy with one vendor, changing to another one is
about as easy as getting a new login and password
 You can influence the direction and R & D of the current
SaaS vendor
 You can easily add or subtract users
 You can easily add or subtract functionality
 It may not be as easy to customize the SaaS offering to
your specific needs
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
39
Reasons for Adoption: Not
Locked into Long Licenses
 True
 Great advantage in BI world where technology is moving
very fast
 Can switch from one SaaS vendor to another
 But watch for cancellation fees
 And make sure you know what the subscription fee
is based on
 Reduction or addition of users may be cross price break
threshold
 Salesforce.com model is typical
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
40
Considerations for BI SaaS
 SaaS – good at supporting particular types of users
 Highly mobile work force
 Field sales personnel
 Product support specialists at customer sites
 Telecommuters
 Highly geographically disbursed workforce
 International enterprises
 Non-office workers (virtual offices)
 Customer or partners worldwide
 Must include support for various mobile devices
 Phones, mobile PCs, handheld devices, PDAs, etc.
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
41
Considerations for BI SaaS
 Ensuring quality of delivered environment




Correct mappings, verified data lineage, transformations
Sufficient data quality processing
Data represented in analytic engine correctly
Appropriate presentation of information, e.g.,
personalized dashboards
 Scalability of environment
 Data volumes – small beginnings to 100’s of terabytes?
 From a few users to 1000’s
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
42
Considerations for BI SaaS
 Performance
 From simple to complex queries
 Response times – operational to strategic BI
 Getting right data to right people at right time
 Open Architecture
 Compliance with best practices?
 Non-proprietary infrastructures?
 Integration with existing infrastructure?
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
43
Considerations for BI SaaS
 What does SaaS vendor bring to the table?
 Best practices
 Quick start BI components like a library of reports,
analytic calculations, KPIs, etc.
 Industry-specific knowledge
 Horizontal business knowledge
 Support for all employees in all levels of enterprise
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
44
BI SaaS
 Pros
 Fixed cost –
subscription model
 Fixed time
 Flexibility /
customization
 Single vendor
responsible for entire
environment
 Quick ramp up
 Cons
 New paradigm –
nervousness?
 Can a company maintain
its uniqueness?
 Loss of “control” over
data, quality, access
 Vendor’s timeliness in
response to changes
 Vendor’s industry
knowledge
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
45
Sample BI SaaS Vendors




LucidEra
Xactly
Eyeris
PivotLink (was
SeaTab)
 Oco
 On Demand IQ







Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
Actuate
Cognos
SAP ERP
SAS
Business Objects
SalesForce.com
Dimensional Insight
46
Putting It All Together
 Going Beyond Traditional BI – Operational BI Takes
the Stage
 Data Warehouse Appliances and Analytic
Databases – Making Life Simpler
 BI Software as a Service – Feeling SaaS-y?
 Open Source BI – Free Software Anyone?
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
47
Open Source Vendors Face
Questions
 The myths and doubts:




Is there support for open source BI?
How many people are really using it?
Will it scale? Is it considered enterprise class?
Is it only for developers?
 These are being overcome…
 According to Aberdeen*, 25% of survey respondents will adopt open
source BI in next 12 to 24 months
 CEOs agree – open source is a worldwide growth story in 2008**
 First nine months of 2007, open source deal flows doubled each
quarter***
 Sun’s commitment to open source - $1 Billion for MySQL
* Source: “The TCO of Business Intelligence – Open Source Takes on Traditional BI”, www.aberdeen.com
** Source: www.OpenSolutionsAlliance.org
*** Source: www.the451group.com
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
48
Why Use Open Source?
 Price!
 Open Source software can be downloaded, installed and
operated free of charge
 Return on investment (ROI) of Open Source model is
good
 Open Source software is reliable and scalable
 Just look at the Internet – its infrastructure relies heavily
on Open Source software
 Wall Street – 8 of top 10 banks use Open Source
technologies
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
49
Why Use Open Source?
 Open Source community grown to significant size
 Millions of developers contributing everyday
 Cost of development is externalized
 Ability to adapt or customize
 Many companies don’t want or need feature bloat
 Easy integration and performance
 New tools for building browser based reports and
dashboards accessible to more people
 Ad-hoc report designers with drag and drop capabilities
 Enhanced wizards for custom data source
implementation
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
50
Open Source Offerings
 BI projects won’t be consumed by license fees
 No huge up-front fees to justify before commencing
a project
 “Safe choice”




Many successful deployments
Professional services experts to work with you
Professional, public training
Support from the experts – project leaders and sponsor
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
51
Open Source BI Vendors
 Proponents of open source BI point out low cost of
entry, flexibility and variety of applications available
 Opponents believe open source BI lacks
functionality needed to succeed right now
 Regardless, don’t be fooled by “numbers of
downloads”…
 Vendors* – Actuate, JasperSoft, Jpivot, Mondrian,
Pentaho, SpadoBI
* For a more complete list, go to http://www.manageability.org/blog/stuff/open-source-java-business-intelligence
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
52
Open Source ETL
 Open Source ETL
 ETL alternative follows industry standards for ease of use,
quick deployment, and fit into company’s needs
 Users download open source ETL code and get started
 Can collaborate with open source community to share
integrations and extend tool’s functionality
 Will probably need to buy support and services from
company’s professional services and support
 Sample Vendors*: Talend, JitterBug, KETL, Pentaho,
Octopus, CloverETL
* For a more complete list, go to http://www.manageability.org/blog/stuff/open-source-etl
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
53
Open Source
 Pros
 Cons
 Cost effective
 Easy to install and
deploy
 Large development
community
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
 How do they make
money?
 Many are small
companies
 Some offerings not
truly open source
54
Right Place at Right Time –
Get Going!
 Once you have your ducks in a row, you are ready
to create the proper environment
 Create an infrastructure that can withstand change –
you’ll need it
 Pick technologies that support that infrastructure and
move you toward SOA compliance
 Constantly monitor business community usage
 Measure ROI and publish it
 IT infrastructure should be to information as a power
grid is to electricity
 Information should flow as freely as electricity does
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved
55
Questions
Claudia Imhoff, Ph.D.
President
Intelligent Solutions, Inc.
www.IntelSols.com
[email protected]
Copyright © 2008, Intelligent Solutions, Inc., All Rights Reserved