Transcript Agenda enForce 2013

Big Data
HAVEn – HP’s Big Data Portfolio
Hans-Jürgen Fuks
Presales EMEA
25.11.2014
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Big Data Landscape
2
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Agenda
Big Data – Was steckt dahinter und welche Herausforderungen gibt es
HP‘s Lösungsportfolio mit der HP HAVEn Architektur
Use Cases für HAVEn
3
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
It’s New Sources of Data
4
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Every Enterprise action leaves a unique footprint
1% -5 % of the Digital Universe that actually is
being tagged and analyzed
$€¥
Customer
purchase
Download
a web page
Tweet
5
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Video chat
Sensors
Machine Data: Micro-transactions from machines
Security
Data Generated by
“Internet of Things”
Retail
Travel / logistics
Utilities
Automotive
2010
6
2015
McKinsey : Big Data – The next frontier for
innovation, competition and productivity
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
•
•
•
•
•
•
•
•
•
Medical equipment
Utility networks and meters
Car and truck fleets
Security sensors
Home automation
Touch-streams from games
Drones
Pollution sensors
Transport sensors
Human Information: Meaning from interaction
• Faces
• Transcripts
• Words
• Places
• People
• Meaning
• Logos
• License Plates
Images
Video
Audio
• Sentiments
• Complaints
• Numbers
• Meaning
• Sensitive info
• Words
• Huge volumes
Social media
7
Email
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Documents
The challenges of information
Variety, velocity, volume, time to value
90%
75%
48%
86%
of digital content
created by 2015 will
be unstructured data
types ¹
of currently deployed
data warehouses will
Worldwide
information volume
growth of digital
content¹
of corporations
cannot deliver the
right information, at
right time to support
enterprise outcomes ³
fail to include
unstructured data
support (by 2016)2 to
meet new information
velocity and
complexity of demands
¹Source: IDC Predictions 2012: Competing for 2020
8
²Source: Gartner - The State of Data Warehousing in 2012
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
³Source: Coleman Parkes Survey Nov 2012
Why does the business love big data?
+5%
Annual labor productivity growth percentage
2.82%
Productivity
+6%
2.70%
1.49%
Mainframe
Mini-computers and PCs
Internet and Web 1.0
Profitability
9
Source : Harvard Business Review,
October 2012
2.50%
Mobility &
Web 2.0
Source : McKinsey : Big Data –
The next frontier for innovation, competition and productivity
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
The changing Big Data landscape
Annual
Growth
~100%
Machine Data
Human Information
90% of Information
Business
Data
~10%
10% of Information
10
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Move from Reactive to Predictive – And Beyond
Right information. Right person. Right time.
Foresight –
In Flight
Predictive: What will happen?
Prescriptive: How to make it happen?
information
Insight –
Better,
Better,
Right Time
Right
Time
Decisions
Active
Inactive
Decisions
information
information
Real-Time Descriptive:
What is happening now?
11
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Hindsight –
Descriptive: What happened?
Diagnostic: Why did it happen?
Big Data landscape
Machine Data
Human Information
Business
Data
Autonomy
Vertica
Enterprise Security
Hadoop
12
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Welcome to the world of HP HAVEn  Value
HAVEn
Approaching Big Data with an intelligent software architecture
Consume
Dashboards & alerts
Business intelligence
Packaged apps
Understand
Analyze
Hadoop/
Autonomy
HDFS
IDOL
Catalogue massive
volumes of
distributed data
Process and
index all
information
Volume
Social media
13
Video
Email
Vertica
Enterprise
Analyze at
extreme scale
in real-time
Collect and
unify machine
data
Security
Velocity
Variety
Audio
Custom apps
Texts
Mobile
Powering
HP Software
+ your apps
Vulnerability
Transactional
Documents
data
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
nApps
IT/OT
Search engine
Images
Legacy BI solutions were built for a different world
Yesterday’s data warehouse and
analytic infrastructure
•
•
•
•
•
•
14
Proprietary, so expensive
Centralized, monolithic
Siloed
Procedural
Batch, not real time
Slow
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Hadoop
15
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Intel on Hadoop
"Within a couple of years, Hadoop will be the number
one application. It will be running on more servers than
any other single app. It will be more common for
enterprise IT than their ERP system,"
Diane Bryant, senior VP and general manager of Intels data center group,
at the Intel Developer Forum in San Francisco September, 2014
16
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Apache Hadoop is a software framework
Two Core HADOOP system components
An Open source Linux-based platform
for data storage and data processing
that is…
 Scalable
 Fault tolerant
 Distributed
It has the flexibility to store
and mine any type of data
• Query previously inaccessible
structured and unstructured data
• Not bound by single schema
17
Storage for Big Data
Data processing
Hadoop
Distributed File
System (HDFS)
MapReduce
Self-healing,
high bandwidth
clustered storage
Distributed
Computing
Framework
It excels at processing
complex data
• Scale-out architecture divides
workloads across multiple nodes
• Flexible file system eliminates
ETL bottlenecks
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
It scales
economically
• Deployable on commodity
hardware
• Open source platform guards
against vendor lock
HP and Hadoop 2.0
Being ready for the future of Hadoop
18
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Hadoop ecosystem
Map Reduce
HDFS
Pig
YARN
ZooKeeper
Sqoop
19
Storm
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Impala
Core HP Hadoop solutions architecture
System management
Hadoop distribution
HP Insight CMU
• Provisioning
• Monitoring
• Management
Linux OS
HP Networking Top of Rack switches
HP Gen8 ProLiant Servers
20
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Analytics Database
Autonomy
21
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Autonomy IDOL (Intelligent Data Object Layer)
HP Autonomy IDOL Platform
High-Performance Human Information Processing
400+ connectors
Seamlessly access virtually any enterprise content repository including file systems, email or knowledge
bases
Over 500 functions
Leverage the power of functions like sentiment, categorization & clustering to deliver intelligence, insight
1,000+ file types
Process virtually any file-type such as text (email, tweet, document), audio, video and even people
profiles & behavior
Distributable architecture
Achieve big data scalability & high performance with distributable ingest & query architecture
All data types, all content repositories - unmatched understanding
22
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
*IDOL - Intelligent Data Operating Layer
HP Autonomy is a leader in 2014 Gartner Magic
Quadrant for Enterprise Search
Gartner Enterprise Search
Magic Quadrant 2014
•
•
•
23
Handles searches driven by queries that
include surmised or contextual information
exceptionally well
HP Autonomy's sophistication and
extensibility enable it to tackle the most
demanding use cases
A very long list of data connector types
enables almost any data source to be
discovered and indexed
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
IDOL Insight – analyze and gain insights on all sources of customer and business information
– Web, documents, databases, social media, news
Multi-channel analytics across all contact points
Brand
reputation
mgmt.
Cross
channel
optimization
Next-gen
speech
analytics
Voice
of the
customer
Customer
experience
analytics
Customer
interaction
survey
Operational
efficiency
First
contact
resolution
Social
media
monitoring
Survey
Video
Apps
HP IDOL Insight
IDOL
Web
24
News
Email
Blog
Voice
IDOL
Review
Extensible architecture to ingest new public, social, enterprise data sources
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Human information ~ unstructured data ?
Meaning based technology
Human Information is made up of ideas, is diverse, and has context
Social Media
Email
Documents
Video
Audio
Texts
Mobile
Search Engine
Example : sentiment in a tweet, smile in a
picture, somebody laughing in a video or
phone call… -> happiness / positive
sentiment
Images
90% of data is human information locked as unstructured content – difficult to analyze, impossible to summarize or gain insights
Unstructured content is especially relevant for Market & Customer Insights
25
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
26
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Vertica
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Vertica Analytics platform 7.0
High-performance data analytics platform purpose-built for big data
Blazing fast analytics
Gain insight into your data in near-real time by running queries 50x -1,000x faster than legacy products
Massive scalability
Infinitely scale your solution by adding an unlimited number of industry-standard servers
Open architecture
Protect and embrace your investment in hardware and software with built-in support for Hadoop, R, and a
range of ETL and BI tools
Optimized data storage
Store 10x-30x more data per server than row databases with patented columnar compression
Load & analyze growing forms of semi-structured data
Quickly and easily load, explore, analyze emerging and rapidly growing forms of semi-structured data.
Easy Set-Up and Administration
Get to market quickly with your analytics initiatives at low cost of administration and maintenance
28
Speed, scalability, and openness at lower TCO
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Vertica is named : „Best Columnar Database”
Here are the winners of the
2014 DBTA Readers'
Choice Awards for
Best Columnar Database
Winner: HP Vertica
Finalists:
SAP IQ
InfiniDB Source: http://www.dbta.com/Editorial/Trends-and-Applications/Best-Columnar-Database-98437.aspx
29
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
High-Performance DBMS
With A Unique Combination Of Innovations
Leverages existing
BI, ETL, Hadoop /
MapReduce and
OLTP investments
No disk I/O
bottleneck;
simultaneously
load & query
Built-in redundancy
that also speeds up
queries
Standard SQL
Interface
Column
Orientation
50x – 200x faster
performance
>
High scalability from
TBs to PBs
>
Simple integration with
existing ETL and BI
solutions
>
Superior performance
on off-the-shelf
hardware
>
Ultimate deployment
flexibility
High
Availability
Auto
Database
Design
MPP
Massively Parallel Advanced
Compression
Processing
30
>
Automatic
setup,
optimization,
and DB
management
Native DB-aware
Up to 90% space
clustering on lowreduction using 15+
cost x86 Linux nodes
algorithms
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Evolution of EDW and Vertica Migration
700+ Data Marts
across HP
Data
Mart
Data
Mart
Data
Mart
Data
Mart
Data
Mart
Data
Mart
Data
Mart
Data
Mart
Data
Mart
Data
Mart
Data
Mart
Data
Mart
Data
Mart
EDW
Finance
Data
Mart
Sales & Marketing
Data
Mart
ES Service
Data
Mart
1990s-2004
31
Formation of multi
petabyte EDW from
700+ Data Marts
Acquisition of Vertica
Initial Platform and +
Analytics POCs
Vertica Enablement of
Business Capabilities
leveraging the EDW
and/or Hadoop
Sales Pipeline
Analytics
Vertica Platform
POC
Channel
Analytics
Sales
Pipeline
Analytics
Customer 360
Analytics
EG Service
Supply Chain
Software
Vertica
Channel
Analytics
Customer
360
Analytics
EDW
Grey
Market
Analytics
Hadoop
Data Lake
Grey Marketing
Analytics
Analytics POC
EDW
GRC/Core IB
2005-2010
HP/Vertica Big Data, EDW,
and Analytics Convergence
Hadoop
Data Lake
2011-2012
2013-2014
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
2015 and Beyond…
Enterprise Security - ArcSight
32
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Security
Performance Suite
HP Enterprise Security
HP Security Performance Suite Pillars
Application
Security
33
Security
Intelligence
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Network
Security
HP ArcSight SIEM
HP ArcSight Universal log management platform
High-performance universal log management to consolidate machine data across IT
315+ connectors
Collect, normalize, and categorize machine data such as logs, events, and flows from any device, any
time, anywhere from any vendor
Search over 1,000,000 events per second
The unified machine data through filtering and parsing is enriched with rich metadata, which allows you to search
machine data through simple text-based keywords without the need of domain expertise
Store years’ worth of data
The unified data is stored through high compression ratio in any of your existing storage formats,
eliminating the need for expensive databases and DBAs
Analytics & intelligence
Built-in content packs, algorithms, rules, and the unified machine data help you deploy IT
security, IT operations, IT GRC, and log analytics
Collect, store, and analyze any machine data across IT
34
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
n-Apps
35
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HAVEn and BIG DATA apps from partners - examples
36
Analytical
security
Commercial
insurance risk
assessment
Ecommerce
predictive
analytics
Healthcare
analytics
Insider
threat
analysis
Service
analytics for
manufacturing
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HAVEn
Two (out of many) Use Case Examples
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Creating a 360o view of your customers
$€
¥
Transactions
Tweets
Facebook
Texts
Video
Searches
Public
Records
eMail
Audio
+
=
Geospatial
Health
Documents
Mobile
Images
44% of companies use
customer data
38
70% willing to share
personal information
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Micro segmented offerings and
services
Customer 360 ̊ “I just became a father”
Understanding customer life events for personalized commercial actions
39
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Solution Architecture
Catalog massive
volumes of distributed
data
Process and index all
information
Analyze at extreme
scale
in real time
Collect and unify
machine data
Enhanced CRM
application with
commercial actions
Life event driven
business rules
CRM Data
Powering
your apps
Graph/network
metrics, Linear
scorecard model
Social Media Data
Provider API
Hadoop/ HDFS
40
Qualitative Data
Enrichment
Autonomy IDOL
Pattern matching
and text
processing
Vertica
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Social media
CRM Data
Enterprise security
nApps
Big Data &
Security Analytics
FSI Use Case
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
“Simple” audit question
?
Internal /
External
Audit
Run separate
queries on each
system
Who has accessed the
customer’s account in last 2
months when he/she is not in a
branch or using bank’s services
via web, mobile, call center, etc?
60 TB
Security
100 TB
RSA NetWitness
IT Security
Team
20 TB
Response in 12 Weeks
Operations
Process &
Correlate
Data
80 TB
Guardium
150 TB
42
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Inhouse Apps. (Transactions)
“Simple” audit question using HAVEn
… has reduced this query response time
from 40 hours to 55 Secs.
EMEA Bank, running Big Data Security Analytics
on #1 Big Data Platform HP HAVEn with HP
Vertica and HP ArcSight
43
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
%97 less
OPEX
Before HP HAVEn
With HP HAVEn
Solution Architecture
Catalog massive
volumes of distributed
data
Process and index all
information
Analyze at extreme
scale in real time
Collect and unify
machine data
Advanced
Visualization
SQL Functions on
Vertica
All system logs
(data in rest &
streaming)
Data / Log
Correlation on
Vertica
IDOL Connectors &
Hadoop/ HDFS
RSA
NetWitness
44
Autonomy IDOL
HP BSM
Guardium
Complex Event
Management Sys.
ArcSight SIEM
Logger / Secuity
Event Logs
Qualitative Data
Enrichment
Log Collectors &
Parser
Powering your apps
Vertica
Enterprise security
Inhouse Apps.
(Transactions)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CRM Data
(Phase-2)
nApps
HP
Autonomy
(Phase-2)
HP Big Data Discovery Experience services
Rapid, low-risk, secure path to Big Data value
Technologies
Autonomy,
Vertica, Hadoop…
Services
Discovery Workshop
One to two-day workshop to align business and
IT and determine priorities
Discovery Diagnostics
Two-week assessment for a Discovery
Experience roadmap and value justification
Discovery Experience
A private and secure Big Data “test-drive”
environment
Discovery Production
Transform your business processes with
validated use cases to realize insights
45
Expertise
HP data
scientists,
technology
experts, industry
SMEs
Guided process
Accelerate timeto-value with
guided standard
process
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Discovery
services
Platform
HP Virtual Private
Cloud
Use case library
Industry and
business function
examples
HP is leading in Big Data innovations
Fastest Time to Value—Purpose built for Big Data scale and performance
Delivering innovative
Big Data solutions
Reference
Architecture for
Vertica
Delivering simplicity with
an end-to-end approach
AppSystem for
Autonomy
ConvergedSystem
500/900 for
SAP HANA
Delivering Big Data
value for you—today
Reference
Architecture for
Hadoop
ConvergedSystem
300 for for
Microsoft Analytics
Platform
HP solutions deliver more choice to meet specific workloads , data volumes, and variety
versus our competitors’ “one-size-fits-all” approach
46
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP Lab’s hardware research for big data
photonics for faster
interconnect
specialized
big data task
processors
memristers for
universal main
memory
47
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Questions ?
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.