Transcript *** 1 - Eventworld.cz

Huawei Big Data Storage N9000
Jan Stibor
Storage & Server Product Manager
1
Definition of Big Data
2
HUAWEI Big Data Storage Solution
3
Application Scenarios of HUAWEI Solution
2
Era of Big Data Is Coming
2.9 million emails per second globally
250 million pictures
per day
Videos of 28,800 hours per day
24 PB data processed per day
200 PB per safe city
6.3 million orders per day
50 million messages per day
More than 1 PB per 3D movie
By 2015, data amount will reach 8 ZB with the valued of $16.9 billion.
Page 3
Page 3
What Is Big Data?
Mass data
Exponential growth of unstructured data
Quick data flow and dynamic data system
Real-time analysis, not post analysis
Various data types
Unstructured data such as files, emails, and videos accounting
for 90% of the data amount generated in the next 10 years
Veracity of Great data
Prediction and analysis of the future and the behavior mode
IDC: four characteristics of big data
Page 4
Page 4
Driving Force of Big Data
1
•
Collection
Storage — open architecture, gradual scale out, and
integration of near-line systems and offline systems
2
•
Storage
Big
Data
Analysis — compound data, unified storage of multiple data
sources, unified query, and sharing of storage resources
3
•
Management
Archiving — lifecycle management of mass data at a low
cost and high energy efficiency
4
Analysis
•
Maintenance — easy to use
Page 5
Page 5
Contents
1
Definition of Big Data
2
HUAWEI Big Data Storage Solution
3
Application Scenarios of HUAWEI Solution
6
CSS/CSE
HUAWEI Big Data Storage Solution
High-performance solution
Analysis
Large-capacity solution
Big
Data
Storage
Archiving
Mass data query and
analysis solution
Page 7
Page 7
HUAWEI OceanStor N9000—X in 1
•Quick data
import
•Support for
JDBC and ODBC
•SQL 92/2003
•Data
compression
Scale-out NAS
•Linear expansion from 3 to 288
nodes
•More than 3,000,000 OPS
•Capacity up to 100 PB
•NFS/CIFS interfaces
•HDFS interface
Scale-out DB
Scale-out Backup
Page 8
•Global online
deduplication
•Support for LTFS
Page 8
HUAWEI OceanStor N9000 Architecture
Application
layer
Media asset Internet
HPC
…
Behavior prediction, real-time analysis, …
……
Unstructured data, object data, structured data
NAS interface Object interface
Big data sharing
Archiving interface
BI
Database interface
Big data analysis
Map-reduce
HDFS
Unified management and namespace
Lifecycle policy
Index of quick file
search
Remote replication and
Global deduplication
snapshot
Storage
Analysis
Storage
Archiving
10GE/Infiniband
layer
Node
Node
Node
Node
Page 9
Periodic data
inspection
Node
Node
Page 9
N9000 Hardware
Front view
High-performance node：
The scene of frequent
Rear view
Front view
Rear view
Front view
Rear view
read and write small files
High-bandwidth node：
The scene of large file
sequential read and
write
Large-capacity node：
Near-line storage
Acceleration node
Page 10
2 U high
25 disk slots
2-channel hexa-core CPU
48 GB memory at least
SSDs for storing metadata
4 U high
36 disk slots
2-channel hexa-core CPU
48 GB memory at least
4 U high
36 disk slots
2-channel quad-core CPU
16 GB memory at least
1 U high
Page 10
N9000 Software — Wise Series



Self-developed distributed file system
Mandatory, one-off purchase
Wise Data Protection (WiseDP)
2 WiseLink



Cluster load balancing software
Support for multiple load balancing policies
Optional, allowing for separate purchase
3 WiseQuota


Quota management software that sets
space usage limits on directories or users.
Optional, allowing for separate purchase



DST software
Support for policy-based automatic migration
Optional, allowing for separate purchase
1 WushanFS
WushanFS
WushanFS: Wushan
File System
4 WiseTier
Page 11
Page 11
Extensive Flexibility: WiseDP
+1
Number ODC RDC
of
Nodes
3
4
4
6
5
8
6
4
8
6
10
8
14
12
14
10
20
12
+2
ECC
ECC
Switch
+3
+4
ECC
ECC
RDN
2
2
2
2
2
2
2
4
4
Max. No. of
Max. No. of
Utilization
Allowed Faulty Allowed Faulty
Disks
Nodes
1
2
1
67%
1
2
1
75%
1
2
1
80%
2
2
2
67%
2
2
2
75%
2
2
2
80%
2
2
2
86%
4
4
4
71%
4
4
4
75%
Original data counts (ODC)
Redundant data counts (RDC)
Redundant data nodes (RDN)
N+1 to N+4 data protection
File-level data protection
1 hr/TB data recovery
Page 12
Page 12
Outstanding Performance: WiseLink
...
Automatic load balancing among
existent and newly added nodes
Single node expansion within 60s
Without human intervention
and manual modifications
Page 13
Page 13
Outstanding Performance: WiseTier
•
Enables DST to work among diverse types of
nodes.
•
Adopts tiering policies based on I/O access
popularity.
High-OPS
nodes
•
Adopts hotspot anti-shake policy to prevent
unnecessary data migration.
•
Seamlessly adapts to service changes without
interrupting ongoing applications.
Large-capacity
nodes
Hot data
Cold data
•
Consolidates multiple storage systems into a
single storage system, eliminating the possibilities
of frequent data migration and simplifying
management.
Page 14
Page 14
Simplified Management: Refined Space Management
WiseQuota: sets quotas for subdirectories to facilitate
space management.
100 TB
Fast automatic
deployment
Comprehensive
performance monitoring
Unified topology display
10 TB
20 TB
User
User
40 TB
30 TB
User group User group
20 TB
10 TB
User
User
Centralized management of a
single file system of 100 PB
Page 15
Page 15
HUAWEI 9000 — Adapting to Changes and Converging
for Easy Management
Extensive
Flexibility
 Scaling from 3
to 288 nodes
 Support up to
100 PB of
capacity
 Linear scaling of
performance
and capacity
Outstanding
Performance
 World's leading
file system
access
performance of
more than 3
million OPS
Optimal
Integration
 Distributed
scale-out
architecture
 Integration of
scale-out NAS,
scale-out DB,
and scale-out
backup
Page 16
Streamlined
Management
 Unified
management of
all nodes on a
single interface
 LCM
Page 16
Contents
1
Definition of Big Data
2
HUAWEI Big Data Storage Solution
3
Application Scenarios of HUAWEI Solution
17
HUAWEI OceanStor N9000 Scenario— Media Asset Management
Animation
rendering
Non-linear editing system
…
…
…
HD editing
SD editing
HPC
VOD system
LAN
a strong need for data sharing
N9000 highbandwidth
nodes and
large-capacity
nodes
VOD server cluster
PC
Stable bit steams and low single-stream bandwidth, with
Animation rendering solution
N9000 highperformance
nodes
TV set +
set-top box
TV station non-linear editing system solution
•
...
Cataloging
Focusing on five aspects
•
Frame image file storage, with a strong need for high
performance in sequential read of small-sized I/Os
VOD solution
•
Concurrent reads by multiple users, with a strong need
for high bandwidth
PC Smart phone
Media asset management solution
Post-production
system
Audio
Editing
Effect
Media asset
management system
Query/Cataloging
•
Large capacity, high bandwidth, and data sharing
Movie post-production solution
…
•
Stable bid streams and high-bandwidth throughput,
providing data sharing
Page 18
Page 18
HUAWEI ENTERPRISE ICT SOLUTIONS A BETTER WAY
Copyright©2012 Huawei Technologies Co., Ltd. All Rights Reserved.
The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product
portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive
statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time
without notice.