Transcript Cassandra Meetup Nice Jean Armel Luce Orange France Thursday, June 19 2014 Summary  Short description of PnS.

Cassandra Meetup Nice
Jean Armel Luce
Orange France
Thursday, June 19 2014
Summary
2

Short description of PnS. Why did we choose C* ?

Some key features of C*

After the migration …

Analytics with Hadoop/Hive over Cassandra

Some conclusions about the project PnS3.0
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
Short description of PnS3
PnS – Short description

PnS means Profiles and Syndication : PnS is a highly available
service for collecting and serving live data about Orange customers

End users of PnS are :
– Orange customers (logged to Portal www.orange.fr)
– Sellers in Orange shops
– Some services in Orange (advertisements, …)
4
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
PnS – The Big Picture
Millions of HTTP requests
(Rest or Soap)
Fast and highly available
WebService to get or set
data stored by pns :
-
postProcessing(data1)
postProcessing(data2)
postProcessing(data3)
postProcessing(datax)
…
End users
DB Queries
R/W operations
Thousands of files
(Csv or Xml)
Scheduled data injection
PNS
Data providers
5
Jean Armel Luce - Orange-France
Database
Cassandra Meetup Nice – June 19 2014
PnS – Architecture
2 DCs architecture for high availability

Until 2012, data were stored in 2
differents backends :

MySQL cluster (for volatile data)

PostGres « cluster » (sharding and
replication)
Bagnolet
and

web services
Sophia
Antipolis
(read and writes)
for batch updates

6
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
Timeline – Key dates of PnS 3.0
PNS 2
• Study phase
2010
to 2012
We did a large study about a few NoSQL databases (Cassandra, MongoDB, Riak, Hbase, Hypertable, …)
 We chose Cassandra as the single backend for PnS
• Design phase
06/2012
We started the design phase of PnS3.0
• Proof Of Concept
09/2012
We started a 1st (small) Cassandra cluster in production for a non critical application : 1 table, key value
access
• Production phase
04/2013
Migration of the 1st subset of data of PnS from mysql cluster to Cassandra in production
• Complete migration
05/2013
to
12/2013
Migration of all other subsets of data from Mysql cluster and Postgres to Cassandra
Add new nodes in the cluster (From 8 nodes in each DC to 16 nodes in each DC)
• Analytics
0
04/2014
7
Add a 3rd datacenter
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
PnS – Why did we choose Cassandra ?

Cassandra fits our requirements :
– Very high availability
PnS2 = 99,95% availability
we want to improve it !!!
– Low latency
20 ms < RT PnS2 web service < 150 ms
we want to improve it !!!
Higher load, higher volume next years
unpredictable; better scalability
?
brings new businesses
– Scalability

And also :
– Ease of use : Cassandra is easy to administrate and operate
– Some features that I like (rack aware, CL per request, …)
– Cassandra is designed to work naturally and plainly in a multidatacenter
architecture
8
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
Some key features of C*
Who is Cassandra ?

Cassandra is a NoSQL database, developped in Java,

Cassandra was created at Facebook in 2007, was open sourced
and then incubated at Apache, and is nowadays a Top-LevelProject.

2 distributions of Cassandra :
– Community edition : http://cassandra.apache.org/
 distributed under the Apache License
– Enterprise distribution : http://www.datastax.com/download
 distributed by Datastax
10
Jean Armel Luce - Orange-DSIF/SDFY-PnS 3.0
Cassandra Meetup Nice – June 19 2014
Cassandra : architecture/topology


11
The main characteristic of Cassandra : all the nodes play the same role

No master, no slave, no configuration server  no SPOF

Rows are sharded among the nodes in the cluster

Rows are replicated among the nodes in the cluster
The parameter TopologyStrategy defines how/where rows in a keyspace are
replicated (monodatacenter, multidatacenter, …)
Jean Armel Luce - Orange-DSIF/SDFY-PnS 3.0
Cassandra Meetup Nice – June 19 2014
Cassandra : how requests are executed

The application sends a request to one of the nodes (not always the same; try to
balance the load among the nodes). This node is called the coordinator

The coordinator routes the query to the datanode(s)

The datanodes execute the query and return the result to the coordinator

The coordinator returns the result to the application
Application
12
Jean Armel Luce - Orange-DSIF/SDFY-PnS 3.0
Cassandra Meetup Nice – June 19 2014
Cassandra : what happens when a node crash?

Read query Case 1 : a READ query is executed while a datanode is crashed :

the coordinator has already received the information (via Gossip) that
a node is down and do not send any request to this node
Replica1
Replica 2
Application
13
Jean Armel Luce - Orange-France
Replica 3
Cassandra Meetup Nice – June 19 2014
Cassandra : what happens when a node crash

Read query Case 2 : the coordinator crashes while a READ query is being executed :

the application receives a KO (or timeouts), then re-sends the request
to another node which acts as a new coordinator
Application
14
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
Cassandra : what happens when a node crash ?

Write query Case 1 : a WRITE query is executed while a datanode is
crashed and there are enough replica up :
– A “Hinted Handoff” is stored in the coordinator
Replica1
Replica 1
HH
Replica 2
Application
15
Jean Armel Luce - Orange-France
Replica 3
Cassandra Meetup Nice – June 19 2014
Cassandra : what happens when a node crash ?

Write query Case 1 : a WRITE query is executed while a datanode is
crashed :
– The write is executed in all replica which are available.
– A Hinted Handoff is stored in the coordinator, and the query will be
executed when the datanode comes back again (within 3 hours)

3 tips for keeping consistency between nodes :
– Hinted Handoffs (repair when node comes back in the ring after a
failure)
– Read repairs (automatic repair in background for 10% of read queries)
– Anti entropy repairs (manual read repair for all data)
16
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
The eventual consistency with Cassandra


We can specify the consistency level for each read/update/insert/delete
request
Latency
Consistenc
Availability
CL mostly used :
y
 LOCAL_ONE
Weak
Higher
Lower

ONE

ANY

LOCAL_QUORUM

QUORUM

ALL

SERIAL
Strong
Lower
Strong consistency : W + R > RF
17
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
Higher
Cassandra : sharding with virtual nodes
(versions 1.2 +)

Virtual nodes are available since C* 1.2.

With virtual nodes, adding a new node (or many nodes) in the cluster is easy


18
data are moved from ALL the old nodes to new node :

few data to move between nodes

after the move of data, the cluster is still well balanced
procedure totally automatized

Adding a new node in the cluster is a normal operation which is done on-line
without interruption of service

When adding nodes, replica are also moved between nodes
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
Cassandra : Adding a node using Vnodes
Example : adding a 5th node in a 4-nodes cluster
Node 4
Node 1
Node 5
Node 3
Node 2
19
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
CQL (Cassandra Query language)


20
DDL :

CREATE keyspace, CREATE table, CREATE INDEX

ALTER keyspace, ALTER table

DROP keyspace, DROP table, DROP INDEX
DML :

SELECT

INSERT/UPDATE (INSERT equivalent to UPDATE : improve performances)

DELETE (delete a ROW or delete COLUMNS in a ROW)
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
CQL (Cassandra Query language)

Exemple :
jal@jal-VirtualBox:~/cassandra200beta1/apache-cassandra-2.0.0-beta1-src/bin$
./cqlsh -u cassandra -p cassandra
Connected to Test Cluster at localhost:9160.
[cqlsh 4.0.0 | Cassandra 2.0.0-beta1-SNAPSHOT | CQL spec 3.1.0 | Thrift protocol
19.37.0]
Use HELP for help.
Keyspace instead
of database
cqlsh> CREATE KEYSPACE fr
Replication
strategy
Replication
factor
... WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
cqlsh> use fr;
Connexion to the
keyspace
21
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
CQL (Cassandra Query language)

Exemple :
cqlsh:fr> CREATE TABLE customer (
... custid int,
INSERT equivalent
to UPDATE
... firstname text,
... lastname text,
... PRIMARY KEY (custid) );
cqlsh:fr>
cqlsh:fr> UPDATE customer set firstname = ‘Bill', lastname = ‘Azerty' WHERE
custid = 1;
cqlsh:fr> INSERT INTO customer (custid , firstname , lastname ) values (2,
‘Steve', ‘Qwerty');
cqlsh:fr>
22
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
CQL (Cassandra Query language)

Exemple :
cqlsh:fr> SELECT firstname, lastname FROM customer WHERE custid = 1;
firstname | lastname
SELECT with clause
WHERE on primary
key
-----------+----------
Bill
| Azerty
(1 rows)
cqlsh:fr> SELECT * FROM customer WHERE custid = 2;
custid | firstname | lastname
--------+------------+---------2 | Steve
|
Qwerty
(1 rows)
23
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
CQL (Cassandra Query language)

Exemple :
cqlsh:fr> SELECT * FROM customer WHERE lastname= ‘Azerty';
Bad Request: No indexed columns present in by-columns clause with Equal
operator
SELECT rejected.
No other operator than = accepted in
WHERE clause
(<, >, != rejected)/ The column in the
WHERE clause must be indexed
This request requires an index on column
‘lastname’
24
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
After the migration …
The latency

Comparison before/after migration to Cassandra

Some graphs about the latency of the web services are very
explicit :
Service push mail
Service push webxms
dates of
migration to C*
26
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
The latency

Read and write latencies are now in microseconds in the datanodes :
Thanks to
and
This latency will be improved by (tests in progress) :
ALTER TABLE syndic WITH compaction = { 'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : ?? };
27
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
The availability
•
We got a few hardware failures and network outages
•
28
No impact on QoS :
•
no error returned by the application
•
no real impact on latency
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
The scalability
•
Thanks to vnodes (available since Cassandra 1.2), it is easy to scale
out
With NetworkTopologyStrategy, make sure to distribute evenly the nodes
in the racks
29
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
Analytics with Hadoop/Hive
over Cassandra
Basic architecture of the Cassandra cluster

Cluster without Hadoop : 2 datacenters, 16 nodes in each DC

RF (DC1, DC2) = (3, 3)

Web servers in DC1 send queries to C* nodes in DC1

Web servers in DC2 send queries to C* nodes in DC2
Pool
of
web
servers
DC1
31
Jean Armel Luce - Orange-France
DC1
Cassandra Meetup Nice – June 19 2014
DC2
Pool
of
web
servers
DC2
Architecture of the Cassandra cluster with the datacenter for
analytics

Cluster with Hadoop : 3 datacenters, 16 nodes in DC1, 16 nodes in
DC2, 4 nodes in DC3

32
RF (DC1, DC2, DC3) = (3, 3, 1)

Because RF = 1 in DC3, we need less storage space in this
datacenter

We favor cheaper disks (SATA) in DC3 rather than SSDs or
FusionIo cards
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
Architecture of the Cassandra cluster with the datacenter for
analytics
Pool
of
web
servers
DC1
DC1
DC2
DC3
33
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
Pool
of
web
servers
DC2
Some conclusions about
the project PnS3
Conclusions

35
With Cassandra, we have improved our QoS

Lower response time

Higher availability

More flexibility for exploitation teams

We are able to open our service to new opportunities

There is a large ecosystem around C* (Hadoop, Hive, Pig, Storm,
Shark, Titan, …), which offers more capabilities.

The Cassandra community is very active and helps a lot. There are a
lot of resources available : mailing lists, blogs, …
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
Thank you
for your attention
Questions
37
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
A few answers about hardware/OS version /Java
version/Cassandra version/Hadoop version

Hardware :


38
16 nodes in DC1 and DC2 at the end of 2013 :

2 CPU 6cores each Intel® Xeon® 2.00 GHz

64 GB RAM

FusionIO 320 GB MLC
4 nodes in DC3

24 GB de RAM

2 CPU 6cores each Intel® Xeon® 2.00 GHz

SATA Disks 15Krpm

OS : Ubuntu Precise (12.04 LTS)

Cassandra version : 1.2.13

Hadoop version : CDH 4.5 (with Hive 0.10) : Hadoop 2 with MRv1

Hive handler : https://github.com/cscetbon/hive/tree/hive-cdh4-auth

Java version : Java7u45 (GC CMS with option CMSClassUnloadingEnabled)
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014
A few answers about data and requests



39
Data types :

Volume : 6 TB at the end of 2013

elementary types : boolean, integer, string, date

collection types

complex types : json, xml (between 1 and 20 KB)
Requests :

10.000 requests/sec at the end of 2013

80% get

20% set
Consistency level used by PnS for on line queries and batch updates
:

LOCAL_ONE (95% of the queries)

LOCAL_QUORUM (5% of the queries)
Jean Armel Luce - Orange-France
Cassandra Meetup Nice – June 19 2014