Transcript redis-io-copy
REmote DIctionary Server.
http://redis.io/
open source, advanced key-value store, data
structure server
http://www.yumianfeilong.com
Hello Redis
Install
$ wget http://redis.googlecode.com/files/redis-2.4.8.tar.gz $ tar xzf redis-2.4.8.tar.gz $ cd redis-2.4.8 $ make Start Redis server & make connection [binzhang@phxrueidb04 redis-2.4.8]$ src/redis-server redis.conf [16849] 04 Mar 02:03:59 * Server started, Redis version 2.4.8
[16849] 04 Mar 02:03:59 * The server is now ready to accept connections on port 6379 Connect server $ src/redis-cli redis > set player:666:name binzhang OK redis > get player:666:name "binzhang" 2
Learn More : data structure server
3
Agenda
Redis Manifesto 宣言 Data structures : strings, hashes, lists, sets and sorted sets.
Leveraging Redis Redis Admin and maintenance The architecture of REDIS Cases 4
Redis Manifesto
http://antirez.com/post/redis-manifesto.html
1. Redis is a DSL (Domain Specific Language) that manipulates abstract data types and implemented as a TCP daemon. keys are binary-safe strings and values are different kinds of abstract data types.
2. Redis has persistence option but Memory storage is #1. 3. The Redis API is a direct consequence of fundamental data structures.
4. Code is like a poem. 5. We believe designing systems is a fight against complexity. Most of the time the best way to fight complexity is by not creating it at all.
6. Redis API has two levels: 1) a subset of the API fits naturally into a distributed version of Redis and 2) a more complex API that supports multi-key operations. 7. We optimize for joy. When there is no longer joy in writing code, the best thing to do is stop. 5
Redis Manifesto
: where is Redis?
1. Redis is extremely fast, making it perfectly suited for applications that are write heavy, data that changes often, and data that naturally fits one of Redis’s data structures (for instance, analytics data). 2. A scenario where you probably
shouldn’t
your dataset doesn’t fit in memory.
use Redis is if you have a very large dataset of which only a small part is “hot” (accessed often) or a case where 6
Who is using Redis
Taobao Sina weibo 7
Agenda
Redis Manifesto
宣言
Simple/fast/
Data structure : strings, hashes, lists, sets and sorted sets.
Leveraging Redis
The architecture of REDIS
Redis Admin and maintenance
Cases
8
Data structure: Key-Value Data Store
Keys are strings which identify pieces of data (values) Values are arbitrary byte arrays that Redis doesn't care about Redis is implemented as five specialized data structures
Strings, hash, list, set, sort set
Pub/Sub
Querying with Redis ; 9
the above make Redis fast and easy to use, but not suitable for every scenario
Data structure : KEY value
Before we dive into the specific data types, it is important to look at a few things you should keep in mind when
designing the key structure
that holds your data.
1.a key can contain any characters, you can use separators to define a namespace with a semantic value for your business. An example might be using cache:project:319:tasks, where the colon acts as a namespace separator.
2.When defining your keys, try to limit them to a
reasonable size
. Retrieving a key from storage requires comparison operations, so keeping keys as small as possible is a good idea. Additionally, smaller keys are more effective in terms of memory usage.
3.Even though keys shouldn’t be exceptionally large, there are no big performance improvements for extremely small keys. This means you should design your keys in such a way that combines
readability (
to help you) and
regular key sizes
(to help Redis).
10
Data structure :
KEY
value
Redis support different kind of values 1.Binary-safe strings.
2.Lists of binary-safe strings.
3.Hash map of strings 4.Sets of binary-safe strings, that are collection of unique unsorted elements. 5.Sorted sets, similar to Sets but where every element is associated to a floating number score. The elements are taken sorted by score. 11 6.pubsub channels are a new addition to Redis
Data structure : strings ,
hashes, lists, sets , sorted sets.
1. Strings: the simplest and most basic data type (max 512M in length) 2. Redis Strings are binary safe, this means that a Redis string can contain any kind of data, for instance a JPEG image or a serialized Ruby object.
– redis > users:leto "{name: leto, planet: dune, likes: [spice]}" – OK – – The C structure sdshdr redis > x users:leto "{name: leto, planet: dune, likes: [spice]}" (integer) 0 Redis string: redis > strlen users:leto (integer) 42 redis > getrange users:leto 27 40 "likes: [spice]" redis > append users:leto " OVER 9000!!" (integer) 54 redis > get users:leto "{name: leto, planet: dune, likes: [spice]} OVER 9000!!" struct sdshdr { long len; long free; }; char buf[]; 12
•
Data structure : strings , hashes, lists, sets , sorted sets.
if we store a counter key, we can use commands such as INCR (or INCRBY) and DECR (or DECRBY) to increment or decrement its contained value. – To store page visit data, we could have a key “visits:pageid:totals” redis > SET visits:2:totals 1367894 OK redis > get visits:2:totals "1367894" redis > INCR visits:635:totals (integer) 1 redis > INCR visits:2:totals (integer) 1367895 redis > get visits:2:totals "1367895“ 13
Data structure :
strings
, hashes
, lists, sets , sorted sets.
• • • • Much like traditional hashtables, hashes in Redis store several fields and their values inside a specific key. so they are the perfect data type to represent objects (eg: A User with a number of fields like name, surname, age, and so forth): Example: Designing a key namespace to store our users.
redis> hset users:jdoe name "John Doe" (integer) 1 redis> hmset users:jdoe email [email protected]
phone "+1555313940" OK redis> hincrby users:jdoe visits 1 (integer) 1 redis > hget users:jdoe phone "+1555313940" A hash with a few fields (where few means up to one hundred or so) is stored in a way that takes very little space, so you can store millions of objects in a small Redis instance.
Every hash can store up to 2 32 - 1 field-value pairs (more than 4 billion).
14
• • • • • • • • •
Data structure : strings, hashes , lists, sets , sorted sets.
HDEL key field [field ...] Delete one or more hash fields HGETALL key Get all the fields and values in a hash HINCRBY key field increment Increment the integer value of a hash field by the given number HKEYS key Get all the fields in a hash HLEN key Get the number of fields in a hash HMGET key field [field ...] Get the values of all the given hash fields HMSET key field value [field value ...] Set multiple hash fields to multiple values HSETNX key field value Set the value of a hash field, only if the field does not exist HVALS key Get all the values in a hash redis > hgetall users:jdoe 1) "name" 2) "John Doe" 3) "email" 4) "[email protected]" 5) "phone" 6) "+1555313940" 7) "visits" 8) "1" 15
Data structure :
strings
,
hashes,
lists
, sets , sorted sets.
• • • • Redis Lists are simply lists of strings, sorted by insertion order. It is possible to add elements to a Redis List pushing new elements on the head (on the left) or on the tail (on the right) of the list.
LPUSH mylist a # now the list is "a" LPUSH mylist b # now the list is "b","a" RPUSH mylist c # now the list is "b","a","c" (RPUSH was used this time) support for constant time insertion and deletion of elements near the head and tail, even with many millions of inserted items. Accessing elements is very fast near the extremes of the list but is slow if you try accessing the middle of a very big list, as it is an O(N) operation.
You might want to use lists in order to implement structures such as queues The max length of a list is 2 32 list).
- 1 elements (4294967295, more than 4 billion of elements per 16
• • •
Data structure : strings, hashes, lists , sets , sorted sets.
Model a timeline in a social network, using LPUSH time line, and using LRANGE in order to add new elements in the user in order to retrieve a few of recently inserted items. – Lrange to do paging You can use LPUSH together with LTRIM (O(N))to create a list that never exceeds a given number of elements, but just remembers the latest N elements.
– Capped Collections in MongoDB.
Lists can be used as a message passing primitive, – BLPOP key [key ...] timeout Remove and get the first element in a list, or block until one is available – – – – – – – BRPOP key [key ...] timeout Remove and get the last element in a list, or block until one is available LINDEX key index Get an element from a list by its index LLEN key Get the length of a list LPOP key Remove and get the first element in a list RPOP key Remove and get the last element in a list LRANGE key start stop Get a range of elements from a list LTRIM key start stop Trim a list to the specified range 17
• • • •
Data structure :
strings
,
hashes, lists,
sets ,
sorted sets.
Sets are an unordered collection of Strings. It is possible to add, remove, and test for existence of members in O(1) (constant time regardless of the number of elements contained inside the Set).
Elements in a given set can have no duplicates. this means that adding a member does not require a check if exists then add operation.
Sets are a natural fit for circles, because sets represent collections of data, and have native functionality to do interesting things like intersections and unions. The max number of members in a set is 2 32 per set).
– - 1 (4294967295, more than 4 billion of members You can track unique things using Redis Sets. Want to know all the unique IP addresses visiting a given blog post? Simply use SADD every time you process a page view. – Redis Sets are good to represent relations. – You can use Sets to extract elements at random using the SPOP or SRANDMEMBER commands. 18
Data structure :
strings
,
hashes, lists,
sets ,
sorted sets.
We want to store several circles for each of our users, so it makes sense for our key to include a bit about the user and a bit about the actual circle. (circle:jdoe:family etc) circle:jdoe:family users:anna redis> sadd circle:jdoe:family users:richard 1) "users:mike" redis> sadd circle:jdoe:family users:mike redis> sadd circle:jdoe:soccer users:mike 1) "users:anna" redis> sadd circle:jdoe:soccer users:adam (integer) 1 4) "users:adam" circle:jdoe:family 1) "users:richard" 3) "users:anna" redis>
hgetall
users:mike (...) 19
• • • • • • • • • • • • •
Data structure : strings, hashes, lists, sets , sorted sets.
SADD key member [member ...] Add one or more members to a set SCARD key Get the number of members in a set SDIFF key [key ...] Subtract multiple sets SDIFFSTORE destination key [key ...] Subtract multiple sets and store the resulting set in a key SINTER key [key ...] Intersect multiple sets SISMEMBER key member Determine if a given value is a member of a set SMEMBERS key Get all the members in a set SMOVE source destination member Move a member from one set to another SPOP key Remove and return a random member from a set SRANDMEMBER key Get a random member from a set SREM key member [member ...] Remove one or more members from a set SUNION key [key ...] Add multiple sets SUNIONSTORE destination key [key ...] Add multiple sets and store the resulting set in a key 20
•
Data structure : strings, hashes,
implement tags
–
lists, sets , sorted sets.
To get all the tags for a given object : sadd news:1000:tags 1 redis> smembers 1. 5 2. 1 3. 77 4. 2 news:1000:tags – (integer) 1 – sadd news:1000:tags tags 1, 2, 10, and 27 at the same time 2 we may want the list of all the objects having as
Sinter
tag:1:objects tag:2:objects tag:10:objects tag:27:objects – (integer) 1 – sadd news:1000:tags 5 21
Set : Wildcard autocomplete • • • Split every username to three letter chunks – Simonw => sim, imo, mon, onw Create a set for each chunk – Sim => { simonw, asimov, fasim} If the user types “simo”, return the intersection of the “sim” and “imo”.
22
1.
2.
3.
Data structure : strings, hashes, lists, sets , sorted sets .
Every member of a Sorted Set is associated with score, that is to sort set, from the smallest to the greatest score. members are unique, scores may be repeated.
With sorted sets you can add, remove, or update elements in a very fast way (in a time proportional to the logarithm of the number of elements, O(log(N))). Get ranges by score or by rank (position) in a very fast way. – ZADD can be used both to add items to the set and to update the score of an existing member. The ZRANGE family of commands return items by their index position within the ordered set. The optional WITHSCORES argument returns the score for each item in the same response.
ZRANGEBYSCORE query the ordered set by score, instead of by index.
zadd friends:leto 100 ghanima 95 paul 95 chani 75 jessica 1 vladimir redis > zrange friends:leto 0 -1 withscores 1) "vladimir" 2) "1" 3) "jessica" 4) "75" 5) "chani" 6) "95" 7) "paul" 8) "95" 9) "ghanima" 10) "100" 23
Data structure : strings, hashes, lists, sets , sorted sets .
Zset as index : any time you need to look up data based on range queries, you should be storing it in a sorted set. They're indexes that you have to maintain yourself.
redis > zadd hackers 1940 "Alan Kay" 1953 "Richard Stallman" 1969 "Linus Torvalds" 1912 "Alan Turing" redis > zrange hackers 0 -1 1) "Alan Turing" 2) "Alan Kay" 3) "Richard Stallman" 4) "Linus Torvalds“ redis > zrangebyscore hackers 1950 1990 1) "Richard Stallman" 2) "Linus Torvalds“ redis > zrangebyscore hackers -inf 1950 1) "Alan Turing" 2) "Alan Kay" redis > zremrangebyscore hackers 1940 1960 (integer) 2 redis > zrange hackers 0 10 1) "Alan Turing" 2) "Linus Torvalds" 24
Sort set : Prefix autocomplete
• Type “binz”, return “binzhang” 1. Turn the first 4 or 5 characters of the strings into an integer (you can imagine every char as a digit of a radix 256 number for instance, but there are better representation) and add all your usernames into a sorted set with score=integer. 2. Then using ZRANGEBYSCORE you can get all the elements between a given range.
--ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]
25 3. This method is much more scalable as it's an
sorted sets : Inverted-Index Text Search with Redis 1. an inverted index - a bunch of sets mapping terms to document IDs. 2. Assign each document an ID 3. Apply stemming and stopwords first 4. Create an inverted index, with one set per word 5. Create a set of docIDs for each term –
ZINTERSTORE destination-zset number-of-zsets-to-intersect zset1 [zset2 ...] [WEIGHTS weight1 [weight2 ...]] [AGGREGATE SUM | MIN | MAX]
26
Learn More Data structure : Pub/Sub • Redis has native support for the publish/subscribe (or pub/sub) pattern 1. receivers subscribe to messages that match a specific pattern (for instance, messages that are sent to a specific “channel”), 2. an procedurer/emitter to send messages to me 3. emitter and receivers to be loosely coupled-- hey don’t need to know each other.
•
2.
1.
The pub/sub command PSUBSCRIBE pattern [pattern ...] Listen for messages published to channels matching the given patterns PUBLISH channel message Post a message to a channel 27
Learn More Data structure :
Pub/Sub redis > PUBLISH irc:football "Rock you" (integer) 0 redis > PUBLISH irc:football "have a good day" (integer) 1 redis > PUBLISH irc:football "water" redis > subscribe irc:football Reading messages... (press Ctrl-C to quit) 1) "subscribe" 2) "irc:football" 3) (integer) 1 1) "message" 2) "irc:football" 3) "have a good day" 1) "message" 2) "irc:football" 3) "water" 28
Agenda
Redis Manifesto
宣言
Data structure : strings, hashes, lists, sets and sorted sets.
Leveraging Redis
The architecture of REDIS
Redis Admin and maintenance
Cases
29
Leveraging Redis
1. Operations on KEYS 2. Big O Notation 3. Sort 4. EXPIRE 5. Transaction 6. Optimistic locking using check-and-set ( select for update) 7. Pipelining ( commands in batch) 30
Keys operation
KEYS pattern Lists all the keys in the current database that match the given pattern. [slow] TYPE key-name Tells the type of the key. Possible types are: string, list, hash, set, zset, and none.
MONITOR Outputs the commands received by the Redis server in real time. [debug purpose only] KEYS h*llo KEYS h?llo KEYS h[ae]llo.
redis > type circle:jdoe:soccer Set DEL/EXISTS/EXPIRE/TTL/PERSIST/RANDOMKEY/RE NAME Redis can handle up to 2^32 keys 31
Big O Notation
How fast a command is based on the number of items we are dealing with.
O(1): fastest, Whether we are dealing with 5 items or 5 million, you'll get the same performance.
Sismember : if a value belongs to a set O(N) : linear commands, fts etc Keys ltrim, N is the number of elements being removed.
O(log(N)): 32 zadd is a O(log(N)) command, where N is the number of elements already in the set.
sort
• • • Sort the values within a list, set or sorted set.
redis > rpush users:leto:guesses 5 9 10 2 4 10 19 2 (integer) 8 redis > sort users:leto:guesses 1) "2" 2) "2" 3) "4" 4) "5" 5) "9" 6) "10" 7) "10" 8) "19" redis 9> sadd friends:ghanima leto paul chani jessica alia duncan (integer) 6 redis > sort friends:ghanima limit 0 3 desc alpha 1) "paul" 2) "leto" 3) "jessica" Redis is single-thread, so sort on salve if large dataset Sort can store result to a key, one pattern for paginating through expensive sort results (millions of items, for example) is to save the result to a temporary key, set an expiry on it and use that for pagination via the LRANGE command.
33
Transaction
Every Redis command is atomic, including the ones that do multiple things.
– – – – incr is essentially a get followed by a set getset sets a new value and returns the original setnx first checks if the key exists, and only sets the value if it does not Msetnx fails if any key already exist (less important, now we’ve hash) MULTI command can run multiple commands as an atomic group.
1.
2.
3.
4.
Marks the start of a transaction block. Subsequent commands will be queued for atomic execution using EXEC .
DISCARD can be used in order to abort a transaction. In this case, no commands are executed and the state of the connection is restored to normal.
The commands will be executed in order The commands will be executed as a single atomic operation (without another client's command being executed halfway through) That either all or none of the commands in the transaction will be executed redis > multi OK redis > hincrby groups:1percent balance -9000000000 QUEUED redis > hincrby groups:1percent balance -9000000000 QUEUED redis > exec 1) (integer) -9000000000 2) (integer) -18000000000 34
• • •
Optimistic locking using check-and set
WATCH / UNWATCH – keys are monitored in order to detect changes against them. If at least one watched key is modified before the EXEC command, the whole transaction aborts, and EXEC returns a Null multi-bulk reply to notify that the transaction failed .
let's suppose Redis doesn't have INCR WATCH mykey val = GET mykey val = val + 1 MULTI SET mykey $val EXEC Using WATCH to implement ZPOP WATCH zset element = ZRANGE zset 0 0 MULTI ZREM zset element EXEC 35
Expiration
1. Redis allows you to mark a key for expiration. 2. You can give it an absolute time in the form of a Unix timestamp (seconds since January 1, 1970) or a time to live in seconds. a. expire pages:about 30 -- delete key after 30 seconds b. expireat pages:about 1356933600 -- delete key at 12:00 a.m. December 31st, 2012. c. ttl pages:about -- check ttl d. persist pages:about -- remove expire limit e. setex pages:about 30 '
about us
’ --set a string and specify a expire time 3. Lazy Expiration algorithmKeys are expired simple when some clients tries to access a key and the key is found to be time out
4. Once every second,
I. Tests 100 random keys from expired keys set.
II. Deletes all the keys found expired.
III. If more than 25 keys were expired, it starts again from
36
What if no available memory
Redis will return an error on write operations, but read-only query still works Can specify “maxmemory” to define a hard limit for memory usage maxmemory-policy: specify the algorithm to use when we need to reclaim memory
a.
b.
c.
d.
e.
volatile-lru (default) remove a key among the ones with an expire set, trying to remove keys not recently used.
volatile-ttl remove a key among the ones with an expire set, trying to remove keys with short remaining time to live.
volatile-random remove a random key among the ones with an expire set.
allkeys-lru like volatile-lru, but will remove every kind of key, both normal keys or keys with an expire set.
allkeys-random like volatile-random, but will remove every kind of keys, both normal keys and keys with an expire set.
LRU and minimal TTL algorithms a) b) are not precise algorithms for default Redis will check three keys(“maxmemory-samples”) and pick the one that was used less recently redis > set newkey "maxsize" (error) ERR command not allowed when used memory > 'maxmemory' 37
Redis Pipelining(How)
Send multiple commands to the server without waiting for the replies at all, and finally read the replies in a single step.
$ (echo -en "PING\r\nPING\r\nPING\r\n"; sleep 1) | nc localhost 6379 +PONG +PONG +PONG Not paying the cost of RTT for every call;
Client:
INCR X
Client:
INCR X
Client:
INCR X
Client:
INCR X
Server:
1
Server:
2
Server:
3
Server:
4 38
Redis Pipelining(why we need) • Redis is a TCP server using the client-server model and what is called a Request/Response protocol.
– The client sends a query to the server, and reads So for instance a four commands sequence is something like this: from the socket, usually in a blocking way, for the
Server: Client:
1 INCR X
Server:
– 2 The server processes the command and sends
Server: Client:
3 INCR X
Server:
4 39
Agenda
Redis Manifesto 宣言 Data structure : strings, hashes, lists, sets and sorted sets.
Leveraging Redis
Keys/O(n)/sort/expire/transaction/watch
Redis Admin and maintenance
Select database Monitor Redis Configure Persistence Starting a Redis Slave Handling a Dataset larger than memory Upgrade Redis BackUp Redis Sharding Redis Benchmarks The architecture of REDIS 40
Databases in Redis 1.A database contains a set of data. 2.A database is to group all of an application's data together and to keep it separate from another application's.
3.databases are simply identified by a number with the default database being number 0.
4.Number of databases is set via “databases” param in config 5.change to a different database via select command 41
Monitor Redis
1. MONITOR , is actually part of the Redis replication system. If you telnet directly to Redis and type monitor, you'll see a live dump of all commands executing against the database. This is really useful for debugging.
2. config set slowlog-log-slower-than 0 3. Redis-stat : similar like prstat 4. Info command – – – – – – – – – – – – – – – – – – – – – redis > info redis_version:2.4.8
redis_git_sha1:00000000 redis_git_dirty:0 arch_bits:64 multiplexing_api:epoll gcc_version:4.1.2
process_id:5898 uptime_in_seconds:163519 uptime_in_days:1 lru_clock:1033766 used_cpu_sys:1.19
used_cpu_user:2.36
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:2 connected_slaves:1 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 used_memory:1384280 42
Redis Admin and maintenance: Configure Persistence(1)
Persistence Mode: snapshotting and AOF. It should be configured in a way that suits your dataset and usage patterns.
1. snapshotting, which consists of saving the entire database to disk in the RDB format (a compressed database dump). This can be done periodically at set times, or every time a configurable number of keys changes.
# save
./redis-server --appendonly yes 3. It is possible to combine both AOF and RDB in the same instance. 4. Master ->> Slave can be a option.
5. Both are sequential IO 6. When Redis starts, it will read RDB or AOF to load all data into memory.
43
Redis Admin and maintenance: Configure Persistence(2)
1. Snapshotting
– – performs point-in-time snapshots of dataset at specified intervals. ( a full dump of your database to disk, overwriting the previous dump only if successful.
– Can manually trigger snapshotting with the SAVE and BGSAVE commands.
• BGSAVE forks the main Redis process and saves the DB to disk in the background.
• SAVE performs the same operation as BGSAVE but does so in the foreground, thereby blocking your Redis server.
– are also used when performing a master -> slave synchronization.
2. Append Only File(AOF)
– keeps a log of the commands that change your dataset in a separate file.
– an append only log. no seeks, nor corruption problems (redis-check-aof ) – Appendfsync: how often the AOF gets synched to disk (fsync syscall) : • Always (be able to group commit), every sec, and no.
– BGREWRITEAOF rewrites the AOF to match the current database; can reduce size of AOF greatly. (For example, if you are incrementing a counter 100 times, you'll end up with a single key in your dataset containing the final value, but 100 entries in your AOF. 99 of those entries are not needed to rebuild the current state.) 44
Redis Admin and maintenance: Master --Slave
Use slave : Load balance read queires, standby, Backup ,DW queries master-slave replication natively: A master can have multiple slaves.
Slaves are able to accept other slaves connections.
Redis replication is non-blocking on the master side, this means that the master will continue to serve queries when one or more slaves perform the first synchronization.
configure replication on the configuration file before starting a server slaveof master-ip-or-hostname masterport masterauth master-password by connecting to a running server and using the SLAVEOF command.
SLAVEOF master-ip-or-hostname [masterport] CONFIG SET masterauth password 45
• •
Handling a Dataset Larger Than Memory
Memory (VM) since version 2.0 ( deprecated after Redis 2.4
) . vm-enabled yes vm-swap-file Allow a dataset bigger than your available RAM by swapping rarely used values to disk and keeping all the keys and the frequently used values in memory.
1.
The keys are always kept in memory. Values can be swapped.
2.
3.
4.
5.
Redis server might end up blocking clients in order to fetch the values from disk.
Slow snapshot, Redis needs to read all the values swapped to disk in order to write them to the RDB file. AOF is better at this case.
VM also affects the speed of replication, because Redis masters need to perform a BGSAVE when a new slave connects.
SSDs such as Flash is encouraged 46
Upgrading Redis
Redis can’t do online binary upgrades solution 1. starting a new Redis server in slave mode, 2. switching over the clients to the slave 3. promoting the new server to the master role.
make sure to test before doing it on your production servers.
47
Backing up Redis
• Depending on which Redis persistence model you’re using.
1.With the default persistence model (snapshotting), you’re best off using a snapshot as a backup. /* cold backup */ redis-cli BGSAVE Copy 2.If you’re using only AOF, you’ll have to back up your log in order to be able to replay it on startup.
48 BGREWRITEAOF regularly
• •
Sharding Redis
Where is Redis Cluster – Under development. Probably reasonable beta for summer 2012 and ship the first stable one before end of 2012.
Have to implemented in the client library or application – – you should probably use consistent hashing.
you will not be able to perform some operations that affect multiple keys, because those keys might be in different shards (servers).
49
Benchmarks--- How fast is Redis?
• redis-benchmark utility that simulates SETs/GETs done by N clients at the same time sending M total queries [hadoop@phxrueidb03 src]$ ./redis-benchmark -q -n 100000 PING (inline): 97370.98 requests per second PING: 101214.58 requests per second MSET (10 keys): 66357.00 requests per second SET: 105263.16 requests per second GET: 103199.18 requests per second INCR: 104493.20 requests per second LPUSH: 104931.80 requests per second LPOP: 104384.13 requests per second SADD: 104931.80 requests per second SPOP: 103950.10 requests per second LPUSH (again, in order to bench LRANGE): 104931.80 requests per second LRANGE (first 100 elements): 44964.03 requests per second LRANGE (first 300 elements): 22825.84 requests per second LRANGE (first 450 elements): 16564.52 requests per second LRANGE (first 600 elements): 12701.64 requests per second 50
Benchmarks- Redis VS memcached
51
Benchmarks--- How fast is Redis?
• • • • Redis is a server: all commands involve network or IPC roundtrips. Cost of most operations is precisely dominated by network/protocol management.
– low latency network Redis commands return an acknowledgment for all usual commands. Redis is an in-memory data store with some optional persistency options. Some persistency option would bring latency.
– huge page & SSD Redis is a single-threaded server. It is not designed to benefit from multiple CPU cores. People are supposed to launch several Redis instances to scale out on several cores if needed. – Redis favors fast CPUs with large caches and not many cores. 52
Agenda
Redis Manifesto 宣言 Data structure : strings, hashes, lists, sets and sorted sets.
Leveraging Redis Redis Admin and maintenance Configure Persistence/Redis Slave/Handling a Dataset larger than memory Upgrade Redis/BackUp Redis/Sharding Redis/ benchmarks
The architecture of REDIS
How Redis works
Latency in Redis
53
How Redis works
How a command received by a client is processed internally by Redis: 1.
Redis uses a single thread that manages synchronously all network connection. A thin event library has been implemented to abstract several unix system calls (epoll, select, kqueue).
2.
Requests are managed with commands. Using a command table and according what event is read from sockets a command handler is invoked to perform desired action.
54
Latency in Redis
• • • • • •
Latency induced by network and communication
– use aggregated commands (MSET/MGET) and Pipelining
Single threaded nature of Redis
– a mostly single threaded design ( I/O threads in background since 2.4) – all the requests are served sequentially
Latency generated by slow commands
– – a request is slow to serve all the other clients will wait for this request to be served commands operating on many elements, like SORT, LREM, SUNION and others. For instance taking the intersection of two big sets can take a considerable amount of time.
– run all your slow queries on replciations
Latency generated by fork
– The fork operation (running in the main thread) can induce latency by itself.
Latency induced by swapping (operating system paging) Latency due to AOF and disk I/O
55
Memory efficient for list adlist.h: A generic doubly linked list implementation typedef struct list { listNode *head; listNode *tail; void *(*dup)(void *ptr); void (*free)(void *ptr); int (*match)(void *ptr, void *key); unsigned int len; } list; typedef struct listNode { struct listNode *prev; struct listNode *next; void *value; } listNode; O(1) is cool but *prev/*next would take amounts of bytes if *value is few bytes.
Ziplist (list-max-ziplist-entries 512 & list-max-ziplist-value 64) I.Save memory by using a little more CPU II.Pack list in a single block of memory III.Value header holds encoding / value length IV.O(memory size) LPUSH / LPOP V.Good fit for small payload, limited size 56
Memory efficient for hash • • Zmap (hash-max-zipmap-entries 512 & hash-max-zipmap-value 64) – keys and values are prefixed length "objects", the lookup will take O(N) where N is the number of elements in the zipmap and *not* the number of bytes needed to represent the zipmap. Other data structure also have similar improve ( sort sets, intset) 57
Skip-list for sort set 1.
2.
3.
4.
5.
6.
Consists of several levels, Each level is a sorted list All keys appear in level 1 If key x appears in level n, then it also appears in all levels below n An element in level n points (via down pointer) to the element with same key in the level below Each level has int_min and int_max Top points to the smallest element in the highest level 58
Memory structure lazy rehashing ： The more operation you run into an hash table that is rhashing, the more rehashing "steps" are performed, so if the server is idle the rehashing is never complete and some more memory is used by the hash table.
active rehashing ： uses 1 millisecond every 100 milliseconds of CPU time in order to help rehashing the main Redis hash table (the one mapping top-level keys to values). 59
memory fragmentation
Info used_memory:21279952 memory allocated to redis used_memory_human:20.29M
used_memory_rss:23654400 memory from OS, result of ps or top used_memory_peak:21704152 used_memory_peak_human:20.70M
mem_fragmentation_ratio:1.11 = used_memory_rss/used_memory mem_allocator:jemalloc-2.2.5 default in linux 2.4 and 2.6 String: dictEntry(12bytes)+sds(store key)+redisObject(12bytes)+sds(store value) Set hello word = 16(dictEtnry) + 16 (redisObject) + 16(“hello”) + 16(“world”), 60
Redis Security
1. Redis is designed to be accessed by trusted clients inside trusted environments.
Firewall on redis port 2. Redis is not optimized for maximum security but for maximum performance and simplicity.
3. Authentication feature The password is in clear text inside redis.conf file and client configuration AUTH command, like every other Redis command, is sent unencrypted 4. Data encryption support - None 5. Disabling of specific commands -- rename-command FLUSHALL "" 61
Agenda
Redis Manifesto 宣言 Data structure : strings, hashes, lists, sets and sorted sets.
Leveraging Redis Redis Admin and maintenance The architecture of REDIS
Event library /Memory efficient
Latency /Security
Cases
http://lloogg.com
a simple Twitter clone
Sina Weibo
62
CASE 1: http://lloogg.com/
1.
2.
3.
List, lpush,ltrim show access history Strings:incr show pageveiws zset: show opt references, ref as a score 63
a simple Twitter clone Register: INCR global:nextUserId => 1000 SET uid:1000:username antirez SET uid:1000:password p1pp0 SET username:antirez:uid 1000 Cookie: SET uid:1000:auth fea5e81ac8ca77622bed1c2132a021f9 SET auth:fea5e81ac8ca77622bed1c2132a021f9 1000 Pageing $posts = $r->lrange($key,$start,$start+$count);
Making it horizontally scalable
1. Split by hash key Circles and posts: uid:1000:followers => Set of uids of all the followers users uid:1000:following => Set of uids of all the following user uid:1000:posts => a List of post ids, every new post is LPUSHed here .
New post INCR global:nextPostId => 10343 SET post:10343 "$owner_id|$time|I'm having fun with Retwis" LPUSH to user’s followers foreach($followers as $fid) { $r->push("uid:$fid:posts",$postid,false); } Push to latest news: $r->push("global:timeline",$postid,false); $r->ltrim("global:timeline",0,1000); 64
Sina weibo 65
Summary Redis Manifesto Memory #1;data structure server Data structure : strings, hashes, lists, sets and sorted sets, pub sub Leveraging Redis KEYS/Big O /Sort/EXPIRE/Transaction/Optimistic locking/Pipelining Redis Admin and maintenance Select /Redis/Persistence/Replication/ VM/Upgrade/BackUp/Sharding /Benchmarks The architecture of REDIS Event library /Memory efficient Latency /Security Cases http://lloogg.com
/ a simple Twitter clone /Sina Weibo 66
• Q&A
Redis
67