Up to Something

Using redis for data caching

What is Redis ?

“Redis is an open-source, networked, in-memory, persistent, journaled, key-value data store similar to memcached.” Wikipedia .

Why and for What am I using Redis ? (Context)

Have five python (multiprocessing  module based) processes that access common system queue and fetch certain web pages then store them to db. These process loop on the queue forever and the queue is never empty. Each process timeouts after say 20 sec if unable to retrieve the webpage. I want to ensure that none of the process try to fetch a web page which was fetched in last 20 minutes.

My initial idea was to simply keep a in memory python dictionary (hash) with list of all URLs and timestamp when they were fetched.  Just then Ankit mailed me suggesting I could use redis somewhere in this software that I am making. So I did.

Installing and Running Redis on Ubuntu/Linux

$ wget http://redis.googlecode.com/files/redis-1.02.tar.gz
$ tar xzf redis-1.02.tar.gz
$ cd redis-1.02
$ make
$ ./redis-server

The above five commands and Redis is up and running.

Installing and Using the Python Client on Ubuntu/Linux

$ easy_install redis

or

get the redis client source code from here extract and

$ python setup.py build

$ sudo python setup.py install

$ python

>> import redis

>> server = redis.Redis(“localhost”)

>> server.set(“http://cnn.com”, “CNN web page data”)

>> server.get(“http://cnn.com”)

‘CNN web page data’

>> server.exists(“http://cnn.com”)

True

>> server.expire(“http://cnn.com”, 120)

True

>> #after 120 seconds the key and value pair self destruct … yeah mission impossible

>> server.exists(“http://cnn.com”)

False

To sum up what happen above is we started the redis server. Opened Python interpreter and imported redis module. Connected to the redis server on local machine. Created a key value pair using the set API. Key being a url in our case, value is the content of the webpage. Fetched the value against the key using the get API. Now I wanted a caching layer. Say the HTTPWorker Process checks the cache if it gets data from the cache it doesn’t seek the same from the web. So I use the expire API. I set the key to expire after a given no of seconds. After that the key will simply not exist anymore and thus HTTP Worker will fetch from the web.

Redis is much much more than the above. It provides lot of features like ability of value to be list, set, and hash too. It allows you to treat value as a counter increment and decrement it atomically. Fetch all keys stored.

What I like about Redis

a) C code,

b) Easy API and Python Client,

c) 5 minutes and it’s deployed no config required whatsoever to boot,

d) Runs well on single machine,

e) Learning/Deployment/Usage curve is very low.