Creating a distributed Redis system using Docker

Untitled-11
redis-on-docker

A common problem I face on a daily basis is a lack of hardware / resource in order to test things out to the fullest. For example, in days gone by I’d have needed 3 servers for what i’m about to do – and in more recent times, 3 virtual machines. I dont have the time to continuously build these items, nor the resource if we were going physical. This is where my new found interest in Docker can help me out!

What I want to do on my Ubuntu ‘host’ server is create 3 Docker containers running Redis, and link them all together so that I can then develop and test the best way to monitor h-scaled Redis. Below I will show you how i’ve done it, and the benefits (even beauty) of it!

1. Download and configure our base image

For my base images, I use the excellent ‘phusion’ image which adds in a lot features that Docker omits (by design or not), including ordered startup of services and more. For more information on the phusion-base image, click here.

First, lets pull the phusion image as below, using Docker:

On completion, you should be able to run ‘docker’ images and see the new image as below:

This image is great as you can tell it easily to startup certain services on start of the container, which is a bit of a pain in the arse otherwise (in my other blog i outlined how when using a standard base image you need to use bash to essentially start things!). One of the things I wanted to ADD to this image is the ability to login via SSH using a password instead of keys; given this is only going to be running on a locally contained box im not too worried around security!

To do this on phusion, we need to create an instance of phusion, login via SSH and modify the config, then restart SSH. We will then be able to login using the root user.

First, create a new container from the phusion base image:

The ‘/sbin/my_init’ is what allows phusion to start your services on boot of the container. The ‘enable-insecure-key’ allows us to ssh into the new container using an ‘insecure key’ (see below).

Now that the container has been deployed, we need to get its IP address. To do this, use ‘docker inspect’ as below:

Next, lets pull down the ‘insecure key’ and using it to login to our new container:

Congratulations, you are now ssh’d into the container! Next we need to modify SSH so we dont need to use the insecure key in the future. To do this, open up /etc/ssh/sshd_config and find and uncomment the following line:

Then save the file and exit the text editor. Finally, we need to give our root user a password. To do this run ‘passwd’ as below:

And thats that for SSH on our base image. Next we will install redis on the container.

2. Install Redis

Now that we have our base image configured, we will need to download and install Redis. For my example I am using the latest version of redis, which can be downloaded from the page here. For my example i will be using Redis 3.0.0 RC1 from this link – https://github.com/antirez/redis/archive/3.0.0-rc1.tar.gz .

Note: You will need to install wget, gcc, make and a few other tools – these are properly base images 🙂

First we will need to download the Redis 3.0.0 RC1:

Next, lets unpack it and make it:

Finally, move the binaries into /usr/bin/ so we can access them system-wide (easier!):

Now open up /etc/redis/redis.conf, find the line ‘daemonize no’ and change it to ‘daemonize yes’. Next, lets start it up!

And thats that – Redis is now installed and running on your container, using the config file at /etc/redis/redis.conf!

3. Tell Docker to start services on boot

Now that our container is running redis and has SSH access, we need to tell it to start redis automatically on ‘start’ of the container.

To do this using the phusion base image, its actually remarkably simple. First, because we are starting docker using /sbin/my_init it will run anything we put in /etc/service/* on start which is excellent. SO, for our situation we need to create a new folder and ‘run’ file for docker here, as below:

Within run, we need to paste the following:

Finally, remember to set the run file to be executable:

And thats all we need to do! Now, lets commit this image to our local library so we can start deploying it en-mass!

4. Commit your image so you can re-use it

Now that our ‘redis’ container is working a treat, we want to save it as a pseudo ‘template’ so we can deploy it over and over again on Docker – VERY quickly.

To do this is remarkably simple: we just ‘docker commit ..’ the image to our local library, as below:

There we have it – our image is saved locally. Next we can deploy it over and over to ours hearts content, as below.

5. Deploy 3 instances of your new template

For claritys sake, I want to delete my ‘build’ container now as it is confusing to have around in the future. To do this first ‘docker stop redis’ and ‘docker rm redis’ as below:

Simples. Next, lets deploy 3 instances of our template. I like to port forward using docker so i can access it using my ‘host servers’ IP. So for 3 instances, I am going to associate:

  • node1 with hostip:7001
  • node2 with hostip:7002
  • node3 with hostip:7003

So that i can hit ‘redis-cli -h 192.168.0.2 -p 7001’ and it will redirect to ‘172.17.0.x -p 6379’, for example. Below, I have deployed 3 instances from my image:

Now I can do ‘docker ps -as’ and view all 3 of my new, shiny Redis containers:

Here we can see 3 redis containers, named node1, node2 and node3 – all with a dedicated ‘7001-7003’ port that maps straight to the Redis servers port.

To test this is working, we can use ‘redis-cli’ on another server and try and login to the ‘host server’ using the port 7001 and the host servers 192.168.x.x IP, as below:

As you can see, I can connect through to the 3 instances using their port + the host servers IP (all docker IPs’ are on 172.17.0.0/24 range). When i try an incorrect port it fails (just to prove that ALL ports dont end up in a redis server!!).

6. Configure Redis slaves

Now we have 3 individual redis servers, we want to link them together so that we can test scalability/monitoring of clusters or whatever our reason is for setting this all up! 🙂

We will need to configure node2 and node3 to be slaves of node1 which will be our master; we can do this easily by modifying /etc/redis/redis.conf.

To do this we will take advantage of our SSH configuration earlier – simply find out the IP address of the containers for node2 and node3, SSH into them, modify the config and then restart the container, as below:

In this config file we need to find the line ‘slaveof’ and uncomment and modify it to the IP address of node1 – similar to below:

Finally, restart the container using ‘docker restart node2’ (for example), and it should now be a slave of the master Redis server on node1. To verify this, use redis-cli to login to node1 and node2 on seperate terminals and use ‘set hello world’ on node1, and ‘get hello’ on node2. If it works it should look like the following:

And there you have it! 3 redis servers setup in a h-scale fashion (well, using slaves!) in docker – using your own phusion-based image. Marvellous!

Next steps

In my next blog, I will show you how to monitor your Redis cluster using Opsview so that you can get a nice pretty dashboard for your Redis stats, as below:

Screen Shot 2015-01-13 at 17.24.03

Leave a Reply