A common problem I face on a daily basis is a lack of hardware / resource in order to test things out to the fullest. For example, in days gone by I’d have needed 3 servers for what i’m about to do – and in more recent times, 3 virtual machines. I dont have the time to continuously build these items, nor the resource if we were going physical. This is where my new found interest in Docker can help me out!

What I want to do on my Ubuntu ‘host’ server is create 3 Docker containers running Redis, and link them all together so that I can then develop and test the best way to monitor h-scaled Redis. Below I will show you how i’ve done it, and the benefits (even beauty) of it!

1. Download and configure our base image

For my base images, I use the excellent ‘phusion’ image which adds in a lot features that Docker omits (by design or not), including ordered startup of services and more. For more information on the phusion-base image, click here.

First, lets pull the phusion image as below, using Docker:

[email protected]:/home/sam# docker pull phusion/baseimage

On completion, you should be able to run ‘docker’ images and see the new image as below:

[email protected]:/home/sam# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
phusion/baseimage 0.9.15 cf39b476aeec 3 months ago 289.4 MB
phusion/baseimage latest cf39b476aeec 3 months ago 289.4 MB

This image is great as you can tell it easily to startup certain services on start of the container, which is a bit of a pain in the arse otherwise (in my other blog i outlined how when using a standard base image you need to use bash to essentially start things!). One of the things I wanted to ADD to this image is the ability to login via SSH using a password instead of keys; given this is only going to be running on a locally contained box im not too worried around security!

To do this on phusion, we need to create an instance of phusion, login via SSH and modify the config, then restart SSH. We will then be able to login using the root user.

First, create a new container from the phusion base image:

[email protected]:/home/sam# docker run -d --name redis phusion/baseimage /sbin/my_init --enable-insecure-key

The ‘/sbin/my_init’ is what allows phusion to start your services on boot of the container. The ‘enable-insecure-key’ allows us to ssh into the new container using an ‘insecure key’ (see below).

Now that the container has been deployed, we need to get its IP address. To do this, use ‘docker inspect’ as below:

[email protected]:/home/sam# docker inspect node1 | grep IPA
"IPAddress": "172.17.0.46",

Next, lets pull down the ‘insecure key’ and using it to login to our new container:

[email protected]:/home/sam# curl -o insecure_key -fSL https://github.com/phusion/baseimage
[email protected]:/home/sam# docker/raw/master/image/insecure_key
[email protected]:/home/sam# chmod 600 insecure_key
[email protected]:/home/sam# ssh -i insecure_key [email protected]
The authenticity of host '172.17.0.52 (172.17.0.52)' can't be established.
ECDSA key fingerprint is aa:bb:cc:xx:xx:xx:xx:xx:xx:xx:xx:yy:zz:04:bf:04.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.52' (ECDSA) to the list of known hosts.
[email protected]:~#

Congratulations, you are now ssh’d into the container! Next we need to modify SSH so we dont need to use the insecure key in the future. To do this, open up /etc/ssh/sshd_config and find and uncomment the following line:

PermitRootLogin yes

Then save the file and exit the text editor. Finally, we need to give our root user a password. To do this run ‘passwd’ as below:

[email protected]:~# passwd
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
[email protected]:~#

And thats that for SSH on our base image. Next we will install redis on the container.

2. Install Redis

Now that we have our base image configured, we will need to download and install Redis. For my example I am using the latest version of redis, which can be downloaded from the page here. For my example i will be using Redis 3.0.0 RC1 from this link – https://github.com/antirez/redis/archive/3.0.0-rc1.tar.gz .

Note: You will need to install wget, gcc, make and a few other tools – these are properly base images 🙂

First we will need to download the Redis 3.0.0 RC1:

[email protected]:/home# wget https://github.com/antirez/redis/archive/3.0.0-rc1.tar.gz

Next, lets unpack it and make it:

[email protected]:/home# tar -zxvf 3.0.0-rc1.tar.gz
[email protected]:/home# cd redis-3.0.0-rc1/
[email protected]:/home/redis-3.0.0-rc1# make

Finally, move the binaries into /usr/bin/ so we can access them system-wide (easier!):

[email protected]:/home/redis-3.0.0-rc1# cd src
[email protected]:/home/redis-3.0.0-rc1/src# mv redis-cli redis-server redis-sentinel /usr/bin/
[email protected]:/home/redis-3.0.0-rc1/src# cd ..
[email protected]:/home/redis-3.0.0-rc1# mkdir -p /etc/redis/
[email protected]:/home/redis-3.0.0-rc1# cp redis.conf /etc/redis/redis.conf

Now open up /etc/redis/redis.conf, find the line ‘daemonize no’ and change it to ‘daemonize yes’. Next, lets start it up!

[email protected]:/home/redis-3.0.0-rc1/src# redis-server /etc/redis/redis.conf
[email protected]:/home/redis-3.0.0-rc1/src#

And thats that – Redis is now installed and running on your container, using the config file at /etc/redis/redis.conf!

3. Tell Docker to start services on boot

Now that our container is running redis and has SSH access, we need to tell it to start redis automatically on ‘start’ of the container.

To do this using the phusion base image, its actually remarkably simple. First, because we are starting docker using /sbin/my_init it will run anything we put in /etc/service/* on start which is excellent. SO, for our situation we need to create a new folder and ‘run’ file for docker here, as below:

[email protected]:/etc/service# cd /etc/service
[email protected]:/etc/service# mkdir redis
[email protected]:/etc/service# cd redis
[email protected]:/etc/service/redis# nano run

Within run, we need to paste the following:

#!/bin/sh
set -e
exec /usr/bin/redis-server /etc/redis/redis.conf

Finally, remember to set the run file to be executable:

[email protected]:/etc/service/redis# chmod +x run

And thats all we need to do! Now, lets commit this image to our local library so we can start deploying it en-mass!

4. Commit your image so you can re-use it

Now that our ‘redis’ container is working a treat, we want to save it as a pseudo ‘template’ so we can deploy it over and over again on Docker – VERY quickly.

To do this is remarkably simple: we just ‘docker commit ..’ the image to our local library, as below:

[email protected]:/home/sam# docker ps -as
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE
e3919192d9e3 phusion/baseimage:0.9.15 "/sbin/my_init --ena 3 hours ago Up 3 hours redis 164.9 MB
[email protected]:/home/sam# docker commit redis redis-cluster-node
[email protected]:/home/sam# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
redis-cluster-node latest babfb02edf4d 5 hours ago 561.2 MB

There we have it – our image is saved locally. Next we can deploy it over and over to ours hearts content, as below.

5. Deploy 3 instances of your new template

For claritys sake, I want to delete my ‘build’ container now as it is confusing to have around in the future. To do this first ‘docker stop redis’ and ‘docker rm redis’ as below:

[email protected]:/home/sam# docker stop redis
redis
[email protected]:/home/sam# docker rm redis
redis
[email protected]:/home/sam#

Simples. Next, lets deploy 3 instances of our template. I like to port forward using docker so i can access it using my ‘host servers’ IP. So for 3 instances, I am going to associate:

  • node1 with hostip:7001
  • node2 with hostip:7002
  • node3 with hostip:7003

So that i can hit ‘redis-cli -h 192.168.0.2 -p 7001’ and it will redirect to ‘172.17.0.x -p 6379’, for example. Below, I have deployed 3 instances from my image:

[email protected]:/home/sam# docker run -d --name node1 -p 7001:6379 redis-cluster-node /sbin/my_init
cd1c1f96346bdf9c1cec04333c2e849992ecbc4375dcea6b30902dd9842d8c99
[email protected]:/home/sam# docker run -d --name node2 -p 7002:6379 redis-cluster-node /sbin/my_init
cd1c1f96346bdf9c1cec04333c2e849992ecbc4375dcea6b30902dd9842d8c99
[email protected]:/home/sam# docker run -d --name node3 -p 7003:6379 redis-cluster-node /sbin/my_init
cd1c1f96346bdf9c1cec04333c2e849992ecbc4375dcea6b30902dd9842d8c99
[email protected]:/home/sam#

Now I can do ‘docker ps -as’ and view all 3 of my new, shiny Redis containers:

[email protected]:/home/sam# docker ps -as
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE
a532b4ac60d9 redis-cluster-node:latest "/sbin/my_init" 5 hours ago Up 5 hours 0.0.0.0:7003->6379/tcp node3 2.267 MB
6c8a87a0a76a redis-cluster-node:latest "/sbin/my_init" 5 hours ago Up 5 hours 0.0.0.0:7002->6379/tcp node2 2.318 MB
39e02633ccf8 redis-cluster-node:latest "/sbin/my_init" 5 hours ago Up 5 hours 0.0.0.0:7001->6379/tcp node1 2.334 MB
[email protected]:/home/sam#

Here we can see 3 redis containers, named node1, node2 and node3 – all with a dedicated ‘7001-7003’ port that maps straight to the Redis servers port.

To test this is working, we can use ‘redis-cli’ on another server and try and login to the ‘host server’ using the port 7001 and the host servers 192.168.x.x IP, as below:

[email protected]:/home/sam# redis-cli -h 192.168.0.2 -p 7001
redis 192.168.0.16:7001>
[email protected]:/home/sam# redis-cli -h 192.168.0.2 -p 7002
redis 192.168.0.16:7002>
[email protected]:/home/sam# redis-cli -h 192.168.0.2 -p 7003
redis 192.168.0.16:7003>
[email protected]:/home/sam# redis-cli -h 192.168.0.2 -p 7005
Could not connect to Redis at 192.168.0.2:7005: Connection refused
not connected>

As you can see, I can connect through to the 3 instances using their port + the host servers IP (all docker IPs’ are on 172.17.0.0/24 range). When i try an incorrect port it fails (just to prove that ALL ports dont end up in a redis server!!).

6. Configure Redis slaves

Now we have 3 individual redis servers, we want to link them together so that we can test scalability/monitoring of clusters or whatever our reason is for setting this all up! 🙂

We will need to configure node2 and node3 to be slaves of node1 which will be our master; we can do this easily by modifying /etc/redis/redis.conf.

To do this we will take advantage of our SSH configuration earlier – simply find out the IP address of the containers for node2 and node3, SSH into them, modify the config and then restart the container, as below:

[email protected]:/home/sam# docker inspect node1 | grep IPA
"IPAddress": "172.17.0.46",
[email protected]:/home/sam# docker inspect node2 | grep IPA
"IPAddress": "172.17.0.47",
[email protected]:/home/sam# ssh [email protected]
[email protected]'s password:
Last login: Tue Jan 13 11:47:31 2015 from 172.17.42.1
[email protected]:~# nano /etc/redis/redis.conf

In this config file we need to find the line ‘slaveof’ and uncomment and modify it to the IP address of node1 – similar to below:

[email protected]:~# cat /etc/redis/redis.conf | grep slaveof
# Master-Slave replication. Use slaveof to make a Redis instance a copy of
slaveof 172.17.0.46 6379
[email protected]:~#

Finally, restart the container using ‘docker restart node2’ (for example), and it should now be a slave of the master Redis server on node1. To verify this, use redis-cli to login to node1 and node2 on seperate terminals and use ‘set hello world’ on node1, and ‘get hello’ on node2. If it works it should look like the following:

[email protected]:/home/sam# redis-cli -h 192.168.0.16 -p 7001
redis 192.168.0.12:7001> set hello world
OK
redis 192.168.0.12:7001> exit
[email protected]:/home/sam# redis-cli -h 192.168.0.16 -p 7002
redis 192.168.0.12:7002> get hello
"world"
redis 192.168.0.12:7002> exit
[email protected]:/home/sam#

And there you have it! 3 redis servers setup in a h-scale fashion (well, using slaves!) in docker – using your own phusion-based image. Marvellous!

Next steps

In my next blog, I will show you how to monitor your Redis cluster using Opsview so that you can get a nice pretty dashboard for your Redis stats, as below: