Scaling with Docker Compose

AaronTuesday, February 23rd 2016

We'll be using Docker, Docker Compose, nginx, consul, consul-template, and node.js to build a load balanced stack that can be started on any server with docker installed just by uploading a few files and running docker-compose up -d

Setting up Docker compose

Start by creating a file called docker-compose.yml and then we'll add consul to it be adding the following code.

consul:  
  command: -server -bootstrap -ui-dir /ui
  image: progrium/consul:latest
  hostname: consul
  ports:
    - "8400:8400"
    - "8500:8500"
    - "8600:53/udp"
  volumes:
    - /mnt:/data

Consul is a service that provides automatic service discovery for any other service you might run on your server, we're going to use Consul to allow us to tell nginx where our web servers are so that it can load balance across them.

If you now run docker-compose up -d you will have a single node consul server running you can test this by navigating to http://<your ip>:8500/ui.

As you can see there are not currently any services registered on this node (not even consul) so lets fix that now by adding registrator to your docker-compose.yml file like so.

registrator:  
  command:  -internal consul://consul:8500
  image: gliderlabs/registrator:latest
  volumes:
    - "/var/run/docker.sock:/tmp/docker.sock"
  links:
    - consul

Registrator is a service that automatically finds all docker containers running on the current server and registers them with Consul (this includes consul). Note the -internal option that we pass in thats very important as it allows Registrator to register the details on services that are not exposed publically but are exposed through Dockers link functionality, this means that when we run our web services they will only be accessable through the publically exposed nginx load balancer. This becomes more important when you add in database servers as it means the database server does not need to be publicly accessible.

Now if you run docker-compose up -d again and navigate to http://<your ip>:8500/ui in your web browser you'll see that now the Consul service is listed :)

Configuring nginx

Next we need to configure nginx to be our load balancer, to do this we need our nginx config to be updated whenever a web service is added or removed from Consul. In order to do this we'll use consul-template but there is no nginx image on hub.docker.com that includes consul-template and works reliably, fortunatly its not hard to make our own. The first thing to do is create a folder called lb this will house all the files we to to build our nginx and consul-template load balancer.

Next create a file called Dockerfile and add the following to it.

#Get the latest nginx image
FROM nginx:latest

#nginx does not include unzip so we need to add it
COPY unzip /usr/bin

#Install Consul Template
ADD https://releases.hashicorp.com/consul-template/0.12.2/consul-template_0.12.2_linux_amd64.zip /usr/bin/
RUN unzip /usr/bin/consul-template_0.12.2_linux_amd64.zip -d /usr/local/bin

#Setup Consul Template Files
RUN mkdir /etc/consul-templates
ENV CT_FILE /etc/consul-templates/nginx.conf

#Setup Nginx File
ENV NX_FILE /etc/nginx/conf.d/app.conf

#Default Variables
ENV CONSUL consul:8500
ENV SERVICE consul-8500

# Command will
# 1. Write Consul Template File
# 2. Start Nginx
# 3. Start Consul Template

CMD /usr/sbin/nginx -c /etc/nginx/nginx.conf \
& consul-template \
  -consul=$CONSUL \
  -template "/etc/consul-templates/nginx.ctmpl:$NX_FILE:/usr/sbin/nginx -s reload";

That looks like we've done a lot but all were doing is getting instructing docker on how to build out load balancer container. First we get the latest official nginx image then we add the unzip command to the image as it isn't included and we'll need it to extract consul-template. Next we install consul template by adding it to the image from the contul-template url. We then set some environmental variables to the location of the nginx and consul-template config files after that we tell docker which command should be run when the container is started and thats it.

Now it looks like we haven't actually configured nginx or consul-template, which we haven't but there is a good reason for this and thats because the best thing to do when services inside a docker container need config files is to provide them via dockers volumes functionality as this allows you to keep the config files outside the container (where you can modify them easily).

So now its time to create the config files so you'll need to create a new folder inside you lb folder (where your Dockerfile is) call it templates and then create a new file inside it called lb.ctmpl this is going to form your consul-template config which in turn will be used to generate your nginx config.

Now you've got your lb.ctmpl file open it and add the following to it.

server {
  listen 65333;

  location / {
    types {
      application/json json;
    }
    default_type "application/json";
    return 501 '{
      "success": false,
      "deploy": false,
      "status": 501,
      "body": {
        "message": "No available upstream servers at current route from consul"
      }
    }';
  }
}

{{range services}}
  {{ if .Tags.Contains "production" }}
  upstream {{.Name}} {
    least_conn;
    {{range service .Name}}
    server  {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;
    {{else}}server 127.0.0.1:65535;{{end}}
  }
  {{end}}
{{end}}


server {
  listen 80;
  
  {{if service "web"}}
  location / {
    proxy_pass http://web;
  }
  {{end}}
}

That looks a bit complicated but really all we are doing is defining a server that will be a default fallback in case there are no web services available and thats just standard ngix config. Next the magic happens as we tell consul-template to find all the registered services that consul has then filter that list by the services tagged with production and for each of them add an upstream service with the same name as the consul service (web in our case) that load balances by the server with least connections we then set some extra settings for failover and define a service in port 80 that proxy passes all requests to the web upstream services.

Next we need to add our load balancer to our docker-compose.yml so add the following.

lb:
  build: ./lb
  links:
    - consul
  ports:
    - "80:80"
  volumes:
    - ./lb/templates/lb.ctmpl:/etc/consul-templates/nginx.ctmpl
  environment:
    SERVICE_80_CHECK_HTTP: /
    SERVICE_80_CHECK_INTERVAL: 5s
    SERVICE_80_CHECK_TIMEOUT: 1s

This is slightly different from the other services we've added to the docker-compose.yml so far because in this one we don't have a pre-built image so instead we tell docker to build an image based on the Dockerfile we created eariler. We then link this container to consul as it'll need to be able to acces the consul container and we expose port 80 publicly as port 80 so that you can visit the site. Now some magic happens as we use volumes to map the config file in our lb folder to the location in the container where we said the config file was ;).

We then define a set of health checks that Consul can use to determine if this service is healthy, so in this case we are telling consul that it should try to hit the root of the site over http on port 80 every 5 seconds and if it takes more than 1 second to reply or sends anything but a 200 response back the service is unhealthy. I'll write another post in future that shows you how to use this information to restart unhealthy services.

Adding our web service

Up till now we've created a service discovery service that can discover itself and a load balancer but we still can't actually navigate to our web service so next we'll add the web service and you'll have a fully load balanced and scalable web service that you can start on any docker host regardless of the hosting provider.

For simplicity we're going to use a prebuilt web service container called tutum/hello-world this simply serves up a web page with Hello World on it but it'll do for our purposes. You could of course replace this container for your own (I'll be covering creating your own container in another article).

Add the following to your docker-compose.yml.

web:
  image: tutum/hello-world
  ports: 
    - 80
  links:
    - consul
  environment:
    SERVICE_NAME: web
    SERVICE_TAGS: production
    SERVICE_3000_CHECK_HTTP: /
    SERVICE_3000_CHECK_INTERVAL: 5s
    SERVICE_3000_CHECK_TIMEOUT: 1s

You'll also need to modify the load balancer config (lb) like so.

lb:
  build: ./lb
  links:
    - consul
    - web
  ports:
    - "80:80"
  volumes:
    - ./lb/templates/lb.ctmpl:/etc/consul-templates/nginx.ctmpl
  environment:
    SERVICE_80_CHECK_HTTP: /
    SERVICE_80_CHECK_INTERVAL: 5s
    SERVICE_80_CHECK_TIMEOUT: 1s

As you can see we've added a link to web.

Now if you run docker-compose up -d and navigate to http://<your ip>/ you should see a page with hello world on it.

Scaling your service

To scale your new service or any docker-compose service simply run

docker-compose scale web=10

where web is the name of the service you want to scale and the number is the number of containers you want to scale to. This can be more or less than the current running containers, if it is less then the current number of running containers then docker-compose will destroy excess containers.