docker-gen: automatic nginx config with a human touch

In this article I will demonstrate a setup using Docker, docker-gen and nginx that will add new Docker containers to nginx automatically as an upstream instruction, but still let you manually configure how these containers are publicly served by writing the site instructions yourself.

Background

In an earlier article I set up my server with Docker and Dokku. One of the things I showed is how to deploy Docker containers on your server and serve them on a specific URL by using nginx as a reverse proxy. I automated this by using docker-gen. Docker-gen watches your Docker containers, and automatically generates nginx config when new containers are deployed or taken down. I used a config template from jwilder’s nginx-proxy container, which works pretty simply: you set an environmental VIRTUAL_HOST on your container, and docker-gen creates all the necessary nginx config to host your container on this URL. For example:

docker run \
       -e WORDPRESS_DB_USER=foo -e WORDPRESS_DB_PASSWORD=bar \
       -e WORDPRESS_DB_NAME=foo -e VIRTUAL_HOST=blog.foo.com \
       --link mysql:mysql --restart=always wordpress

This command is all you need to run (provided you’ve already set up your database of course) to deploy a new Wordpress blog and serve it on blog.foo.com. However, after having used this process a number of times, I found that it has a couple of drawbacks:

I can’t change the URL my container is hosted at: I have to destroy and re-create the container with a different VIRTUAL_HOST value
I can’t configure nginx for individual containers: every rule I add has to go into my configuration template and is then applied to every nginx site that docker-gen generates

Fortunately, there is a simple solution for this. The nginx config generated by this template consists of two components: an upstream and a site. The simple solution then is to have docker-gen generate the upstream instructions, but write the site by hand. This method is flexible enough to deal with the container’s randomly assigned IP address yet still allows you to configure your site any way you want. Here is the new way to deploy a Wordpress blog. I create a Wordpress container and give it a new environmental: NGINX_UPSTREAM rather than VIRTUAL_HOST:

docker run \
       -e WORDPRESS_DB_USER=foo -e WORDPRESS_DB_PASSWORD=bar \
       -e WORDPRESS_DB_NAME=foo -e NGINX_UPSTREAM=fooblog \
       --name fooblogcontainer --link mysql:mysql \
       --restart=always wordpress

docker-gen automatically updates the config file /etc/nginx/conf.d/docker_upstream_hosts.conf:

upstream fooblog {
        # fooblogcontainer
        server 172.17.0.42:80;
}

Now I write the file /etc/nginx/sites-available/com.foo.blog by hand:

server {
        server_name blog.foo.com;

        access_log  /var/log/nginx/com.foo.blog_access.log;
        error_log   /var/log/nginx/com.foo.blog_error.log notice;

        include global/restrictions.conf;
        include global/wordpress.conf;

        location / {
                proxy_pass http://fooblog;
        }
}

I link to this new file from /etc/nginx/sites-enabled and then reload nginx. Now my new Wordpress site is available at blog.foo.com. Yes, it is more work than simply starting the container with VIRTUAL_HOST=blog.foo.com to begin with, but this way allows me to apply additional nginx configuration that applies to Wordpress blogs but no other web applications. The two included files for example are config files with additional security rules and additional Wordpress rules respectively. You want this, right? Otherwise you wouldn’t still be reading this. Don’t worry! I’m going to show you how I did it.

Setup

VIRTUAL_HOST

I wanted to keep the old setup that triggers on VIRTUAL_HOST as well as the new mechanism, so I moved around some files. I kept the docker-gen executable in /etc/nginx/docker-gen (see my previous article). I moved the config template (which is a straightforward copy of jwilder’s nginx-proxy template) to /etc/nginx/docker-gen-vhost-template. The command that runs docker-gen with this template is embedded in a script at /etc/nginx/docker-gen-vhost-service with this content:

#!/bin/bash
/etc/nginx/docker-gen -only-exposed -watch -notify "service nginx reload" \
      /etc/nginx/docker-gen-vhost-template \
      /etc/nginx/sites-available/docker_virtual_hosts

I put the output config file in sites-available and link to it from sites-enabled so that I can disable auto-generated site config if needed. Now I make an upstart job for this script (feel free to use another service manager) in /etc/init/docker-virtualhosts.conf:

# docker-virtualhosts - Nginx config generator for Docker containers
# using the VIRTUAL_HOST env

description "Nginx vhost config generator for Docker containers"
author "Sombody <sombody@foo.com>"

start on filesystem and started docker
stop on runlevel [016]
respawn
console log

pre-start script
    [ -d /etc/nginx/certs ] || mkdir -p /etc/nginx/certs
end script

exec /etc/nginx/docker-gen-vhost-service

Now I simply run initctl start docker-virtualhosts and my config generator service is up.

NGINX_UPSTREAM

The setup that triggers on NGINX_UPSTREAM is even simpler. I wrote this config template to /etc/nginx/docker-gen-upstream-template:

{{ range $host, $containers := groupByMulti $ "Env.NGINX_UPSTREAM" "," }}

upstream {{ $host }} {
    {{ range $container := $containers }}
        {{ $addrLen := len $container.Addresses }}
        {{/* If only 1 port exposed, use that */}}
        {{ if eq $addrLen 1 }}
                {{ with $address := index $container.Addresses 0 }}
                   # {{$container.Name}}
                   server {{ $address.IP }}:{{ $address.Port }};
                {{ end }}
        {{/* If more than one port exposed, */}}
        {{/* use the one matching VIRTUAL_PORT env var */}}
        {{ else if $container.Env.VIRTUAL_PORT }}
                {{ range $address := .Addresses }}
                   {{ if eq $address.Port $container.Env.VIRTUAL_PORT }}
                   # {{$container.Name}}
                   server {{ $address.IP }}:{{ $address.Port }};
                   {{ end }}
                {{ end }}
        {{/* Else default to standard web port 80 */}}
        {{ else }}
                {{ range $address := $container.Addresses }}
                        {{ if eq $address.Port "80" }}
                        # {{$container.Name}}
                        server {{ $address.IP }}:{{ $address.Port }};
                        {{ end }}
                {{ end }}
        {{ end }}
    {{ end }}
}

{{ end }}

Then I wrote the script /etc/nginx/docker-gen-upstream-service:

#!/bin/bash
/etc/nginx/docker-gen -only-exposed -watch -notify "service nginx reload" \
      /etc/nginx/docker-gen-upstream-template \
      /etc/nginx/conf.d/docker_upstream_hosts.conf

And created an upstart job in /etc/init/docker-upstreams.conf exactly like the last one but executing /etc/nginx/docker-gen-upstream-service. The output config file /etc/nginx/conf.d/docker_upstream_hosts.conf is loaded by default by nginx, so I can reference the generated upstreams from my hand-written site instructions like I’ve shown you. In case you’re curious, the additional config files I used in my site example look like this: /etc/nginx/global/restrictions.conf:

# Global restrictions configuration file.
# Designed to be included in any server {} block.
location = /favicon.ico {
        log_not_found off;
        access_log off;
}

location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
}

# Deny all attempts to access hidden files such as .htaccess, .DS_Store (Mac).
# Keep logging the requests to parse later
location ~ /\. {
        deny all;
}

And /etc/nginx/global/wordpress.conf:

# Deny access to any files with a .php extension in the uploads directory
# Works in sub-directory installs and also in multisite network
# Keep logging the requests to parse later
location ~* /(?:uploads|files)/.*\.php$ {
   deny all;
}

client_max_body_size 8M;

You may also want to include the restrictions.conf file in your VIRTUAL_HOST config template as well. So, if you’ve been following along with this tutorial, you should have these files:

/etc/init/docker-virtualhosts.conf
/etc/init/docker-upstreams.conf
/etc/nginx/docker-gen
/etc/nginx/docker-gen-upstream-service
/etc/nginx/docker-gen-upstream-template
/etc/nginx/docker-gen-vhost-service
/etc/nginx/docker-gen-vhost-template
/etc/nginx/global/restrictions.conf
/etc/nginx/global/wordpress.conf
/etc/nginx/sites-enabled/docker_virtual_hosts
   -> /etc/nginx/sites-available/docker_virtual_hosts

These files should be generated automatically:

/etc/nginx/conf.d/docker_upstream_hosts.conf
/etc/nginx/sites-available/docker_virtual_hosts

And to deploy blog.foo.com you should write /etc/nginx/sites-available/com.foo.blog and link to it from /etc/nginx/sites-enabled/.

Background

Setup

VIRTUAL_HOST

NGINX_UPSTREAM

Share what you've learned