How to allow Docker containers to see the source IP address

I’m sorry as I’m about to tell you the full story and all my sufferings getting this to work, but I need to do it. If you just want to know how to solve it, just scroll down bellow to the “TL;DR” chapter; I’ll explain there how to fix the issue, and you’ll avoid the 3 hours of suffering and pain that I had.

Most important thing to be aware about: by default, containers don’t see which source IP address is connecting when they have a listening port, so every logging, banning, filters and firewalls are just pointless. Unless you don’t care about whom is connecting to your containers, this is something that needs to be solved.

…fear leads to anger, anger leads to hate, hate leads to suffering.

I was working on a Docker article about filtering IP addresses that can access to the container and I ended really salty about NAT, networks, iptables and with myself.

Just finished adding my session cookie with a variety of information to verify the user authenticity, as the login date, user agent and IP address used for login. When I checked out I saw that Flask (python web framework) does not return the IP address of my container but the gateway address from Docker.

I went to the documentation, StackOverflow and Google, and I get only one solution. When I tried that out got same results. Thought about if Nginx is not providing the address properly, so I tested another bunch of other stuff. Neither. Then I saw the Nginx logs with the same gateway address!

Crap, this is not going to end well…

Googled why nginx doesn’t see the source address in containers and there’s lots of Docker users with same problem. Docker uses to do this on its containers. So I started using “lsof” and “nc” to verify it. And sure, lsof command actually reports the gateway as source address instead of the actual host that performed the call.

More searches led me to know that I can either move Nginx to a network of “host” type instead of “bridge”, or disable something called “userland-proxy”. Changing the network doesn’t seem a good idea, so I researched more on the userland-proxy. It turns out it has to be done for Docker itself and not by container, so I went back to try to change the network to “host”. This has to work, surely, but puts the container on the same network as the host, which I don’t like.

But when I start my containers I noticed that Docker Compose does not allow that a container is in both, the host network and bridge network at the same time. As I have several containers I was expecting to only expose one and leave everything else as “bridge”. But, wait, anyway if the Nginx container is in “host” mode, then it must have access to any other container, right? Well, yes, but no. It turns out that in this mode Docker does not create the hostnames of the containers in /etc/hosts so I can’t resolve container addresses.

So again, back to Google to research about creating some sort of DNS for that. Every solution is either really convoluted or with software that needs to be installed in the host, outside of Docker. In this moment I realized that my FTP containers will suffer from the same. Will I put the FTP containers in the host network? Hell, no! Not a chance. If in my quest for more security I have to open the network in most containers I’m doing something really wrong. So here is where I started to get really salty.

Went to iptables to continue debugging with lsof/ncat and try to understand what the hell Docker is doing. Everything it needs to do is a stupid NAT from the host IP address to the container IP address. That should be 1-4 iptables rules more or less. This is done by any domestic router and our computers can see who is connecting from the other side of the network. What is Docker doing with the iptables? Had to review what a “masquerade” was again and inspect carefully iptables to see if there was anything out of place. Nothing that I could tell.

And then I realize that lsof displays a process “docker-proxy” that seems to be a middleman between the host and the container, routing packages manually. Literally like an implementation of “socat” itself, reading packages on one side and sending them in the other. That would explain why the packages seem to come from inside. Because they actually are! How do I tell Docker to stop doing this crap and just do a standard NAT?

After researching about “docker-proxy” and finding around 50 links about docker containers with proxies, I could find some articles that explain that this process is the responsible of doing the “userland-proxy”. This is the same as I found two hours before, but thankfully, if it can be disabled globally, and now that I understand the crappy Docker default NAT , it would be way better to just remove it. Now.

Removing it is simple, just add {“userland-proxy”: false} to “/etc/docker/daemon.json” and restart Docker. Works? Well, I still see the same thing again. The source address seems to be an internal gateway from docker-proxy. Searching again seems that doesn’t work for a bunch of people and gives some problems to others. Finally I found in GitHub issues another project using Docker saying that sometimes the NAT rules start up late, the connection might be done earlier and the routing gets stored and reused. So the solution is to use “conntrack -D”. But also doesn’t work. Well, just restart the host and that should fix it…

You might have guessed already: it didn’t work. And now there’s another process listening for connections for routing, in this case the main Docker process. After reviewing the iptables again I gave up: I’ll add rules myself.

One single rule by its own is not enough for NAT, not expecting that it will work, but at least iptables should count the incoming packets in it. But they weren’t moving. I wasn’t able to hit my own rule.

At this point I started realizing that there was something fundamental that I was missing from the start. I remembered how some routers fail to NAT properly when the source address is inside, and the iptables rules to solve that are really convoluted. Could be this? All this time spend only because I’m testing from the same host instead of using another computer?

Went a bit in shock. From one side, I wanted to be this the culprit and solve it finally. But from the other, I was desiring not having lost three hours of my life by missing such a stupid thing. So I went to my laptop, opened the shell and wrote “nc 192.168.1.10 1026” and it connected. What?? But this doesn’t work from my main computer!

Checked the counters of my iptables rule and yes, it counted properly and redirected. Not only that, also the connection was damn perfect. With my poker face I went to try directly to Nginx at port 80 and see what IP address appears as the source, without messing with iptables myself.

And worked. Logs show the IP from my laptop; but if I do the test locally I see the docker gateway IP. This explains it perfectly, seems that rules from PREROUTING chain don’t execute when the connection is from inside.

So what’s wrong if local connections appear as 127.0.0.1, 192.168.1.10 or whatever; everything is local and for our intentions is virtually the same. For external addresses it does work. Victory, but bittersweet.

(NOTE: If this seemed so damn long let me add that I missed more than half of the tests that I ran and terms that I searched. Three hours of panicking really allow for a lot of stuff to be tried)

TL;DR

Let’s explain how to solve this. Basically we have to disable “userland-proxy”. It is enough if we create the file “/etc/docker/daemon.json” with the following content:

{
"userland-proxy": false
}

Then, restart Docker:

$ sudo service docker restart

And now the most important hint ever: you have to test it FROM OTHER COMPUTER.

Some users reported problems with IPv6 after doing this, others can’t start Docker. But all cases seem to have something in common: Either they were running CentOS/RedHat or a really old Ubuntu (12.04).

If the Linux distro is recent it should work properly. For me it worked without any problem at all. CentOS and RedHat publish releases every four years and have support for ten, so it is not strange to see servers with an OS more than ten years old. So this explains why they have problems.

Well, hope I have saved you from a few hours or digging and panicking!

Introduction to Docker Compose and Hub

Again, translating another old post in Spanish to English. In this one we’re going to continue talking about Docker, giving a more technical overview as well as having a look on Docker Compose.

What is Docker?

In my previous post explained very gently what is Docker. Let’s give it a quick refresher.

Docker is an interface for application containers that uses LXC under it. And a container is a sort of a better chroot. Applications will be executed on the same box without any intermediate layer, but using kernel security measures to sandbox it from the host. It’s midway between a Virtual Machine and a Chroot.

On Docker, containers are application containers, which means that they only contain the minimal files for an application to run and no services. If you prefer regular containers with services, have a look to LXC.

Security aside, my favourite advantage of using Docker is that allows local environments to be exactly the same as on the server. Because of this we can avoid the phrase “works on my computer” and the differences between development environment and server are mostly nullified.

What is Docker Compose?

Compose is a simple application for orchestrating Docker containers in a single machine that is well integrated with Docker itself.

If we want to execute a handful of services as different containers where they have to interact between them, Docker Compose is responsible of configuring them, briging them up and keeping them alive so everything goes like a charm with no effort.

For example, in a LAMP server we sould have Apache, PHP and MySQL. With Compose is easy to bring up all three containers and establish communications between them. Also, it provides an easy way of autostart them when the server reboots.

About Docker Hub

Docker has a site called “Docker hub” in https://hub.docker.com/, similar to GitHub, this is a place where people can upload ther images.

Also, Docker has its own official account and provides official images. It can be easily identified because its username is underscore: “_”. For example if we search for Redis, the first result is:

https://hub.docker.com/_/redis

Already made images are really useful because creating them ourselves takes time and complexity. But we have to consider that we’re using third party binaries and as with anything on internet we should be careful.

Because of this, the official images are the best option; next we could have a look on varified authors.

What makes Docker Hub an excellent service is that they provide building pipelines for free to registered users as long as the code for them is stored publicily in GitHub. This allows anyone having an account on both services to have a CI/CD pipeline for Docker. We can do “git push” an Docker Hub will build the image automatically.

This free CI/CD has a good unexpected effect: A lot of people uses it, so there is a strong guarantee that the image is untampered and build exactly from the GitHub source. Also it allows us to fork them and custom the build to our purposes. But still, beware because malware can be hidden in plain sight on the sources, or whatever is good today might be infected tomorrow.

Anyway the Docker company makes a huge effort to secure their repository and as clean as possible from malware.

How does Docker work in practice?

Docker can be installed on Debian-alike distros via “apt-get”. If you go to Docker website you’ll find a guide that recommends their own repositories for this. I also recommend this approach as Docker is changing month by month and it will give you the best experience possible.

https://docs.docker.com/install/linux/docker-ce/debian/

Once installed, beware that you need either to be root or your user has to be inside the group “docker”.

The simplest command is:

docker run -it --rm [image-name]

For example if we execute this command with “danielkraic/asciiquarium” we’ll see a nice aquarium in ASCCI art.

This “run” command creates a container with the specified image. The arguments “-it” tell Docker to create an interactive terminal; in case it is just for a service, this is not needed. The “–rm” argument tells docker to remove the container after has finished.

Also, it will download the image from Docker Hub if we don’t have it, which is really nice and simplifies the process a lot.

If we want to see which containers are running, we will use “docker ps”. If we want to stop them, “docker stop”. To see which images do we have downloaded, “docker images”.

Dockerfile

With Docker, to create new images we use files called “Dockerfile”. Those contain instructions on how to generate an image. Start by a parent image that already exists, for example a basic one for Debian and tell Docker to follow your installation steps, configure and finish the container so it is ready to start your desired application.

On running “docker build” it will execute those steps and will save it as an image named as you specified; then you can use it to create as many containers from it as you want. Also, you can upload the image to Docker Hub for others to use it.

For example:

FROM ubuntu:15.04
RUN apt-get update
RUN apt-get install python-psycopg2 cython
RUN pip install cython
COPY . /app
CMD python /app/app.py

With this we can include our application an its dependencies in an image. By default, when a container is created and started it will execute our application specified on the CMD command.

Docker Compose

Compose follows the same declarative idea from Docker, but it uses a YAML file. Here we can specify which images to build, which containers to start, and which shares they need. For example:

version: '3'
services:
master-webserver:
image: nginx:1.14
volumes:
- ./master-webserver/sites-enabled:/etc/nginx/conf.d
- ./www:/var/www
ports:
- "85:85"
- "443:443"
restart: always
php-apache-1:
image: php:7.3-apache-stretch
volumes:
- ./www/php-apache-1:/var/www/html/

Save it as “docker-compose.yml” and upon running “docker-compose up” will start both containers at once, and they will be on the same subnet.

What do you think? Is it interesting? Next articles will be around how to use Compose for creating a secured LAMP server for not so secure applications. Stay tuned!

Gentle introduction on Docker and containers

Following the earlier posts about Docker in Spanish I’m planning to translate them into English for non-Spanish speakers. In this series I explain how to create a LAMP server that is as secure as possible despite the fact of using not so secure applications inside. We’ll see that later. For now, this post is about a gentle introduction on containers. (Original post published on January 3, 2017)

About Docker and containers

Nowadays almost everyone have heard about Docker, but what actually is and what is useful for? Why does have so much press? Let’s have a quick look.

Docker is a solution for application containers, but is not the only one. It is getting famous as its good marketing and great tooling for a particular use case for containers: application containers. This was certainly unexplored by the time that Docker came, and was the first one to popularize them.

Containers are midway between virtual machines (VMWare / Virtualbox) and executing processes directly on the machine. Generally containers are provided from the OS, in our case Linux Kernel. They are a really improved version of chroot where not only the filesystem is limited, but any possible resource, up to the point that a container resembles to a machine on its own.

Regular containers would be OS containers, like our standard chroot, we install a full OS inside a folder and the container makes sure that anything running in this guest OS cannot exit it. Application containers on the other hand are minimal chroots where an application is hosted with minimal libraries, so the target is having just one application jailed. The main difference between OS and application containers is the start-up/shutdown of those. For an OS container we expect to start up their services and put the OS running while we’re using anything inside. For an application container we just expect to start it to run the application, and close everything when the application ends.

Docker is based on lxc/lxd, which in turn is using the kernel container APIs. While lxc is more oriented for OS containers, Docker makes a good tooling for application containers. This is another way of securing applications as for example AppArmor or SeLinux. Also, they can be combined to give even more security, but this requires expertise.

When we run regular applications, there are certain measures to prevent this application to overtake the whole system: file permissions and user space / kernel space separation. While this is effective to some extent, an application can still do a lot of harm to a system if it wants to. For Linux/BSD related OS we use package managers and signatures to prove that our applications are untampered. But still, some applications even being pristine can cause real damage under certain scenarios. For example, scripting languages will run whatever script they are asked for, so given a Web server with PHP or Python, if a third party manages to change any script it could lead to remote access to your computer, stolen credentials and more.

With containers, we present the application with emulated hardware and limited impact. This can be used to limit to which extent a harmful application would damage our host. But Docker instead of presenting to the application a full OS, it only gives the minimal needs so the application has no means of interacting with other apps that might have its own containers.

In short, while lxc or virtual machines limit the damage to a full guest OS, Docker isolates each application in its own container. This gives us even more fine grained control on what can be done per application.

Some advantages over Docker Containers versus Virtual Machines:

  • Do not reserve memory beforehand. Only the memory used is the memory required for the application running as it was a process in the host.
  • Containers only waste just a bit of space as they don’t need a full OS and common OS files between containers are shared. A differential saving is used to allow the container write to those files without affecting others.
  • They execute instructions directly on CPU without middlewares. There’s no translation layer or hypervisor as the Virtual Machines have, so they don’t suffer any of their slowness.
  • Start-up time is almost the same as running the application in the host, as there’s no OS to boot or services when we start our application container.

There are other useful things from Docker aside of sandboxing:

  • You can run the same container in any machine without any configuration. Containers are host-independent.
  • You can run the same container several times in the same computer, so you can run the same application in parallel with minimal configuration.
  • You can run a set of containers in an orchestrated way. For example you can run a LAMP server using three containers in the same subnet (NGINX + PHP/FCGI + MySQL)
  • Easier to run old software in modern computers as you can install old dependencies in a containers. (Or newer software in older computers as long as your kernel is new enough for it)

Anyway, Docker is not a goose that lays golden eggs. It’s a brand new thing that opens the door to lots of things that weren’t possible before, but it’s not the perfect fit for everything. For starters, containerizing an application is not straightforward. Changing the network is hard. And specifically for application containers, data persistence and container upgrade are common problems.

Docker containers are ephemeral, usually when the app closes the container finishes and its state might be deleted or lost (depending on your approach). If we customize settings inside a running container we might have to do it again next week. Nothing unsolvable, but it takes a while to get used to this new way of thinking.

I do believe that containers are the future in several ways. Still it will take some time to settle, but someday we will have them everywhere.