Code: Simulating network links with Docker
Available on GitHub.
Problem
Quite often I need to test a multi-robot setup with degraded network links, most recently in DexROV and before in Spacebot Cup. This can be done well with products such as WANem (which was used in Spacebot Cup). The big drawback is that a physical computer with multiple network interfaces is required, while of course the advantage is no further modification to the system under test. Such a thing is great for larger tests, once the system is up and running. For daily distributed development in a team, a more convenient and scalable solution is needed as the above is often hardware-bound.
With the advent of linux containers (see e.g. Docker and LXD), we actually have a great tool for developing for distributed systems, but it still does not allow for testing under network limitations. We focus on Docker, as it is the most generally supported implementation of containers with a large community and comes with some related tools.
Solution
The internal working of WANem is quite simple, it uses standard traffic shaping tools included in the Linux kernel. Specifically, netem and the Hierarchical Token Bucket filter are used to reduce bandwidth, introduce delays, simulate packet drops and duplications as well as bit errors in the payload.
All these tools are also applicable in a local link, as they work directly on outgoing data of a network interface, no matter if it is physical or virtual. We would like to re-use the quite nice Docker Networking feature, where containers can be added and removed from networks at runtime. This is in direct contrast to rather static configurations allowed with tools such as IMUNES. Local bridge networks generated by Docker can be connected with a virtual pair of network interfaces, on which traffic shaping rules can be applied.
Implementation
mini-network-simulator implements such a solution with only four shell scripts.
Two scripts exists to make or break connections between existing Docker networks.
One script updates the routes and writes to the /etc/hosts file of each running and connected container, so the new connection can be used.
A final script can apply netem rules to a given existing connection.
Note that multiple networks can be connected in a tree. Loops are not supported, as the configuration of the networks can be likened to a number of switches (the Docker networks, which are not routers!) connected by cables (the links established by mini-network-simulator).
Example
# download scripts and cd into the new directory
git clone https://github.com/maxpfingsthorn/mini-network-simulator.git
cd mini-network-simulator
# create networks
docker network create test1
docker network create test2
# connect networks
sudo ./connect-networks.sh test1 test2
# degrate link (can be run anytime)
sudo ./degrade-link.sh test1 test2 --delay 200ms
sudo ./degrade-link.sh test2 test1 --delay 100ms
# run docker containers
docker run -itd --net=test1 --name test1cont --hostname test1host ubuntu:trusty bash
docker run -itd --net=test2 --hostname test2host ubuntu:trusty bash
# update routes and hosts of containers
sudo ./update-routes-and-hosts.sh
# attach to first container's bash by name (that's why we gave it one, it was optional)
docker attach test1cont
# you are now connected to the bash running in the first container, and can ping the other from here
ping test2host