One Big App (High-Availability) #
You have an application (which is composed of multiple Docker services) and want to increase its availability by distributing it across multiple nodes so that one node can go down and your application stays online.Setup #
Let’s say we have three nodes:
node001
node002
node003
Our app consists of two two stateless services (http
and api
), and one
database (etcd in this case, a distributed and fault-tolerant key-value store).
General Considerations #
The Docker Swarm setup will look very similar to the one tuned for high-performance with two important differences:
Traffic Ingress #
The ingress/load-balancer needs to be able to tolerate a node going down. And it obviously also can’t just run on one single node.
If you order your cluster with our managed Traefik loadbalancer, this part will be taken care of for you. For custom setups, you’ll have to watch out for that yourself.
Stateful Services #
The same sort of thing goes for databases and storage engines and similar. You’ll probably want to configure some sort of multi-master system.
Details differ widely, depending on what database you use. We’ll be using etcd here as an example.
Docker volumes are always specific to each node. Any service using volumes must be pinned to a specific node - so it always stays where the data is.
The basic idea is to start databases as multiple services, one for each node, and set them up as a cluster.
Stack.yml #
Putting it all together:
|
|
In detail:
x-etcd-cluster-config: ...
-x-*
keys are ignored by Docker Swarm - this sets up a YAML-Anchor so we can avoid duplicating all these values for each etcd.http
andapi
replicas: 3
- we start 3 containers in total..max_replicas_per_node: 1
- …one on each node
etcd1
,etcd2
,etcd3
- one etcd service per node, set up to form a cluster
- placement constraint
node.hostname == node00X
: each one of them is pinned to one of the nodes - volumes:
etcd1_data
,etcd2_data
,etcd3_data
: we’ve given each etcd its own volume name - that makes it clear that these are actually not the same volumes.
Even if we named all of those
etcd_data
, they would still be three different volumes:
etcd_data
on node001etcd_data
on node002etcd_data
on node003The different names we used just make that explicit.
Behavior #
Now, if one node goes down, the application will continue to work:
- we still have two
http
services, and theirapi
services, on the remaining nodes - the etcd cluster has two of three nodes left, which is a state in which etcd is still fully functioning
In practice, we recommend actually testing the failure scenarios. Complex systems like clustered databases sometimes behave in complex ways.
Some things to look out for:
Some databases, even in a clustered configuration, enter a “degraded” state if a node fails. So in some cases, you might need to write your app such that it can deal with a read-only DB.
Other databases don’t have automatic recovery. That is, the failed node will come up again, but it won’t join the database cluster. That may not lead to immediate failures - but if later another node goes offline, the now-2-node cluster can probably not handle that failure gracefully.