Docker swarm — services and high availability
Docker swarm allows you to deploy services distributed to the nodes of the swarm, this allows high availability and high performance of…
Docker swarm — manage services and high availability
Docker swarm allows you to deploy services distributed to the nodes of the swarm, this allows high availability and high performance of the service. To start a single service we can enter something very simple like this
Create a simple service and cause a on the running node$ docker service create nginx
spesyzvgu8pkh1eidhpjcti7d
overall progress: 1 out of 1 tasks
1/1: running [==================================================>]
verify: Service converged
This command started a new service with the container nginx.
To see details about the services we can run$ docker service list
ID NAME MODE REPLICAS IMAGE PORTS
spesyzvgu8pk nifty_lehmann replicated 1/1 nginx:latest
Note that the replicas are 1 and does not equal the number of my worker nodes which are 3 in my setup, This is because by default docker swarm creates one replica but still enjoys high availability, in case of a failure of the node that runs this container the service will move to another node. Lets first find on which node it runs$ docker service ps nifty_lehmann
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
md9qwxb69y56 nifty_lehmann.1 nginx:latest worker_node3 Running Running 5 minutes ago
To do this we need to pass the name of the container to docker service ps which is nifty_lefmann.1 in mycase and the host of the swarm that executes the container is worker_node3.
Lets simulate a real world scenario, lets stop docker on this worker which equals to a problem in the node.worker_node3$ sudo systemctl stop docker
We can verify the status of the nodes with docker node ls. Using this command we can see that the status of workder_node3 is “Down”.$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
ncold3meymr0sl9pd9v71dweb * swarm_manager Ready Active Leader 18.09.5
d913ucvquyjigsl4mnwmcxn4s worker_node2 Ready Active 18.09.5
ch02clvvojp6qosy20i9k9nim worker_node3 Down Active 18.09.5
Now lets see the state of the service from the swarm managerswarm_manager$ docker service ps nifty_lehmann
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
pqkvpdxtyumm nifty_lehmann.1 nginx:latest swarm_manager Running Running 26 seconds ago
md9qwxb69y56 \_ nifty_lehmann.1 nginx:latest worker_node3 Shutdown Running 46 seconds ago
Here we can see that the container is in shutdown state on worker_node3 and in running state in the swarm_manager, this is happening because if not configured the swarm managers are workers as well
Lets bring back to life docker on node worker_node3worker_node3$ sudo systemctl start docker
Verify that the node is backswarm_manager$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
ncold3meymr0sl9pd9v71dweb * swarm_manager Ready Active Leader 18.09.5
d913ucvquyjigsl4mnwmcxn4s worker_node2 Ready Active 18.09.5
ch02clvvojp6qosy20i9k9nim worker_node3 Ready Active 18.09.5
Lets see what happens to the service$ docker service ps nifty_lehmann
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
pqkvpdxtyumm nifty_lehmann.1 nginx:latest swarm_manager Running Running 14 minutes ago
md9qwxb69y56 \_ nifty_lehmann.1 nginx:latest worker_node3 Shutdown Shutdown 8 minutes ago
Nothing changes! despite that the node is up the container runs in the swarm_manager node.
Delete a service
To delete a running service enterswarm_manager$ docker service rm nifty_lehmann
nifty_lehmann
Its very simple! it should return the name of the service
Run a service with more than one replicas
Lets do something more complicated, lets define the number of replicas of the service along with some optionsswarm_manager$ docker service create --name nginx --replicas 3 -p 8080:80 nginx
x8cgelqviid3e5l4icybsclg2
overall progress: 3 out of 3 tasks
1/3: running [==================================================>]
2/3: running [==================================================>]
3/3: running [==================================================>]
verify: Service converged
With this command we did the following
- The name of the service is nginx
- The number of replicas are 3
- We exposure port 80 of each container to port 8080 of each node
Running docker service ps nginx we can see the following$ docker service ps nginx
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ek8llokh8ia8 nginx.1 nginx:latest swarm_manager Running Running 7 minutes ago
st654bmojc4u nginx.2 nginx:latest worker_node2 Running Running 7 minutes ago
copaz24xs2zf nginx.3 nginx:latest worker_node3 Running Running 7 minutes ago
Each node has a running container for service nginx
How Load balancing works
Lets do something very interesting, run this on any host of your swarm that is a worker nodefor i in {1..100}; do curl -l localhost:8080 &> /dev/null; done
Then run the following in a swarm nodedocker service logs nginx | grep -i http | cut -d " " -f1 | sort -n | uniq -c
No matter how many times you will run it and no matter on which host you should see that the number of requests should be divided equal to each worker node that runs this container184 nginx.1.ek8llokh8ia8@swarm_manager
184 nginx.2.st654bmojc4u@worker_node2
182 nginx.3.copaz24xs2zf@worker_node3
And this is happening because docker swarm internally load balance the requests in a round-robin fashion despite on which node we perform the request and this is great for simple solutions that do not require a very specialized load balancing scheme.
Changing the number of replicas
We can on the fly increase or decrease the number of running replicas for a service, to do this we can enter in a swarm manager$ docker service update --replicas 2 nginx
nginx
overall progress: 2 out of 2 tasks
1/2: running [==================================================>]
2/2: running [==================================================>]
verify: Service converged
This will reduce the number of nginx replicas from 3 to 2, we can verify this by running$ docker service ps nginx
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ek8llokh8ia8 nginx.1 nginx:latest worker_node2 Running Running 30 minutes ago
st654bmojc4u nginx.2 nginx:latest swarm_manager Running Running 30 minutes ago
To increase the number of replicas$ docker service update --replicas 6 nginx
nginx
overall progress: 6 out of 6 tasks
1/6: running [==================================================>]
2/6: running [==================================================>]
3/6: running [==================================================>]
4/6: running [==================================================>]
5/6: running [==================================================>]
6/6: running [==================================================>]
verify: Service converged
$ docker service ps nginx
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ek8llokh8ia8 nginx.1 nginx:latest worker_node2 Running Running 32 minutes ago
st654bmojc4u nginx.2 nginx:latest swarm_manager Running Running 32 minutes ago
xpw6gl4sos46 nginx.3 nginx:latest worker_node3 Running Running 15 seconds ago
9m286xjd6mnn nginx.4 nginx:latest swarm_manager Running Running 15 seconds ago
zkkaxktfozz6 nginx.5 nginx:latest worker_node2 Running Running 15 seconds ago
qxvqa3svz6js nginx.6 nginx:latest worker_node3 Running Running 14 seconds ago
Note that the containers have been equally balanced to each running node
How to disallow a node running containers
A good practice is not to use swarm managers to run heavy load containers because this can disrupt the consistency of the swarm managers statuses which can cause problems to the quorum of the swarm, to avoid this you can configure a node not to run containers$ docker node update --availability drain swarm_manager
swarm_manager
And to verify that the swarm_manager does not execute any containers we run docker service ps nginx$ docker service ps nginx
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ek8llokh8ia8 nginx.1 nginx:latest worker_node2 Running Running 38 minutes ago
pnvff40fy745 nginx.2 nginx:latest worker_node3 Running Running 56 seconds ago
st654bmojc4u \_ nginx.2 nginx:latest swarm_manager Shutdown Shutdown 57 seconds ago
xpw6gl4sos46 nginx.3 nginx:latest worker_node3 Running Running 6 minutes ago
wu3u0lw42gq9 nginx.4 nginx:latest worker_node2 Running Running 57 seconds ago
9m286xjd6mnn \_ nginx.4 nginx:latest swarm_manager Shutdown Shutdown 57 seconds ago
zkkaxktfozz6 nginx.5 nginx:latest worker_node2 Running Running 6 minutes ago
qxvqa3svz6js nginx.6 nginx:latest worker_node3 Running Running 6 minutes ago
We can see that swarm_manager does not execute any containers but the number of replicas are still 6 and distributed equal to the 2 workers.
How to automatically use all nodes of a swarm
In case you want to use all nodes of the swarm there is not need to define the number of replicas, you can instead use the global option
docker service create --name nginx --mode global nginxThis command will create as many replicas as the number of nodes that can run containers in the swarm.
I hope you found my article interesting! :)
