In our previous posts, we explored how Docker containers and Dockerfiles can be used to package applications neatly. But what happens when your applications become more complex, with multiple components and intricate configurations? Imagine building an online store, where you have separate microservices for the frontend, backend, payment processing, order management and analytics. Each of these microservices might be developed using different programming languages and technologies, and they all need to be built, packaged, and configured correctly.
This is where Docker Compose comes to the rescue. It's a powerful tool specifically designed for managing applications that run in multiple Docker containers. You can define your entire application stack in a single YAML file, including each microservice, its configuration, and how they all interact with each other. With Docker Compose, you can spin up your entire complex application with just a single command, making it super convenient for development, testing, CI/CD pipelines, and even production environments.
The essential features of Docker Compose can be grouped into three categories:
- Isolation: Imagine running multiple instances of your application, each completely isolated from the others. Docker Compose makes this possible, allowing you to replicate your entire application stack on various environments like developer machines, CI servers, or shared hosts. This not only optimizes resource utilization but also simplifies management by reducing operational complexity.
- Stateful Data Management: Say your application needs to store data on disk, like a database. Docker Compose takes care of managing the data volumes associated with your containers, ensuring that your data persists across different runs. This makes it much easier to work with applications that rely on persistent storage.
- Iterative Design: Docker Compose works with a clear configuration that defines all the containers in your application. You can easily add new containers to this configuration without disrupting existing ones. For example, if you have two containers running and decide to add a third, Docker Compose won't touch the first two. It will simply create and connect the new container, making it a breeze to expand your application iteratively.
Docker Compose's ability to isolate multiple instances of your application, manage stateful data, and support iterative development makes it a must-have tool for handling complex applications with multiple containers. In this chapter, we'll take a closer look at how Docker Compose can help you manage the entire lifecycle of your application, from setup to deployment.
We'll start by diving deep into the Docker Compose command-line interface (CLI) and the structure of Compose files. Then, we'll explore different ways to configure your applications using Compose and how to define dependencies between different services within your application stack.
Since Docker Compose is a core tool in the Docker ecosystem, gaining both technical knowledge and practical experience with it will be a valuable addition to your skillset. By the end of this chapter, you'll be well-equipped to handle even the most intricate multi-container applications with ease.
Docker Compose CLI
Docker Compose works hand-in-hand with Docker Engine to orchestrate multi-container applications. It uses a command-line tool called docker-compose
to communicate with the Engine. On Mac and Windows, this tool is conveniently bundled with Docker Desktop. However, if you're running a Linux system, you'll need to install docker-compose
separately after installing Docker Engine. The good news is that it's a single executable, making the installation quite straightforward.
You can find instructions on installing the Compose Plugin here.
Docker Compose CLI Commands
The docker-compose command can handle all aspects of an application's life cycle that use multiple containers. It is possible to start, stop, and restart services with the subcommands. You can also see what's going on with the running stacks and get their logs. The following posts will give you hands-on examples with the most important commands. In the same way, the following command can be used to see a sample of all the features:
docker-compose --help
And you should see something like the following:
Usage: docker compose [OPTIONS] COMMAND
Define and run multi-container applications with Docker.
Options:
--ansi string Control when to print ANSI control
characters ("never"|"always"|"auto")
(default "auto")
--compatibility Run compose in backward compatibility mode
--dry-run Execute command in dry run mode
--env-file stringArray Specify an alternate environment file.
-f, --file stringArray Compose configuration files
--parallel int Control max parallelism, -1 for
unlimited (default -1)
--profile stringArray Specify a profile to enable
--progress string Set type of progress output (auto,
tty, plain, quiet) (default "auto")
--project-directory string Specify an alternate working directory
(default: the path of the, first
specified, Compose file)
-p, --project-name string Project name
Commands:
build Build or rebuild services
config Parse, resolve and render compose file in canonical format
cp Copy files/folders between a service container and the local filesystem
create Creates containers for a service.
down Stop and remove containers, networks
events Receive real time events from containers.
exec Execute a command in a running container.
images List images used by the created containers
kill Force stop service containers.
logs View output from containers
ls List running compose projects
pause Pause services
port Print the public port for a port binding.
ps List containers
pull Pull service images
push Push service images
restart Restart service containers
rm Removes stopped service containers
run Run a one-off command on a service.
scale Scale services
start Start services
stop Stop services
top Display the running processes
unpause Unpause services
up Create and start containers
version Show the Docker Compose version information
wait Block until the first service container stops
watch Watch build context for service and rebuild/refresh containers when files are updated
Run 'docker compose COMMAND --help' for more information on a command.
There are three key docker-compose
commands crucial for managing the lifecycle of applications. Here's a breakdown of these commands and how they fit into the overall lifecycle:
-
docker-compose up
: Thedocker-compose up
command initializes and starts the containers as defined in your configuration file. You can either build container images from scratch or use pre-built images available in a registry. For long-running services, like web servers, it’s often practical to run the containers in detached mode by using the-d
or--detach
flags. This allows the containers to run in the background, freeing up your terminal for other tasks. For a full list of options and flags available with this command, you can usedocker-compose up --help
. -
docker-compose ps
: Thedocker-compose ps
command provides a snapshot of the containers and their current status. This is particularly useful for diagnosing issues and performing health checks on your containers. For example, if you have a two-container setup with a backend and a frontend, you can usedocker-compose ps
to see the status of each container. It helps you identify whether any of your containers are down, not responding to health checks, or have failed to start due to misconfiguration. -
docker-compose down
: Thedocker-compose down
command stops and removes all the resources, including containers, networks, images and volumes.
Docker Compose File
The Docker Compose CLI manages and configures multi-container applications. Typically, these settings are saved in a file called docker-compose.yml
. Docker Compose is a powerful tool, but its efficacy is dependent on the quality of the configuration. As a result, understanding how to create and fine-tune these docker-compose.yml
files is critical, requiring careful attention to detail.
A docker-compose.yaml
file consists of four main sections:
name: myapp
services:
- ...
networks:
- ...
volumes:
- ...
Name
Docker Compose specifies that the top-level name property will be used as the project name if you don't set one yourself. You can change this name, and if the top-level name element is not set, Docker Compose will set a default project name that will be used. Also note that the project name
is exposed for interpolation and we can use the COMPOSE_PROJECT_NAME
to access this name
.
name: myapp
services:
some-service:
image: busybox
command: echo "I'm running ${COMPOSE_PROJECT_NAME}"
Services
A service is a general term for a computer resource in an app that can be changed or grown without affecting other parts of the app. A group of containers backs up the services. The platform runs the containers based on replication needs and placement restrictions. A Docker image and a set of runtime arguments describe a service because it is backed by a container. With these arguments, all containers in a service are made in the same way.
name: myapp
services:
server:
image: nginx:latest
ports:
- 8080:80
This section of the docker-compose.yaml
file defines a service named server, which is set up to run a Nginx web server. The image: nginx:latest
directive instructs Docker to utilize the latest version of the Nginx image from Docker Hub. The ports section maps port 8080
on the host system to port 80
within the container. This configuration allows you to reach the Nginx server via http://localhost:8080 on your host machine.
Networks
Networks let services communicate with each other. By default Δοψκερ Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by the service's name. The top-level networks
element lets you configure named networks that can be reused across multiple services.
services:
frontend:
image: example/webapp
networks:
- front-tier
- back-tier
This section of the docker-compose.yaml
file creates a service named frontend and uses the Docker image example/webapp
. This image is anticipated to include the application code and environment required to operate the frontend component of a web application. The networks section states that this service is connected to two networks: front-tier
and back-tier
. By connecting the frontend service to these networks, it can communicate with other services on the same networks, allowing for easier interaction across different components of a multi-container application.
Volumes
Volumes in Docker are used to handle persistent data that must survive the lifecycle of a container. Volumes in Docker Compose provide a common way for services to mount and use persistent storage areas. You can construct named volumes that are reusable across several services in your application by specifying them at the top level in a docker-compose.yaml
file. This configuration not only helps to guarantee data consistency and integrity, but it also simplifies storage resource management by allowing volumes to be configured and allocated centrally.
services:
backend:
image: example/database
volumes:
- db-data:/etc/data
backup:
image: backup-service
volumes:
- db-data:/var/lib/backup/data
volumes:
db-data:
In this docker-compose.yaml
configuration, the volumes
section defines a named volume called db-data
. This volume is then used by two different services: backend
and backup
.
The backend
service uses the example/database
image and mounts the db-data
volume to the /etc/data
directory inside the container. This setup allows the backend
service to store and persist data in the db-data
volume.
The backup
service, which uses the backup-service
image, also mounts the same db-data
volume but maps it to a different directory inside its container, /var/lib/backup/data
. This configuration enables the backup
service to access and potentially back up the data stored by the backend
service.
Creating a Web Server with Docker Compose
Web servers running within containers frequently require various initial operations before they can begin serving content. These tasks could include setting up configurations, downloading files, or installing dependencies. Docker Compose makes this process easier by allowing you to specify all of these operations as part of a multi-container application configuration. In this exercise, you will build a preparation container specifically for creating static files such as index.html
. Following that, a server container will be configured to serve the static files. With the proper network configuration, this server will be accessible from your host system. You'll also manage the full application lifecycle with various Docker Compose commands, which will make it easier to set up and run your containers.
Create a folder named compose-server
and navigate to it:
mkdir compose-server
cd compose-server
Create a folder named init
and navigate to it:
mkdir init
cd init
Create a bash script to generate a simple HTML page
#!/usr/bin/env sh
# Check if a directory named 'data' exists
if [ -d "data" ]; then
# If it does, delete it and all its contents
echo "Removing existing 'data' directory..."
rm -rf data
fi
# Create a new directory named 'data'
echo "Creating a new 'data' directory..."
mkdir data
# Create a new file named 'index.html' inside the 'data' directory
echo "Creating a new file 'index.html' inside the 'data' directory..."
touch data/index.html
# Append HTML content to the 'index.html' file
echo "Appending HTML content to the 'index.html' file..."
echo "<h1>Welcome to Docker Compose! </h1>" >> data/index.html
echo "<img src='https://www.docker.com/wp-content/uploads/2021/10/Moby-logo-sm.png' />" >> data/index.html
Create a Dockerfile with the name Dockerfile
and the following content:
# Use the busybox base image
FROM busybox
# Copy the prepare.sh script to /usr/bin/prepare.sh in the image
ADD prepare.sh /usr/bin/prepare.sh
# Make the prepare.sh script executable
RUN chmod +x /usr/bin/prepare.sh
# Set the entry point for the container to be the prepare.sh script
# This means that when the container starts, it will run the prepare.sh script
ENTRYPOINT ["sh", "/usr/bin/prepare.sh"]
This Dockerfile starts by using the busybox
base image, which is a minimal and lightweight Linux distribution commonly used for small utilities and embedded systems. It then adds a script named prepare.sh
from the build context to the /usr/bin/prepare.sh
path within the image. To ensure that this script can be executed, the Dockerfile uses the RUN
command to modify its permissions, making it executable with chmod +x
. Finally, the ENTRYPOINT
instruction is specified to define the default command that will be run when the container starts. In this case, it sets the entry point to execute the prepare.sh
script using the sh
shell. This setup means that every time the container is launched, it will automatically run prepare.sh
, allowing for pre-defined setup or configuration tasks to be carried out as the container starts.
Change the directory to the parent folder with the cd ..
command and create a docker-compose.yaml
file with the following content
name: "compose-server"
services:
init:
build:
context: ./init
volumes:
- static:/data
server:
image: nginx
volumes:
- static:/usr/share/nginx/html
ports:
- "8080:80"
volumes:
static:
This Docker Compose file sets up a multi-container application with two services: init
and server
.
-
Service Definitions:
-
init
: This service builds an image from a Dockerfile located in the./init
directory. The Dockerfile in this context likely contains instructions to prepare or generate static files. Thevolumes
section maps a named volume,static
, to the/data
directory inside the container. This allows theinit
service to write data to this volume, which can then be accessed by other services. -
server
: This service uses the officialnginx
image to run an NGINX web server. It also mounts the same named volume,static
, but this time to the/usr/share/nginx/html
directory inside the container. This is where NGINX expects to find static files to serve. Theports
section maps port 80 of the container to port 8080 on the host machine, making the web server accessible viahttp://localhost:8080
on the host.
-
-
Volumes:
- The
volumes
section at the bottom defines a named volume calledstatic
. Named volumes are managed by Docker and provide persistent storage that is shared among containers. In this case, thestatic
volume is used to transfer static files generated by theinit
service to theserver
service.
- The
Start the application with the following command:
docker-compose up --detach
And you should see an output similar to the following:
[+] Running 8/8
✔ server 7 layers [⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 41.0s
✔ efc2b5ad9eec Pull complete 32.9s
✔ 8fe9a55eb80f Pull complete 29.3s
✔ 045037a63be8 Pull complete 0.9s
✔ 7111b42b4bfa Pull complete 1.6s
✔ 3dfc528a4df9 Pull complete 3.0s
✔ 9e891cdb453b Pull complete 5.3s
✔ 0f11e17345c5 Pull complete 6.2s
[+] Building 5.3s (9/9) FINISHED docker:default
=> [init internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 457B 0.0s
=> [init internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [init internal] load metadata for docker.io/library/busybox:latest 4.5s
=> [init auth] library/busybox:pull token for registry-1.docker.io 0.0s
=> [init internal] load build context 0.0s
=> => transferring context: 771B 0.0s
=> CACHED [init 1/3] FROM docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85 0.0s
=> => resolve docker.io/library/busybox@sha256:9ae97d36d26566ff84e8893c64a6dc4fe8ca6d1144bf5b87b2b85a32def253c7 0.0s
=> [init 2/3] ADD prepare.sh /usr/bin/prepare.sh 0.0s
=> [init 3/3] RUN chmod +x /usr/bin/prepare.sh 0.5s
=> [init] exporting to image 0.1s
=> => exporting layers 0.0s
=> => writing image sha256:8d2660840cab8d1e01e7cb9ad823fb2a2e3141b73c6cc0cce53c91355a6d52a6 0.0s
=> => naming to docker.io/library/compose-server-init 0.0s
[+] Running 4/4
✔ Network compose-server_default Created 0.3s
✔ Volume "compose-server_static" Created 0.0s
✔ Container compose-server-init-1 Started 0.1s
✔ Container compose-server-server-1 Started 0.1s
This command creates and starts the containers in detached mode. It starts by creating the compose-server_default
network and the compose-server_static
volume. It then builds the init
container using the Dockerfile from the previous step, downloads nginx
, and starts the containers.
Check the status of the application with the docker-compose ps
command:
docker-compose ps
and you should see the following:
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
compose-server-server-1 nginx "/docker-entrypoint.sh nginx -g 'daemon off;'" server 3 minutes ago Up 3 minutes 0.0.0.0:8080->80/tcp
Open the http://localhost:8080/ on your browser. If everything went as planned you should see the following:
Stop and remove all the resources with the following command if you do not need the application up and running:
docker-compose down
And you should see the following output:
[+] Running 3/3
✔ Container compose-server-server-1 Removed 0.6s
✔ Container compose-server-init-1 Removed 0.0s
✔ Network compose-server_default Removed 0.3s
In this exercise, we successfully created and configured a multi-container application using Docker Compose. The docker-compose.yaml
file was employed to define both networking and volume options, which are crucial for inter-container communication and data persistence. We demonstrated how to use Docker Compose CLI commands to build and manage the application. These commands included starting and stopping containers, checking their status, and removing them when no longer needed.
Summary
This blog post highlighted Docker Compose's ability to manage and orchestrate multi-container applications effectively. Building on our earlier explorations of Docker containers and Dockerfiles, we looked at how Docker Compose simplifies the setup and administration of complicated applications with various components. Docker Compose, which defines your whole application stack in a single YAML file, enables you to spin up complex environments with a single command, making it ideal for development, testing, and production.
We've highlighted Docker Compose's major capabilities, such as its ability to separate container instances, handle stateful data using named volumes, and facilitate iterative development. The practical examples offered demonstrated how to effectively develop, manage, and troubleshoot applications using Docker Compose CLI commands. We also went over the key components of a docker-compose.yaml file, like service definitions, network setups, and volume management.
Looking ahead, our next post will go over the configuration choices accessible in the Docker Compose environment. We'll go over advanced settings and approaches for further optimizing and tailoring your application installations, ensuring that you can fully utilize Docker Compose's capabilities to meet your individual requirements.
See you next Monday!
Top comments (2)
This was a really informative post! The way you broke down the Docker Compose file was very helpful for understanding its components. Could you maybe provide more examples of using Docker Compose in a CI/CD pipeline in a future post? I think that would be a great follow-up!
A very good post! I would just like to add that
docker-compose
is deprecated for the newer versiondocker compose
. See the following for more info: docs.docker.com/compose/migrate/