DevOps ยท 29 min read

DevOps Bootcamp #1 - Docker

Understanding what Docker is, how it works inside, and how to use it for real projects

On this page
  1. 1. The problem Docker solves
  2. 2. What Docker actually is
  3. 3. Containers vs virtual machines
  4. 4. How Docker works under the hood
  5. Namespaces: what a process can see
  6. Control groups (cgroups): what a process can use
  7. Union filesystem: how images stay small
  8. 5. Docker architecture
  9. What happens when you run docker run -d -p 8080:80 nginx
  10. 6. Installing Docker and your first container
  11. 7. Images and layers
  12. Image names and tags
  13. Useful image commands
  14. Choosing a base image
  15. 8. Containers: lifecycle and essential commands
  16. Running containers
  17. Managing containers
  18. Looking inside
  19. Restart policies
  20. 9. Writing a Dockerfile: every instruction explained
  21. Every instruction
  22. CMD vs ENTRYPOINT
  23. Exec form vs shell form
  24. ARG vs ENV
  25. The .dockerignore file
  26. 10. Build cache, multi-stage builds and small images
  27. How the build cache works
  28. Multi-stage builds
  29. More ways to shrink images
  30. BuildKit extras
  31. 11. Storage: volumes, bind mounts and tmpfs
  32. Volumes (use these by default)
  33. Bind mounts (great for development)
  34. tmpfs mounts
  35. The --mount syntax
  36. Backing up a volume
  37. 12. Networking
  38. Network drivers
  39. Publishing ports
  40. Custom networks and service discovery
  41. Handy network commands
  42. A note on localhost
  43. 13. Docker Compose for multi-container apps
  44. Key Compose commands
  45. Things worth knowing
  46. 14. Registries: storing and sharing images
  47. Push an image
  48. Tagging strategy
  49. 15. Security best practices
  50. 16. Debugging and troubleshooting
  51. The debugging checklist
  52. Common problems and fixes
  53. Debugging minimal images
  54. Cleaning up
  55. 17. Docker in production and what comes next
  56. Production habits
  57. Docker in a CI/CD pipeline
  58. From one host to many: orchestration
  59. 18. Cheat sheet and wrap-up
  60. The whole story in seven lines

Docker lets you pack an app and everything it needs into one box that runs the same way on any machine. This is Part 1 of my DevOps series, and by the end of it you will understand what Docker is, how it works inside, and how to use it for real projects.

Here is what we will cover:

  • Why Docker exists and what problem it solves
  • Containers, images and how they differ from virtual machines
  • What happens under the hood (namespaces, cgroups, layers)
  • Writing Dockerfiles, building small images and using the build cache
  • Storage, networking and Docker Compose
  • Registries, security, debugging and running Docker in production

No prior Docker knowledge is needed. If you can open a terminal and run a command, you are ready.

1. The problem Docker solves

Docker exists to kill the sentence "but it works on my machine." Every developer has heard it, and most have said it.

Picture this. You build a Python app on your laptop. It uses Python 3.12, a certain version of a database driver and a few system libraries. You hand it to a teammate. Their laptop has Python 3.9 and a different driver. The app crashes. Then it goes to the test server, which runs a different Linux version, and it breaks again in a new way.

The code was never the problem. The environment was. Each machine had slightly different software around the code.

Before Docker, teams fought this with:

  • Long setup guides ("install these 14 things in this exact order")
  • Configuration tools that tried to make every server look the same
  • Virtual machines, which worked but were heavy and slow to start

This mess has a name: dependency hell. App A needs library version 1, app B needs version 2, and both live on the same server. Something has to give.

Docker fixes this by shipping the environment together with the code. Instead of saying "here is my code, good luck," you say "here is my code, plus the exact operating system files, runtime, libraries and settings it needs." Whoever runs it gets the same result, every time.

2. What Docker actually is

Docker is a tool for building, shipping and running apps inside containers. Three words do most of the work, so let's nail them down.

Image. A read-only package that holds everything your app needs: a slim set of operating system files, the language runtime, libraries, your code and the command to start it. Think of it as a recipe, or a class in programming.

Container. A running copy of an image. It is an isolated process on your machine that thinks it has its own computer. Think of it as the dish cooked from the recipe, or an object created from a class. One image can start many containers.

Dockerfile. A plain text file with step-by-step instructions to build an image. You write it once, and anyone can build the same image from it.

The flow always looks like this:

  1. You write a Dockerfile
  2. docker build turns it into an image
  3. docker push uploads the image to a registry (like Docker Hub)
  4. Anyone runs docker pull and docker run to start a container from it

The shipping container analogy is where the name comes from. Before standard steel containers, loading a ship meant handling barrels, sacks and crates of every shape. Once everything went into the same box, any crane, ship or truck could move it. Docker does the same for software: any machine with Docker can run any container.

3. Containers vs virtual machines

A virtual machine fakes a whole computer, while a container only fakes a separate space on the same computer. That one difference explains why containers are so much lighter.

A virtual machine (VM) runs on a hypervisor. Each VM has its own full operating system with its own kernel (the core of the OS that talks to hardware). Ten VMs means ten kernels, ten copies of the OS and gigabytes of disk and memory.

A container shares the kernel of the host machine. It only carries the files and libraries the app needs. Ten containers means one kernel, with ten isolated processes on top.

Virtual machineContainer
What it virtualizesHardwareThe operating system
KernelIts own, one per VMShared with the host
Typical sizeGigabytesMegabytes
Start timeTens of seconds to minutesUsually under a second
IsolationVery strongStrong, but weaker than a VM
Can run a different OS kernelYes (Windows VM on Linux)No, Linux containers need a Linux kernel
Best forFull OS isolation, mixed OS typesPackaging and running apps

A common question: "How does Docker run Linux containers on my Mac or Windows laptop?" Docker Desktop quietly starts a small Linux VM, and your containers run inside it. On a Linux machine, no VM is needed.

Containers and VMs are not enemies. In the cloud, your containers usually run on VMs. You get the strong isolation of VMs between customers and the speed of containers for each app.

4. How Docker works under the hood

A container is not magic. It is a normal Linux process with three kernel features wrapped around it: namespaces, cgroups and a layered filesystem. Docker just makes these easy to use.

Namespaces: what a process can see

Namespaces give a process its own private view of the system. Inside a container, the process thinks it is alone. The main ones:

  • PID: the container has its own process list. Your app is process 1 inside, even though the host sees it as process 48213.
  • NET: its own network interfaces, IP address and ports.
  • MNT: its own filesystem tree, so it cannot see the host's files.
  • UTS: its own hostname.
  • IPC: its own shared memory and message queues.
  • USER: its own user IDs, so root inside can map to a normal user outside.

Control groups (cgroups): what a process can use

Namespaces hide things. Cgroups limit things. They cap how much CPU, memory, disk I/O and number of processes a container can use. This stops one greedy container from starving everything else on the machine.

bash
# Allow at most half a CPU core and 256 MB of memory
docker run --cpus=0.5 --memory=256m nginx

If a container goes over its memory limit, the kernel kills it. You will see this as an "OOMKilled" status (Out Of Memory).

Union filesystem: how images stay small

Docker images are built from stacked, read-only layers. A union filesystem (Docker uses one called OverlayFS) merges these layers so they look like one normal folder.

When a container starts, Docker adds one thin writable layer on top. Any file the container changes is copied up into that layer first. This trick is called copy-on-write. The image layers below never change, so 50 containers can share the same image on disk and each only stores its own changes.

Put together: namespaces give isolation, cgroups give limits, and layers give speed and small size. That is a container.

5. Docker architecture

Docker is a client-server system. The docker command you type is only a messenger; the real work happens in a background service and two lower-level runtimes.

Each request flows top to bottom: the client calls the daemon, the daemon asks containerd, and containerd uses runc to start the actual process.

The CLI only sends requests; the daemon and runtimes do the work. The Docker client (docker build, docker run, docker pull) talks to dockerd on the Docker host, which builds images, manages networks and volumes, and pulls and pushes images from a registry such as Docker Hub, GHCR or ECR. dockerd calls containerd over gRPC; containerd pulls images and runs and watches containers, with one shim per container; runc sets up namespaces and cgroups, then exits, leaving containers such as nginx, api and redis running.

  • Docker client (CLI). The docker command. It turns what you type into API calls and sends them to the daemon, either on your machine or on a remote one.
  • Docker daemon (dockerd). The background service that listens for API calls. It builds images, and manages networks, volumes and the overall state.
  • containerd. A container runtime that handles the full life of a container: pulling images, storing them, starting, stopping and watching containers. It is also used directly by Kubernetes.
  • runc. The low-level runtime. It talks to the Linux kernel to create the namespaces and cgroups, starts your process, then steps aside. A small shim process stays behind for each container, so containers keep running even if the daemon restarts.
  • Registry. A server that stores images. Docker Hub is the default public one.

What happens when you run docker run -d -p 8080:80 nginx

  1. The CLI sends a "create and start container" request to dockerd
  2. dockerd checks for the nginx image locally; if missing, it pulls it from the registry layer by layer
  3. dockerd sets up networking: connects the container to the bridge network and maps port 8080 to 80
  4. dockerd hands the job to containerd, which prepares the filesystem (image layers plus a writable layer)
  5. containerd calls runc, which creates the namespaces and cgroups and starts Nginx as process 1
  6. runc exits, the shim keeps watching the container, and the CLI prints the container ID

All of this follows open standards from the Open Container Initiative (OCI). The image format and the runtime are standardized, which is why an image built with Docker also runs on Podman, containerd or Kubernetes.

6. Installing Docker and your first container

Install Docker, run one command, and you have a working container in under a minute.

On Mac or Windows: install Docker Desktop from the official Docker website. It bundles the engine, the CLI, Compose and the small Linux VM.

On Linux (Ubuntu example): the quickest way is Docker's convenience script.

bash
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Run docker without sudo (log out and back in after this)
sudo usermod -aG docker $USER

Note: being in the docker group is the same as having root on that machine. Only add users you trust.

Check that it works:

bash
docker version
docker run hello-world

When you run hello-world, this happens:

  1. The CLI sends your command to the Docker daemon
  2. The daemon looks for the hello-world image on your machine and does not find it
  3. It pulls the image from Docker Hub
  4. It creates a container from the image and starts it
  5. The container prints a message and exits

Now try something more useful, a real web server:

bash
docker run -d -p 8080:80 --name web nginx

Open http://localhost:8080 and you will see the Nginx welcome page. You just ran a web server without installing Nginx on your machine.

7. Images and layers

An image is a stack of read-only layers plus some metadata, and understanding layers is the key to fast builds and small images.

Every instruction in a Dockerfile that changes files (FROM, RUN, COPY, ADD) creates a new layer. Each layer only stores the difference from the layer below it.

text
Layer 4:  COPY . /app            (your code, changes often)
Layer 3:  RUN pip install ...    (dependencies)
Layer 2:  RUN apt-get install... (system packages)
Layer 1:  FROM python:3.12-slim  (base OS + Python)

Why this matters:

  • Sharing. If ten images use python:3.12-slim as a base, that base is stored on disk only once.
  • Faster pulls. When you pull a new version of an image, Docker only downloads the layers that changed.
  • Caching. When you rebuild, Docker reuses layers that have not changed (more on this in section 10).

Image names and tags

An image name has up to four parts:

text
registry/namespace/repository:tag
docker.io/library/nginx:1.27-alpine
  • Registry: where it lives. Docker Hub (docker.io) is the default.
  • Namespace: the user or organization. Official images use library.
  • Repository: the image name.
  • Tag: the version. If you leave it out, Docker uses latest.

A warning about latest: it does not mean "newest." It is just a default tag name, and it can point to anything. In production, always use a specific tag like nginx:1.27-alpine, or even the image digest (nginx@sha256:...), which never changes.

Useful image commands

bash
docker images                   # list local images
docker pull redis:7             # download an image
docker history redis:7          # see the layers and their sizes
docker inspect redis:7          # full metadata in JSON
docker rmi redis:7              # delete an image
docker image prune              # remove unused (dangling) images

Choosing a base image

Base imageRough sizeWhen to use it
ubuntu, debian75 to 125 MBYou need familiar tools and wide package support
*-slim (e.g. python:3.12-slim)Much smaller than the full imageA good default for most apps
alpineAbout 5 to 8 MBTiny images; uses musl instead of glibc, which can break some libraries
distrolessVery small, no shellProduction images with the smallest attack surface
scratch0 MB (empty)Static binaries, like Go apps

8. Containers: lifecycle and essential commands

A container moves through a small set of states, and a handful of commands move it between them.

The states are: created (exists but never started), running, paused (frozen in memory), exited (stopped) and removed (gone for good). An exited container still keeps its writable layer, so you can start it again with its changes intact. Only docker rm throws that away.

Running containers

bash
docker run nginx                     # run in the foreground
docker run -d nginx                  # run in the background (detached)
docker run -d --name web nginx       # give it a name
docker run -d -p 8080:80 nginx       # map host port 8080 to container port 80
docker run -it ubuntu bash           # interactive shell (-i keeps input open, -t gives a terminal)
docker run --rm alpine echo hi       # delete the container when it exits
docker run -e APP_ENV=prod myapp     # set an environment variable
docker run --env-file .env myapp     # load variables from a file
docker run --restart unless-stopped nginx   # restart if it crashes or the machine reboots

Managing containers

bash
docker ps                  # running containers
docker ps -a               # all containers, including stopped
docker stop web            # graceful stop (SIGTERM, then SIGKILL after 10s)
docker kill web            # stop immediately (SIGKILL)
docker start web           # start a stopped container
docker restart web
docker pause web           # freeze all its processes
docker unpause web
docker rm web              # delete a stopped container
docker rm -f web           # stop and delete in one go
docker container prune     # delete all stopped containers

Looking inside

bash
docker logs web            # see output
docker logs -f --tail 100 web   # follow the last 100 lines live
docker exec -it web sh     # open a shell inside a running container
docker inspect web         # full details: IP, mounts, env, state
docker stats               # live CPU and memory use
docker top web             # processes inside the container
docker cp web:/etc/nginx/nginx.conf .   # copy a file out

Restart policies

PolicyWhat it does
noNever restart (the default)
on-failure[:N]Restart only if it exits with an error, optionally at most N times
alwaysAlways restart when it stops. Even if you stopped it by hand, it comes back when the Docker daemon restarts
unless-stoppedLike always, but stays stopped if you stopped it yourself

One important idea: containers are meant to be disposable. You should be able to delete one and start a fresh copy at any time without losing anything important. Anything that must survive goes in a volume (section 11).

9. Writing a Dockerfile: every instruction explained

A Dockerfile is a recipe read from top to bottom, and each line is one instruction. Here is a real one for a small Node.js API, followed by what every instruction means.

dockerfile
# syntax=docker/dockerfile:1
FROM node:22-alpine

WORKDIR /app

ENV NODE_ENV=production

COPY package.json package-lock.json ./
RUN npm ci --omit=dev

COPY . .

USER node
EXPOSE 3000

HEALTHCHECK --interval=30s --timeout=3s \
  CMD wget -qO- http://localhost:3000/health || exit 1

CMD ["node", "server.js"]

Build and run it:

bash
docker build -t my-api:1.0 .
docker run -d -p 3000:3000 my-api:1.0

The . at the end of docker build is the build context: the folder whose files Docker can use during the build.

Every instruction

InstructionWhat it doesExample
FROMSets the base image. Every Dockerfile starts hereFROM python:3.12-slim
WORKDIRSets the working folder for the next steps (creates it if missing)WORKDIR /app
COPYCopies files from the build context into the imageCOPY src/ ./src/
ADDLike COPY, but can also unpack local tar files and download URLs. Prefer COPY unless you need theseADD app.tar.gz /opt/
RUNRuns a command at build time and saves the result as a layerRUN apt-get update && apt-get install -y curl
ENVSets an environment variable for build and runtimeENV PORT=8080
ARGA variable available only at build timeARG VERSION=1.0
EXPOSEDocuments which port the app listens on. It does not publish the portEXPOSE 8080
USERSwitches to a user for the next steps and at runtimeUSER appuser
VOLUMEMarks a folder as a place for persistent dataVOLUME /data
LABELAdds metadata like author or versionLABEL org.opencontainers.image.version="1.0"
HEALTHCHECKA command Docker runs to check the app is healthyHEALTHCHECK CMD curl -f http://localhost/ || exit 1
ENTRYPOINTThe main executable of the containerENTRYPOINT ["python", "app.py"]
CMDThe default command, or default arguments to ENTRYPOINTCMD ["--port", "8080"]
SHELLChanges the default shell used by RUNSHELL ["/bin/bash", "-c"]
STOPSIGNALThe signal sent to stop the containerSTOPSIGNAL SIGINT
ONBUILDA trigger that runs when another image uses this one as a baseONBUILD COPY . /app

CMD vs ENTRYPOINT

This confuses almost everyone at first, so here is the simple version.

  • CMD is a default you can easily replace. docker run myimage bash replaces the CMD with bash.
  • ENTRYPOINT is the fixed program. Anything you pass to docker run is added as arguments to it.
  • Used together, ENTRYPOINT is the program and CMD is its default arguments.
dockerfile
ENTRYPOINT ["python", "cli.py"]
CMD ["--help"]
# docker run tool          -> python cli.py --help
# docker run tool process  -> python cli.py process

Exec form vs shell form

  • Exec form: CMD ["node", "server.js"]. Your app runs directly as process 1 and receives stop signals properly. Use this.
  • Shell form: CMD node server.js. Docker wraps it in /bin/sh -c, so the shell is process 1 and your app may not get the stop signal. The result: slow, ungraceful shutdowns.

ARG vs ENV

  • ARG exists only while building. Pass it with docker build --build-arg VERSION=2.0 .
  • ENV is baked into the image and visible to the running container.
  • Never put secrets in either one. Both can be read from the image history.

The .dockerignore file

Like .gitignore, a .dockerignore file tells Docker which files to leave out of the build context. This makes builds faster and keeps secrets out of your image.

text
node_modules
.git
.env
*.log
Dockerfile
.dockerignore

10. Build cache, multi-stage builds and small images

Small images build faster, pull faster, cost less to store and give attackers less to work with. Three techniques get you there: ordering for the cache, multi-stage builds and cleaning up inside each layer.

How the build cache works

Docker checks each instruction in order. If the instruction and its inputs are the same as last time, it reuses the saved layer. The moment one layer changes, every layer after it is rebuilt.

So put things that rarely change at the top and things that change often at the bottom.

Bad order (every code change reinstalls all dependencies):

dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

Good order (dependencies are cached until requirements.txt changes):

dockerfile
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "app.py"]

Multi-stage builds

To build an app you often need compilers, build tools and dev dependencies. To run it, you need none of those. Multi-stage builds let you use a big image to build and a tiny image to run, and only copy the final result across.

dockerfile
# Stage 1: build
FROM golang:1.23 AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /bin/server ./cmd/server

# Stage 2: run
FROM gcr.io/distroless/static-debian12
COPY --from=builder /bin/server /server
USER nonroot
ENTRYPOINT ["/server"]

The builder image is close to 1 GB. The final image holds just your binary and a few files, often around 10 to 20 MB. The Go compiler never ships to production.

The same idea works for Node.js (build the frontend, then serve the static files with Nginx), Java (build with Maven, run on a JRE image) and others.

More ways to shrink images

  • Start from a small base: -slim, alpine or distroless (see section 7).

  • Clean up in the same RUN line. Files deleted in a later layer still take space in the earlier one.

    dockerfile
    RUN apt-get update \
     && apt-get install -y --no-install-recommends curl \
     && rm -rf /var/lib/apt/lists/*
    
  • Skip package caches: pip install --no-cache-dir, npm ci --omit=dev.

  • Use .dockerignore so node_modules, .git and test data never enter the context.

  • Combine related commands into one RUN to cut layer count.

BuildKit extras

Modern Docker uses BuildKit as its builder by default. It runs independent stages in parallel and adds a few handy features:

dockerfile
# Keep a package cache between builds without putting it in the image
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt

# Use a secret during the build without saving it in any layer
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci
bash
docker build --secret id=npmrc,src=$HOME/.npmrc -t app .

To build for several CPU types at once (for example Intel servers and ARM laptops), use docker buildx:

bash
docker buildx build --platform linux/amd64,linux/arm64 -t me/app:1.0 --push .

11. Storage: volumes, bind mounts and tmpfs

Anything a container writes to its own writable layer disappears when the container is removed. To keep data, you mount storage from outside the container. There are three types.

TypeWhere the data livesManaged byBest for
VolumeA folder Docker manages (on Linux, under /var/lib/docker/volumes)DockerDatabases and any data that must survive
Bind mountAny folder or file you choose on the hostYouLocal development, sharing config files
tmpfsThe host's memory only, never on diskThe kernelTemporary or sensitive files that should never hit disk

Volumes (use these by default)

bash
docker volume create pgdata
docker run -d --name db \
  -e POSTGRES_PASSWORD=secret \
  -v pgdata:/var/lib/postgresql/data \
  postgres:16

docker volume ls
docker volume inspect pgdata
docker volume rm pgdata
docker volume prune        # delete unused anonymous volumes (add -a for named ones too)

Now you can delete the db container, start a new one with the same volume, and all your data is still there.

Bind mounts (great for development)

A bind mount links a folder on your machine into the container. Edit code on your laptop and the container sees the change instantly, which is perfect for hot reload.

bash
docker run -d -p 3000:3000 -v "$(pwd)":/app -w /app node:22 npm run dev

# Read-only mount for config files
docker run -v "$(pwd)/nginx.conf":/etc/nginx/nginx.conf:ro nginx

tmpfs mounts

bash
docker run --tmpfs /tmp:size=64m myapp

The --mount syntax

-v is short and handy. --mount is longer but clearer, and it is what Docker recommends for scripts.

bash
docker run --mount type=volume,source=pgdata,target=/var/lib/postgresql/data postgres:16
docker run --mount type=bind,source="$(pwd)",target=/app,readonly node:22

Backing up a volume

A simple trick: start a throwaway container that mounts the volume and a host folder, then tar the data.

bash
docker run --rm -v pgdata:/data -v "$(pwd)":/backup alpine \
  tar czf /backup/pgdata.tar.gz -C /data .

For databases, a proper dump (like pg_dump) is safer than copying files from a running database.

12. Networking

Every container gets its own network stack, and Docker networks decide who can talk to whom. The most useful rule to remember: containers on the same user-defined network can reach each other by name.

Network drivers

DriverWhat it doesWhen to use it
bridgeA private virtual network on one host. The defaultMost single-host setups
hostThe container uses the host's network directly, no isolationMax network performance, Linux only
noneNo network at allLocked-down batch jobs
overlayA network that spans many Docker hostsDocker Swarm clusters
macvlanGives the container its own MAC address on your physical networkLegacy apps that must look like a real device

Publishing ports

Containers are private by default. To reach one from outside, publish a port with -p HOST:CONTAINER.

bash
docker run -d -p 8080:80 nginx            # host 8080 -> container 80
docker run -d -p 127.0.0.1:8080:80 nginx  # only reachable from this machine
docker run -d -P nginx                    # map all EXPOSEd ports to random host ports
docker port <container>                   # see the mappings

Remember: EXPOSE in a Dockerfile is only documentation. -p is what actually opens the port.

Custom networks and service discovery

The default bridge network does not give you name-based discovery. Create your own network and Docker runs a built-in DNS server for it.

bash
docker network create appnet

docker run -d --name db --network appnet \
  -e POSTGRES_PASSWORD=secret postgres:16

docker run -d --name api --network appnet \
  -e DATABASE_URL=postgres://postgres:secret@db:5432/postgres \
  -p 3000:3000 my-api:1.0

The API reaches the database at the hostname db. No IP addresses needed, and the database port is never exposed to the outside world.

Handy network commands

bash
docker network ls
docker network inspect appnet
docker network connect appnet web      # attach a running container
docker network disconnect appnet web
docker network rm appnet

A note on localhost

Inside a container, localhost means the container itself, not your machine. If your app in a container needs a service running on your laptop, use host.docker.internal (works out of the box on Docker Desktop; on Linux add --add-host=host.docker.internal:host-gateway).

13. Docker Compose for multi-container apps

Real apps are rarely one container. Docker Compose lets you describe the whole stack (API, database, cache, worker) in one YAML file and start it with one command.

Here is a full example: a web API with Postgres and Redis.

yaml
# compose.yaml
services:
  api:
    build: .
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgres://app:secret@db:5432/app
      REDIS_URL: redis://cache:6379
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started
    restart: unless-stopped

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: secret
      POSTGRES_DB: app
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app"]
      interval: 5s
      retries: 5

  cache:
    image: redis:7-alpine

volumes:
  pgdata:

Compose creates a private network for the project automatically, so api reaches the others as db and cache.

Key Compose commands

bash
docker compose up -d          # build if needed and start everything in the background
docker compose up -d --build  # force a rebuild
docker compose ps             # status of each service
docker compose logs -f api    # follow logs of one service
docker compose exec db psql -U app   # run a command in a service
docker compose stop           # stop, keep containers
docker compose down           # stop and remove containers and network
docker compose down -v        # also remove volumes (deletes your data)
docker compose up -d --scale worker=3   # run 3 copies of a service

Things worth knowing

  • depends_on only controls start order. By itself it does not wait for the database to be ready. Add a healthcheck and condition: service_healthy, as above.
  • Use a .env file. Compose reads a .env file next to compose.yaml, so you can write ${POSTGRES_PASSWORD} instead of hard-coding secrets.
  • Override files. Keep a base compose.yaml and add compose.override.yaml for dev-only settings like bind mounts. Compose merges them automatically.
  • Profiles let you mark optional services (like an admin UI) that only start with docker compose --profile tools up.
  • Watch mode. docker compose watch can sync code changes into running containers or rebuild them as you edit.
  • The old docker-compose (with a dash) command is retired. Use docker compose (with a space), and you can drop the version: line from your file.

Compose is perfect for local development, testing and small single-server deployments. For many servers, you move to an orchestrator like Kubernetes.

14. Registries: storing and sharing images

A registry is like GitHub for images: you push images to it, and any machine can pull them. Docker Hub is the default, but every major cloud has its own.

RegistryRun by
Docker Hub (docker.io)Docker
GitHub Container Registry (ghcr.io)GitHub
Amazon ECRAWS
Google Artifact RegistryGoogle Cloud
Azure Container RegistryMicrosoft
HarborOpen source, self-hosted

Push an image

bash
docker login                                  # or: docker login ghcr.io
docker tag my-api:1.0 harry/my-api:1.0        # name it for your account
docker push harry/my-api:1.0

# Anywhere else:
docker pull harry/my-api:1.0

Tagging strategy

A good tagging habit saves you from "which version is running in production?" panics.

  • Semantic versions: 1.4.2, plus moving tags like 1.4 and 1
  • Git commit SHA: my-api:3f9c2a1, so every image traces back to exact code
  • Never deploy latest to production. You lose track of what is running and rollbacks get messy.

For quick local experiments you can even run your own registry:

bash
docker run -d -p 5000:5000 --name registry registry:2
docker tag my-api:1.0 localhost:5000/my-api:1.0
docker push localhost:5000/my-api:1.0

15. Security best practices

Containers share the host kernel, so a weak container can become a doorway to the whole machine. These habits close most of those doors.

  1. Do not run as root. By default, processes in a container run as root. Create a normal user and switch to it.

    dockerfile
    RUN addgroup --system app && adduser --system --ingroup app app
    USER app
    
  2. Use small, trusted base images. Official or verified images, in -slim, alpine or distroless form. Fewer packages means fewer vulnerabilities.

  3. Pin versions. Use exact tags or digests so a surprise update never sneaks in.

  4. Scan your images. Tools like Docker Scout (docker scout cves my-api:1.0) and Trivy (trivy image my-api:1.0) list known vulnerabilities in your packages. Run them in CI.

  5. Keep secrets out of images. Never COPY .env or put passwords in ENV or ARG. Pass secrets at runtime (environment variables from a secret manager, Docker secrets, or mounted files) and use BuildKit --mount=type=secret for build-time secrets.

  6. Make the filesystem read-only where you can, with a small tmpfs for scratch space.

    bash
    docker run --read-only --tmpfs /tmp myapp
    
  7. Drop Linux capabilities. Capabilities are pieces of root power. Remove all of them and add back only what you need.

    bash
    docker run --cap-drop ALL --cap-add NET_BIND_SERVICE myapp
    
  8. Block privilege escalation: --security-opt no-new-privileges.

  9. Never use --privileged unless you truly know why. It gives the container almost full access to the host.

  10. Never mount the Docker socket (/var/run/docker.sock) into a container you do not fully trust. Access to that socket is the same as root on the host.

  11. Set resource limits (--memory, --cpus, --pids-limit) so one container cannot take down the machine.

  12. Consider rootless mode. Docker can run the whole daemon as a normal user, which limits the damage if something breaks out.

  13. Rebuild regularly. Base images get security patches. An image built a year ago carries a year of known bugs.

16. Debugging and troubleshooting

Most Docker problems fall into a few common patterns, and a short checklist solves almost all of them.

The debugging checklist

  1. Is it running? docker ps -a shows the status and exit code.
  2. What did it say? docker logs <container> shows the error your app printed before dying.
  3. What does Docker know? docker inspect <container> shows state, exit code, OOMKilled, mounts, env and network.
  4. Get inside. docker exec -it <container> sh lets you look around, check files and test connections.
  5. Is it starving? docker stats shows CPU and memory use live.
  6. What is Docker doing? docker events streams everything the daemon does.

Common problems and fixes

SymptomLikely causeFix
Container exits right after startingThe main process finished or crashed. Containers live only as long as process 1Check docker logs; make sure CMD runs a long-lived process in the foreground
Exit code 137Killed with SIGKILL, often out of memoryCheck OOMKilled in docker inspect; raise --memory or fix the leak
Exit code 143Stopped with SIGTERM (a normal stop)Usually fine; make sure your app handles SIGTERM gracefully
Exit code 126 or 127Command not executable, or not foundCheck the path in CMD/ENTRYPOINT and file permissions (chmod +x)
"port is already allocated"Another process or container uses that host portPick another host port or stop the other one
App works in container but not from browserApp listens on 127.0.0.1 inside the containerMake the app listen on 0.0.0.0
Containers cannot reach each other by nameThey are on the default bridge networkPut them on a custom network or use Compose
"no space left on device"Old images, containers, volumes and build cachedocker system df to see usage, docker system prune to clean
"permission denied" on mounted filesUser IDs inside and outside do not matchMatch the UID with --user or fix ownership on the host
Build is slow every timeCache busted early (e.g. COPY . . before installing deps)Reorder the Dockerfile and add .dockerignore
"exec format error"Image built for a different CPU (ARM vs x86)Build with --platform or use buildx for multi-platform images

Debugging minimal images

Distroless and scratch images have no shell, so docker exec sh fails. Options:

  • Temporarily build a -debug variant (distroless offers these) with a shell
  • Attach a debug container that shares the target's namespaces: docker debug <container> (Docker Desktop) or docker run -it --pid=container:<name> --network=container:<name> busybox

Cleaning up

bash
docker system df                 # what is using space
docker system prune              # remove stopped containers, unused networks, dangling images, build cache
docker system prune -a --volumes # remove everything unused, including anonymous volumes (careful)
docker builder prune             # clear just the build cache

17. Docker in production and what comes next

Running containers on your laptop and running them for real users are different jobs. In production, you care about images that are repeatable, apps that fail gracefully and systems that heal themselves.

Production habits

  • One process per container. Run the API in one container and the worker in another. They scale, restart and log independently.
  • Stateless containers. Keep state in databases, object storage or volumes, never in the container's own layer.
  • Config through environment. The same image should run in dev, staging and prod. Only the environment variables change (the twelve-factor app idea).
  • Log to stdout and stderr. Do not write log files inside the container. Docker collects stdout, and a log driver ships it to your logging system.
  • Health checks. Add a HEALTHCHECK or let your orchestrator probe a /health endpoint so broken containers get replaced.
  • Graceful shutdown. Handle SIGTERM: stop taking new requests, finish current ones, close connections, then exit.
  • Use an init process if your app spawns child processes. docker run --init adds a tiny init that reaps zombie processes and forwards signals.
  • Limit resources for every container so the scheduler can place them well and nothing runs wild.

Docker in a CI/CD pipeline

A typical pipeline looks like this:

  1. Developer pushes code to Git
  2. CI builds the image with docker build, tagged with the commit SHA
  3. CI runs tests inside the container
  4. CI scans the image for vulnerabilities
  5. CI pushes the image to a registry
  6. The deploy step tells the servers or the orchestrator to run the new tag

The image that passed the tests is the exact image that runs in production. That is the big win.

From one host to many: orchestration

Docker alone runs containers on one machine. When you have many machines, you need something to decide where each container runs, restart failed ones, roll out updates and balance traffic. That is an orchestrator.

  • Docker Swarm: built into Docker, simple to learn, good for smaller setups.
  • Kubernetes: the industry standard. More complex, but it handles scaling, self-healing, rolling updates and service discovery across huge clusters.
  • Managed services: AWS ECS and Fargate, Google Cloud Run, Azure Container Apps. You give them an image and they run it.

One helpful fact: Kubernetes does not need Docker itself to run containers. It talks to runtimes like containerd directly. But the images you build with Docker follow the OCI standard, so they run on Kubernetes unchanged. Your Docker skills carry over completely.

That is exactly where this series goes next.

18. Cheat sheet and wrap-up

Keep this table handy. It covers the commands you will use 90% of the time.

TaskCommand
Build an imagedocker build -t name:tag .
List imagesdocker images
Run a container in the backgrounddocker run -d --name app -p 8080:80 image
List running / all containersdocker ps / docker ps -a
See logsdocker logs -f app
Shell into a containerdocker exec -it app sh
Stop / start / removedocker stop app / docker start app / docker rm app
Inspect detailsdocker inspect app
Live resource usedocker stats
Create a volumedocker volume create data
Create a networkdocker network create appnet
Tag and pushdocker tag app user/app:1.0 then docker push user/app:1.0
Start a Compose stackdocker compose up -d
Stop a Compose stackdocker compose down
Clean updocker system prune

The whole story in seven lines

  1. Docker packages an app with its whole environment, so it runs the same everywhere.
  2. A Dockerfile builds an image; an image runs as one or more containers.
  3. Containers are Linux processes isolated by namespaces, limited by cgroups and built on layers.
  4. Order your Dockerfile for the cache and use multi-stage builds for small images.
  5. Use volumes for data that must survive and custom networks so containers find each other by name.
  6. Use Compose for multi-container apps and a registry to share images.
  7. Run as non-root, pin versions, scan images and keep secrets out.

Next in the DevOps series: Kubernetes, where we take these containers from one machine to a whole cluster.