Docker Storage and Volumes

This article assumes you have a basic understanding of Docker and have Docker installed on a Linux system.

Introduction

Docker containers are by design ephemeral. When a container stops or is removed, any data written within its filesystem is lost. For applications that require persistent data, such as databases, web servers storing user uploads, or configuration files, this ephemeral nature is a significant limitation. Docker's storage mechanisms, specifically Docker Volumes and Bind Mounts, are designed to overcome this by allowing you to persist data outside of the container's lifecycle. This guide will walk you through understanding and utilizing these essential Docker features.

Prerequisites

A Linux server with Docker installed.
Basic familiarity with the Linux command line.
Root or sudo privileges to execute Docker commands.

Understanding Docker Storage

Docker uses a layered filesystem for its images. When a container is created from an image, a new writable layer is added on top of the image layers. Any changes made within the running container (like writing log files or creating new data) are written to this top layer. When the container is removed, this writable layer is discarded, and with it, all the data.

To ensure data persistence, Docker provides two primary mechanisms:

**Volumes:** Managed by Docker, stored in a dedicated area on the host filesystem.
**Bind Mounts:** Mount a file or directory from the host machine directly into a container.

Docker Volumes

Docker volumes are the preferred mechanism for persisting data generated by and used by Docker containers. They are created, managed, and owned by Docker.

Creating and Managing Volumes

Volumes can be created explicitly or implicitly when you first use them with a container.

Explicit Volume Creation

To create a volume, use the `docker volume create` command:

sudo docker volume create my_app_data

This command creates a new volume named `my_app_data`. Docker will store this volume in a specific directory on your host machine, typically under `/var/lib/docker/volumes/`.

Verifying Volume Creation

You can list all Docker volumes with:

sudo docker volume ls

Expected output:

DRIVER    VOLUME NAME
local     my_app_data

You can inspect a specific volume to see its mountpoint on the host:

sudo docker volume inspect my_app_data

Expected output (mountpoint will vary):

[
    {
        "CreatedAt": "2023-10-27T10:00:00Z",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/my_app_data/_data",
        "Name": "my_app_data",
        "Options": {}
    }
]

Implicit Volume Creation

If you try to mount a volume that doesn't exist when running a container, Docker will create it for you automatically.

Removing Volumes

To remove a volume (only if it's not currently used by any container), use `docker volume rm`:

sudo docker volume rm my_app_data

You can remove all unused volumes with:

sudo docker volume prune

Be cautious with `prune` as it will remove all volumes not currently attached to a container.

Using Volumes with Containers

You attach a volume to a container using the `-v` or `--mount` flag with the `docker run` command.

Using the `-v` flag

The `-v` flag is a shorthand for mounting volumes. The syntax is `volume-name:container-path`.

Let's create a simple container that writes a file to a volume:

1. Create a volume:

    sudo docker volume create web_content

2. Run an `nginx` container, mounting the `web_content` volume to `/usr/share/nginx/html` (nginx's default web root):

    sudo docker run -d --name webserver -p 8080:80 -v web_content:/usr/share/nginx/html nginx

   *   `-d`: Run container in detached mode.
   *   `--name webserver`: Name the container `webserver`.
   *   `-p 8080:80`: Map host port 8080 to container port 80.
   *   `-v web_content:/usr/share/nginx/html`: Mount the `web_content` volume to the container's web root.
   *   `nginx`: The image to use.

3. Now, let's create a simple `index.html` file on the host within the volume's mountpoint. First, find the mountpoint:

    sudo docker volume inspect web_content | grep Mountpoint

   This will output something like:

        "Mountpoint": "/var/lib/docker/volumes/web_content/_data",

   Create an `index.html` file inside this directory:

    echo "<h1>Hello from Persistent Volume!</h1>" | sudo tee /var/lib/docker/volumes/web_content/_data/index.html

4. Access the webpage from your browser by navigating to `http://your_server_ip:8080`. You should see "Hello from Persistent Volume!".

5. Stop and remove the container:

    sudo docker stop webserver
    sudo docker rm webserver

   The `web_content` volume, and the `index.html` file within it, still exist.

6. Run a new `nginx` container using the same volume:

    sudo docker run -d --name webserver2 -p 8081:80 -v web_content:/usr/share/nginx/html nginx

7. Access `http://your_server_ip:8081`. You will still see "Hello from Persistent Volume!", demonstrating that the data persisted.

Using the `--mount` flag

The `--mount` flag is more verbose but offers more explicit control and support for different storage drivers. The syntax is `type=volume,source=volume-name,target=container-path`.

Let's recreate the scenario using `--mount`:

1. Remove the previous container and volume:

    sudo docker stop webserver2
    sudo docker rm webserver2
    sudo docker volume rm web_content

2. Run the container using `--mount`:

    sudo docker run -d --name webserver_mount -p 8080:80 --mount type=volume,source=web_content,target=/usr/share/nginx/html nginx

   *   `type=volume`: Specifies that we are using a Docker volume.
   *   `source=web_content`: The name of the volume to use (Docker will create it if it doesn't exist).
   *   `target=/usr/share/nginx/html`: The path inside the container where the volume will be mounted.

3. Create the `index.html` file again using the `web_content` volume's mountpoint as found with `docker volume inspect`.

    echo "<h1>Hello from Mount Flag Volume!</h1>" | sudo tee /var/lib/docker/volumes/web_content/_data/index.html

4. Access `http://your_server_ip:8080`.

Volume Drivers

Docker volumes can be managed by different storage drivers. The `local` driver is the default, storing data on the host's filesystem. However, Docker supports other drivers for more advanced use cases, such as:

`local`: Default, stores data on the host.
`nfs`: For mounting volumes to an NFS share.
`rex-ray`: For integrating with storage platforms like AWS, Azure, or Ceph.
Custom drivers: You can develop or use third-party drivers.

To use a different driver, you specify it during volume creation:

sudo docker volume create --driver local --opt type=nfs --opt o=addr=192.168.1.100,rw --opt device=:/path/to/nfs/share my_nfs_volume

This example shows how to create a volume that mounts an NFS share. The `--opt` flags are driver-specific.

Bind Mounts

Bind mounts allow you to mount a file or directory from your host machine directly into a container. Unlike volumes, bind mounts are not managed by Docker; they are simply a reference to a path on the host.

Using Bind Mounts

Bind mounts are also configured using the `-v` or `--mount` flags.

Using the `-v` flag

The syntax for a bind mount with `-v` is `host-path:container-path`.

1. Create a directory on your host:

    sudo mkdir -p /opt/my_app_config
    echo "MY_APP_SETTING=production" | sudo tee /opt/my_app_config/app.env

2. Run a container, mounting this host directory into the container:

    sudo docker run -d --name app_container -p 8000:80 -v /opt/my_app_config:/etc/app/config ubuntu bash -c "cat /etc/app/config/app.env && sleep infinity"

   *   `-v /opt/my_app_config:/etc/app/config`: Mounts the host directory `/opt/my_app_config` to `/etc/app/config` inside the container.

3. Check the container logs to see the content of the mounted file:

    sudo docker logs app_container

   Expected output:

    MY_APP_SETTING=production

4. Modify the file on the host:

    echo "MY_APP_SETTING=development" | sudo tee /opt/my_app_config/app.env

5. Check the container logs again. You should see the updated value. This demonstrates that changes on the host are reflected immediately in the container.

Using the `--mount` flag

The syntax for a bind mount with `--mount` is `type=bind,source=host-path,target=container-path`.

1. Remove the previous container:

    sudo docker stop app_container
    sudo docker rm app_container

2. Run the container using `--mount`:

    sudo docker run -d --name app_container_mount -p 8001:80 --mount type=bind,source=/opt/my_app_config,target=/etc/app/config ubuntu bash -c "cat /etc/app/config/app.env && sleep infinity"

3. Check the logs:

    sudo docker logs app_container_mount

   You should see the current value of `MY_APP_SETTING`.

Bind Mounts vs. Volumes

Security Implications

**Bind Mounts:** Granting a container access to host directories via bind mounts can be a security risk. If a container is compromised, it could potentially access, modify, or delete files in the mounted host directory. Always mount only the necessary directories and ensure appropriate file permissions on the host. Avoid mounting sensitive system directories like `/` or `/etc`.
**Volumes:** Volumes are generally safer as they are managed by Docker and isolated within Docker's storage area. However, if you use custom volume drivers or mount volumes to network shares (like NFS), ensure those shares are properly secured.

Performance Considerations

**Volumes:** Docker volumes are generally optimized for performance, especially for I/O-intensive applications like databases. They leverage the host's filesystem but are managed in a way that can offer better throughput than direct bind mounts in some scenarios.
**Bind Mounts:** Performance can vary. Mounting directories with many small files can sometimes be slower than volumes. For development, bind mounts are excellent for rapid iteration as changes are immediately reflected. For production, volumes are often preferred for their robustness and potential performance benefits.

Troubleshooting

**"Error: No such file or directory" when using bind mounts:**

   *   **Cause:** The specified host path for the bind mount does not exist.
   *   **Solution:** Ensure the source directory or file exists on the host machine before running the container. Use `sudo mkdir -p /path/to/your/directory` to create it.

**Container cannot write to volume/bind mount:**

   *   **Cause:** Permission issues. The user inside the container does not have write permissions to the mounted directory.
   *   **Solution:**
       *   For volumes: Check the permissions of the directory on the host where the volume is mounted (e.g., `/var/lib/docker/volumes/my_volume/_data`). Ensure the user/group ID that the application runs as inside the container has write access. You might need to adjust ownership: `sudo chown -R 1000:1000 /var/lib/docker/volumes/my_volume/_data` (replace `1000:1000` with the correct UID/GID).
       *   For bind mounts: Check the permissions of the source directory on the host. The user running the Docker daemon usually has broad access, but the user inside the container might not.

**Volume not found:**

   *   **Cause:** The volume was not created or was removed.
   *   **Solution:** Use `docker volume ls` to check for the volume. If it's missing, recreate it using `docker volume create` or ensure it's mounted correctly with the `-v` or `--mount` flag.

**Data loss after container removal:**

   *   **Cause:** The data was stored in the container's writable layer, not in a persistent volume or bind mount.
   *   **Solution:** Always use `-v` or `--mount` to specify a persistent storage location for any data that needs to survive container restarts or removals.

Conclusion

Understanding and effectively using Docker volumes and bind mounts is crucial for building robust and stateful containerized applications. Volumes offer a Docker-managed, portable, and often performant solution for data persistence, while bind mounts provide direct access to the host filesystem, ideal for development workflows and configuration sharing. By choosing the appropriate storage mechanism and being mindful of security and performance implications, you can ensure your containerized applications handle data reliably.

Docker Storage and Volumes

Contents

Introduction

Prerequisites

Understanding Docker Storage

Docker Volumes

Creating and Managing Volumes

Explicit Volume Creation

Verifying Volume Creation

Implicit Volume Creation

Removing Volumes

Using Volumes with Containers

Using the `-v` flag

Using the `--mount` flag

Volume Drivers

Bind Mounts

Using Bind Mounts

Using the `-v` flag

Using the `--mount` flag

Bind Mounts vs. Volumes

Security Implications

Performance Considerations

Troubleshooting

Conclusion

Read Also

Navigation menu

Search