docker storage volume management

COW mechanism

Docker image is composed of multiple read-only layers. When the container is started, docker will load the read-only image layer and add a read-write layer at the top of the image stack.

If the running container modifies an existing file, the file will be copied from the read-only layer below the read-write layer to the read-write layer. The read-only version of the file still exists, but it has been hidden by the copy of the file in the read-write layer. This is the "copy on write (COW)" mechanism.

Because there are many layers of images across, the access efficiency will be very low. In order to avoid this disadvantage, we can use the storage volume mechanism.

What is a storage volume

Storage volume is to directly bind a directory existing in the local file system of the host to a directory on the file system inside the container. This means that when we write data in this directory in the container, the container will write its contents directly to the directory on the host that has established a binding relationship with this container.
The directory on the host that forms a binding relationship with the container is called a storage volume.

Benefits of using storage volumes

If all the valid data of the processes running in the container are saved in the storage volume and separated from the container's own file system, the benefit is that when the container is closed or even deleted, we don't have to worry about data loss as long as the storage directory bound to the container on the host computer is not deleted. Therefore, data persistence can be realized, which is separated from the life cycle of the container.

By managing containers in this way, we can break away from the restrictions of hosts, run containers on any host with docker deployed, and its data can be placed on a shared storage file system, such as nfs.

By default, the storage volume of Docker uses the local file system directory on the host, that is, the host has its own hard disk, which is not shared with other Docker hosts, and the storage volume used by the container started on this host is associated with a directory on the hard disk of this host.

This means that the container stops running on this host or is deleted and rebuilt. As long as it is associated with this directory on the hard disk, its data still exists. But if you start a new container on another host, the data is gone. If we manually mount the container data to an nfs server when creating the container, this problem is no longer a problem.

Why use storage volumes

If the container is closed and restarted, its data will not be affected, but if the Docker container is deleted, all its changes will be lost.

Therefore, Docker has the following problems:

  • It is stored in the jointly mounted file system and is not easy to access by the host
  • Inconvenient data sharing between containers
  • Delete the container and its data will be lost

To solve these problems, the solution is to use storage volumes.

Storage volume management mode

The Data Volume is automatically created when the container is initialized, and the data in the volume provided by base image completes replication during this period.

The original intention of Volume is to realize data persistence independent of the life cycle of the container. Therefore, when deleting the container, it will neither delete the Volume nor garbage collect the unreferenced Volume.

Storage volume provides Docker with a container independent data management mechanism. We can imagine the image as a static file, such as "program", and compare the volume to dynamic content, such as "data". So mirrors can be reused and volumes can be shared.

Volume realizes the separation of "program (image)" and "data (volume)", as well as the separation of "program (image)" and "host making image". When making image, users do not need to consider the environment of the host where the container running image is located.

Classification of storage volumes

Docker has two types of volumes. Each type has a mount point in the container, but its location on the host is different:

  • Bind mount volume
    -Points to a volume at a user specified location on the host file system
  • Docker managed volumes
    -The Docker daemon creates managed volumes in a portion of the host file system that is owned by Docker

Container data management

When using Docker, users often need to be able to view the data generated by applications in the container, or need to back up the data in the container, or even share the data among multiple containers, which inevitably involves the data management operation of the container.

There are two main ways to manage data in containers:

  • Data Volumes
  • Data Volumes Containers

Container Volume usage syntax:
Docker-managed volume


Bind mount volume


Using data volumes in containers

Create a data volume inside the container

Next, create a web container using nginx image and mount a data volume to the / webapp directory of the container:

//Download nginx image
[root@localhost ~]# docker pull nginx

//-P is the port that needs to be exposed to allow external access to the container
[root@localhost ~]#  docker run  --name web  -d -P  -v /webapp nginx
[root@localhost ~]# docker ps -a
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS                                     NAMES
8fc44ae50d4c   nginx     "/docker-entrypoint...."   4 minutes ago   Up 4 minutes>80/tcp, :::49153->80/tcp   web

//Enter container view
[root@localhost ~]# docker exec -it web /bin/bash
root@8fc44ae50d4c:/# ls
bin  boot  dev  docker-entrypoint.d  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  webapp

//Create file in container
root@8fc44ae50d4c:/# cd webapp/
root@8fc44ae50d4c:/webapp# touch 123
root@8fc44ae50d4c:/webapp# ls

//View mount location
[root@localhost ~]# docker inspect web

        "Mounts": [
                "Type": "volume",
                "Name": "3380c439f7cf73af73066563ed4cc419e914bbef89e431d144724308fe6c76f0",
                "Source": "/var/lib/docker/volumes/3380c439f7cf73af73066563ed4cc419e914bbef89e431d144724308fe6c76f0/_data",
                "Destination": "/webapp",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""

//Copy path view
[root@localhost ~]# cd /var/lib/docker/volumes/3380c439f7cf73af73066563ed4cc419e914bbef89e431d144724308fe6c76f0/_data
[root@localhost _data]# ls

Mount a host directory as a data volume

[root@localhost ~]# docker run  --name web01 -d -P -v /data:/webapp nginx
[root@localhost ~]# docker ps -a
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                                     NAMES
62374ca06fb9   nginx     "/docker-entrypoint...."   4 seconds ago    Up 3 seconds>80/tcp, :::49154->80/tcp   web01
8fc44ae50d4c   nginx     "/docker-entrypoint...."   11 minutes ago   Up 11 minutes>80/tcp, :::49153->80/tcp   web

//Enter container to create container
[root@localhost ~]# docker exec  -it web01 /bin/bash
root@62374ca06fb9:/# ls
bin  boot  dev  docker-entrypoint.d  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  webapp
root@62374ca06fb9:/# cd webapp/
root@62374ca06fb9:/webapp# touch abc
root@62374ca06fb9:/webapp# ls

//Enter the host directory to view
[root@localhost ~]# cd /data/
[root@localhost data]# ls

The above command loads the / data / directory of the host into the / webapp directory of the container:
This function is very convenient for testing. For example, users can place some programs or data in the local directory, and then run and use it in the container. In addition, the path of the local directory must be absolute. If the directory does not exist, Docker will automatically create it.

The default permission for Docker to mount data volumes is read-write (rw), and users can also specify it as read-only through (ro):

[root@localhost ~]# docker run --name web02 -d -P -v /data/:/webapp:ro nginx
[root@localhost ~]# docker exec  -it web02 /bin/bash
root@3dd5d890ba02:/# cd /webapp/
root@3dd5d890ba02:/webapp# touch qwe
touch: cannot touch 'qwe': Read-only file system

//It can be written on the host and synchronized to the container
[root@localhost data]# touch 123
[root@localhost data]# ls
123  abc

root@3dd5d890ba02:/# cd webapp/
root@3dd5d890ba02:/webapp# ls
123  abc

Mount a local host file as a data volume
-The v option can also mount a single file from the host into a container as a data volume:

[root@localhost ~]# docker run -it --rm -v ~/.bash_history:/.bash_history centos /bin/bash
[root@987ad9892214 /]# ls
bin  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@987ad9892214 /]# cat .bash_history  
wget -O /etc/yum.repos.d/CentOS-Base.repo
yum -y install wget
wget -O /etc/yum.repos.d/CentOS-Base.repo
cd /etc/yum.repos.d/
curl -o docker-ce.repo

In this way, you can record the command history entered in the container.

If you mount a file directly to the container and use file editing tools, including vi or sed, to modify the file content, inode may be changed, which will lead to errors. Therefore, the recommended method is to directly mount the directory where the file is located.

Data volume container

If users need to share some continuously updated data between containers, the simplest way is to use the data volume container. The data volume container is actually an ordinary container. It is specially used to provide data volumes for other containers to mount. The method is as follows:

First, create a data volume container dbdata and mount a data volume in it to / dbdata:

[root@localhost ~]# docker run  --name db -dit -v /dbdata centos
[root@localhost ~]# docker ps 
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                                     NAMES
28184a02bee2   centos    "/bin/bash"              13 seconds ago   Up 12 seconds                                             db

Then you can use – volumes from in other containers to mount the data volumes in the dbdata container, for example, create two containers db1 and db2, and mount the data volumes from the dbdata container:

[root@localhost ~]# docker run --name  db1 -dit --volumes-from db  centos
[root@localhost ~]# docker run --name  db2 -dit --volumes-from db  centos

[root@localhost ~]# docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED             STATUS             PORTS                                     NAMES
407c16690c8d   centos    "/bin/bash"              49 minutes ago      Up 49 minutes                                                db2
ccba3a2bf2cf   centos    "/bin/bash"              50 minutes ago      Up 50 minutes                                                db1
28184a02bee2   centos    "/bin/bash"              53 minutes ago      Up 53 minutes                                                db

At this point, containers db1 and db2 mount the same data volume to the same / dbdata directory. If any of the three containers writes in this directory, other containers can see it.
For example, create a test file in the db1 container:

[root@localhost ~]# docker exec -it db1 /bin/bash
[root@ccba3a2bf2cf /]# cd dbdata/
[root@ccba3a2bf2cf dbdata]# touch 123
[root@ccba3a2bf2cf dbdata]# ls

View in the db2 container:

[root@localhost ~]# docker exec  -it db2 /bin/bash
[root@407c16690c8d /]# cd dbdata/
[root@407c16690c8d dbdata]# ls

You can use the – volumes from parameter multiple times to mount multiple data volumes from multiple containers. You can also mount data volumes from other containers that have container volumes mounted:

[root@localhost ~]# docker run  --name db3 -dit --volumes-from db1 centos
[root@localhost ~]# docker exec  -it db3 /bin/bash
[root@725822c16d4b /]# cd dbdata/
[root@725822c16d4b dbdata]# ls

The container on which the data volume is mounted using the – volumes from parameter does not need to remain running itself.
If you delete a mounted container (including dbdata, db1, and db2), the data volume is not automatically deleted. If you want to delete a data volume, you must explicitly use the docker rm -v command to specify that the associated container be deleted at the same time when deleting the last container that still has it mounted.

[root@localhost ~]# docker rm -f db1
[root@localhost ~]# docker ps -a
CONTAINER ID   IMAGE     COMMAND                  CREATED              STATUS              PORTS                                     NAMES
725822c16d4b   centos    "/bin/bash"              About a minute ago   Up About a minute                                             db3
407c16690c8d   centos    "/bin/bash"              56 minutes ago       Up 56 minutes                                                 db2
28184a02bee2   centos    "/bin/bash"              About an hour ago    Up About an hour                                              db

[root@localhost ~]# docker exec  -it db3 /bin/bash
[root@725822c16d4b /]# cd dbdata/
[root@725822c16d4b dbdata]# ls
[root@localhost ~]# docker rm -f db2 db3
[root@localhost ~]# docker exec  -it db /bin/bash
[root@28184a02bee2 /]# cd dbdata/
[root@28184a02bee2 dbdata]# ls
[root@localhost ~]# docker rm -f -v db 

Tags: Linux Operation & Maintenance Docker Container

Posted on Mon, 06 Dec 2021 14:43:05 -0500 by satal keto