docker storage volume management

COW mechanism

Docker image is composed of multiple read-only layers. When the container is started, docker will load the read-only image layer and add a read-write layer at the top of the image stack.

If the running container modifies an existing file, the file will be copied from the read-only layer below the read-write layer to the read-write layer. The read-only version of the file still exists, but it has been hidden by the copy of the file in the read-write layer. This is the "copy on write (COW)" mechanism.

Because there are many layers of images across, the access efficiency will be very low. In order to avoid this disadvantage, we can use the storage volume mechanism.

What is a storage volume

Storage volume is to directly bind a directory existing in the local file system of the host to a directory on the file system inside the container. This means that when we write data in this directory in the container, the container will write its contents directly to the directory on the host that has established a binding relationship with this container.
The directory on the host that forms a binding relationship with the container is called a storage volume.

Benefits of using storage volumes

If all the valid data of the processes running in the container are saved in the storage volume and separated from the container's own file system, the benefit is that when the container is closed or even deleted, we don't have to worry about data loss as long as the storage directory bound to the container on the host computer is not deleted. Therefore, data persistence can be realized, which is separated from the life cycle of the container.

By managing containers in this way, we can break away from the restrictions of hosts, run containers on any host with docker deployed, and its data can be placed on a shared storage file system, such as nfs.

By default, the storage volume of Docker uses the local file system directory on the host, that is, the host has its own hard disk, which is not shared with other Docker hosts, and the storage volume used by the container started on this host is associated with a directory on the hard disk of this host.

This means that the container stops running on this host or is deleted and rebuilt. As long as it is associated with this directory on the hard disk, its data still exists. But if you start a new container on another host, the data is gone. If we manually mount the container data to an nfs server when creating the container, this problem is no longer a problem.

Why use storage volumes

If the container is closed and restarted, its data will not be affected, but if the Docker container is deleted, all its changes will be lost.

Therefore, Docker has the following problems:

  • It is stored in the jointly mounted file system and is not easy to access by the host
  • Inconvenient data sharing between containers
  • Delete the container and its data will be lost

To solve these problems, the solution is to use storage volumes.

Storage volume management mode

The Data Volume is automatically created when the container is initialized, and the data in the volume provided by base image completes replication during this period.

The original intention of Volume is to realize data persistence independent of the life cycle of the container. Therefore, when deleting the container, it will neither delete the Volume nor garbage collect the unreferenced Volume.

Storage volume provides Docker with a container independent data management mechanism. We can imagine the image as a static file, such as "program", and compare the volume to dynamic content, such as "data". So mirrors can be reused and volumes can be shared.

Volume realizes the separation of "program (image)" and "data (volume)", as well as the separation of "program (image)" and "host making image". When making image, users do not need to consider the environment of the host where the container running image is located.

Classification of storage volumes

Docker has two types of volumes. Each type has a mount point in the container, but its location on the host is different:

  • Bind mount volume
    -Points to a volume at a user specified location on the host file system
  • Docker managed volumes
    -The Docker daemon creates managed volumes in a portion of the host file system that is owned by Docker

Container data management

When using Docker, users often need to be able to view the data generated by applications in the container, or need to back up the data in the container, or even share the data among multiple containers, which inevitably involves the data management operation of the container.

There are two main ways to manage data in containers:

  • Data Volumes
  • Data Volumes Containers

Container Volume usage syntax:
Docker-managed volume

docker run -it --name CONTAINER_NAME -v VOLUMEDIR IMAGE_NAME

Bind mount volume

docker run -it --name CONTAINER_NAME -v HOSTDIR:VOLUMEDIR IMAGE_NAME

Using data volumes in containers

Create a data volume inside the container

Next, create a web container using nginx image and mount a data volume to the / webapp directory of the container:

//Download nginx image
[root@localhost ~]# docker pull nginx

//-P is the port that needs to be exposed to allow external access to the container
[root@localhost ~]#  docker run  --name web  -d -P  -v /webapp nginx
8fc44ae50d4cd72c224896d15d5c2d330d267d6922fdb78bb40a207812ae17d4
[root@localhost ~]# docker ps -a
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS                                     NAMES
8fc44ae50d4c   nginx     "/docker-entrypoint...."   4 minutes ago   Up 4 minutes   0.0.0.0:49153->80/tcp, :::49153->80/tcp   web

//Enter container view
[root@localhost ~]# docker exec -it web /bin/bash
root@8fc44ae50d4c:/# ls
bin  boot  dev  docker-entrypoint.d  docker-entrypoint.sh  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  webapp
root@8fc44ae50d4c:/# 

//Create file in container
root@8fc44ae50d4c:/# cd webapp/
root@8fc44ae50d4c:/webapp# touch 123
root@8fc44ae50d4c:/webapp# ls
123

//View mount location
[root@localhost ~]# docker inspect web

.....slightly
},
        "Mounts": [
            {
                "Type": "volume",
                "Name": "3380c439f7cf73af73066563ed4cc419e914bbef89e431d144724308fe6c76f0",
                "Source": "/var/lib/docker/volumes/3380c439f7cf73af73066563ed4cc419e914bbef89e431d144724308fe6c76f0/_data",
                "Destination": "/webapp",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
....slightly

//Copy path view
[root@localhost ~]# cd /var/lib/docker/volumes/3380c439f7cf73af73066563ed4cc419e914bbef89e431d144724308fe6c76f0/_data
[root@localhost _data]# ls
123

Mount a host directory as a data volume

[root@localhost ~]# docker run  --name web01 -d -P -v /data:/webapp nginx
62374ca06fb939f73bc6c78944351ae4c5fa33f6734b7b326e7874eea1889242
[root@localhost ~]# docker ps -a
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                                     NAMES
62374ca06fb9   nginx     "/docker-entrypoint...."   4 seconds ago    Up 3 seconds    0.0.0.0:49154->80/tcp, :::49154->80/tcp   web01
8fc44ae50d4c   nginx     "/docker-entrypoint...."   11 minutes ago   Up 11 minutes   0.0.0.0:49153->80/tcp, :::49153->80/tcp   web

//Enter container to create container
[root@localhost ~]# docker exec  -it web01 /bin/bash
root@62374ca06fb9:/# ls
bin  boot  dev  docker-entrypoint.d  docker-entrypoint.sh  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  webapp
root@62374ca06fb9:/# cd webapp/
root@62374ca06fb9:/webapp# touch abc
root@62374ca06fb9:/webapp# ls
abc

//Enter the host directory to view
[root@localhost ~]# cd /data/
[root@localhost data]# ls
abc

The above command loads the / data / directory of the host into the / webapp directory of the container:
This function is very convenient for testing. For example, users can place some programs or data in the local directory, and then run and use it in the container. In addition, the path of the local directory must be absolute. If the directory does not exist, Docker will automatically create it.

The default permission for Docker to mount data volumes is read-write (rw), and users can also specify it as read-only through (ro):

[root@localhost ~]# docker run --name web02 -d -P -v /data/:/webapp:ro nginx
3dd5d890ba0208680f6bb1e39d0078612d2592b18bb8df612a9abe4b9f9e159c
[root@localhost ~]# docker exec  -it web02 /bin/bash
root@3dd5d890ba02:/# cd /webapp/
root@3dd5d890ba02:/webapp# touch qwe
touch: cannot touch 'qwe': Read-only file system

//It can be written on the host and synchronized to the container
[root@localhost data]# touch 123
[root@localhost data]# ls
123  abc

root@3dd5d890ba02:/# cd webapp/
root@3dd5d890ba02:/webapp# ls
123  abc

Mount a local host file as a data volume
-The v option can also mount a single file from the host into a container as a data volume:

[root@localhost ~]# docker run -it --rm -v ~/.bash_history:/.bash_history centos /bin/bash
[root@987ad9892214 /]# ls
bin  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@987ad9892214 /]# cat .bash_history  
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-8.repo
yum -y install wget
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-8.repo
cd /etc/yum.repos.d/
curl -o docker-ce.repo https://mirrors.tuna.tsinghua.edu.cn/docker-ce/linux/centos/docker-ce.repo
....slightly

In this way, you can record the command history entered in the container.

If you mount a file directly to the container and use file editing tools, including vi or sed, to modify the file content, inode may be changed, which will lead to errors. Therefore, the recommended method is to directly mount the directory where the file is located.

Data volume container

If users need to share some continuously updated data between containers, the simplest way is to use the data volume container. The data volume container is actually an ordinary container. It is specially used to provide data volumes for other containers to mount. The method is as follows:

First, create a data volume container dbdata and mount a data volume in it to / dbdata:

[root@localhost ~]# docker run  --name db -dit -v /dbdata centos
28184a02bee21dea5aefff86477288716a6c3aee136f6f0bf5693bf99692c608
[root@localhost ~]# docker ps 
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                                     NAMES
28184a02bee2   centos    "/bin/bash"              13 seconds ago   Up 12 seconds                                             db

Then you can use – volumes from in other containers to mount the data volumes in the dbdata container, for example, create two containers db1 and db2, and mount the data volumes from the dbdata container:

[root@localhost ~]# docker run --name  db1 -dit --volumes-from db  centos
ccba3a2bf2cf2c198b6888e07e914435c9a3f626dd776895eb8275a8e310a35d
[root@localhost ~]# docker run --name  db2 -dit --volumes-from db  centos
407c16690c8d5ba4e89e0c88c0bd000fc852737f433e6f6ec1d0c7f782c11a20

[root@localhost ~]# docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED             STATUS             PORTS                                     NAMES
407c16690c8d   centos    "/bin/bash"              49 minutes ago      Up 49 minutes                                                db2
ccba3a2bf2cf   centos    "/bin/bash"              50 minutes ago      Up 50 minutes                                                db1
28184a02bee2   centos    "/bin/bash"              53 minutes ago      Up 53 minutes                                                db

At this point, containers db1 and db2 mount the same data volume to the same / dbdata directory. If any of the three containers writes in this directory, other containers can see it.
For example, create a test file in the db1 container:

[root@localhost ~]# docker exec -it db1 /bin/bash
[root@ccba3a2bf2cf /]# cd dbdata/
[root@ccba3a2bf2cf dbdata]# touch 123
[root@ccba3a2bf2cf dbdata]# ls
123

View in the db2 container:

[root@localhost ~]# docker exec  -it db2 /bin/bash
[root@407c16690c8d /]# cd dbdata/
[root@407c16690c8d dbdata]# ls
123

You can use the – volumes from parameter multiple times to mount multiple data volumes from multiple containers. You can also mount data volumes from other containers that have container volumes mounted:

[root@localhost ~]# docker run  --name db3 -dit --volumes-from db1 centos
725822c16d4bfe0666358f9794049f3418885f46b7392d32252fd2e9acf4fe4a
[root@localhost ~]# docker exec  -it db3 /bin/bash
[root@725822c16d4b /]# cd dbdata/
[root@725822c16d4b dbdata]# ls
123

The container on which the data volume is mounted using the – volumes from parameter does not need to remain running itself.
If you delete a mounted container (including dbdata, db1, and db2), the data volume is not automatically deleted. If you want to delete a data volume, you must explicitly use the docker rm -v command to specify that the associated container be deleted at the same time when deleting the last container that still has it mounted.

[root@localhost ~]# docker rm -f db1
db1
[root@localhost ~]# docker ps -a
CONTAINER ID   IMAGE     COMMAND                  CREATED              STATUS              PORTS                                     NAMES
725822c16d4b   centos    "/bin/bash"              About a minute ago   Up About a minute                                             db3
407c16690c8d   centos    "/bin/bash"              56 minutes ago       Up 56 minutes                                                 db2
28184a02bee2   centos    "/bin/bash"              About an hour ago    Up About an hour                                              db

[root@localhost ~]# docker exec  -it db3 /bin/bash
[root@725822c16d4b /]# cd dbdata/
[root@725822c16d4b dbdata]# ls
123
[root@localhost ~]# docker rm -f db2 db3
db2
db3
[root@localhost ~]# docker exec  -it db /bin/bash
[root@28184a02bee2 /]# cd dbdata/
[root@28184a02bee2 dbdata]# ls
123
[root@localhost ~]# docker rm -f -v db 
db

Tags: Linux Operation & Maintenance Docker Container

Posted on Mon, 06 Dec 2021 14:43:05 -0500 by satal keto