summary
- Federated file system (unifs) is a lightweight high-performance hierarchical file system, which supports the integration of files in the file system
The modification information is submitted as one time and stacked layer by layer. At the same time, different directories can be mounted to the same virtual file system,
The application sees the end result of the mount. - It supports setting read-only, read-write and whiteout able permissions for each member directory (a branch similar to Git)
- The file system is layered, and the branch of readonly permission can be logically modified (incrementally, it does not affect the readonly part)
- Generally, Union FS has two purposes. On the one hand, it can hang multiple disk s in the same directory. On the other hand, it is more commonly used to combine a readonly branch and a writeable branch.
- Federated file system is the technical basis for realizing Docker image
docker startup process
- During initialization, rootfs is loaded and checked in readonly mode. However, next, a rootfs is loaded in union mount mode
The readwrite file system is mounted on the rootfs of readonly; - And it is allowed to set the FS (file system) of the lower layer to readonly again and stack up.
- Such a set of readonly and a writeable structure constitute the runtime state of a container, and each FS is called an FS layer.
docker write operation principle
Because the image has sharing characteristics, the operation of the container writable layer needs to rely on the write time replication and time allocation mechanism provided by the storage driver to support the modification of the container writable layer and improve the utilization of storage and memory resources.
- Copy on write
- Copy on write, i.e. copy on write.
- An image can be used by multiple containers, but it does not need to make multiple copies in memory and disk.
- When the file provided by the image needs to be modified, the file will be copied from the image file system to the writable layer file system of the container for modification, and the files in the image will not be changed.
- The modification of files by different containers are independent and independent of each other.
- Time allocation
- Time allocation allocates space on demand rather than in advance, that is, space is allocated only after a file is created.
Overlay 2 storage driver
Storage driver overview
- In the history of docker, there are many kinds of storage drivers, such as aufs, overlay FS, devicemapper, btrfs, etc.
- At present, the commonly used storage driver is overlay fs2
- In overlay 2, the lower layer of the directory is called lower dir (image layer), the upper layer is called Upper Dir (container layer), and the result of joint mounting is called merged
- The overlay 2 file system supports 128 layers at most, which means that your Dockerfile can only write 128 lines at most
Overlay 2 exercise
1. Create overlay file system
# Create four directories mkdir upper lower merged work # Add content to these four directories respectively echo "I'm from lower" > lower/in_lower.txt echo "I'm from upper" > upper/in_upper.txt echo "I'm from lower" > lower/in_both.txt echo "I'm from upper" > upper/in_both.txt # Mount the upper lower work directory to the merged directory sudo mount -t overlay overlay \ -o lowerdir=./lower,upperdir=./upper,workdir=./work \ ./merged
2. Catalog description
We can see that a mount command of OverlayFS involves four categories of directories: lower, upper, merged and work. What is the relationship between them?
First, the lowest "lower /", that is, the Lower Dir in the mount directory.
In overlay FS, the file in the bottom layer will not be modified. You can think of it as read-only. OverlayFS supports multiple lowerdir s.
Then we look at "uppder /", which is the upper layer (upperdir) in the mount two-tier directory. In overlay FS, if there are file creation, modification and deletion operations, they will be reflected in this layer, which is readable and writable.
Then comes the top "merged", which is the mount point directory and the directory seen by the user. The user's actual file operation is carried out here.
There is also a "work /" directory, which is not in the figure. It is just a directory for storing temporary files. If there are file modifications in overlay FS, the files will be temporarily stored here in the intermediate process.
3. File operation
If we do file operations in the merged / directory, these three types are included.
First, create a new file, which will appear in the upper / directory.
The second is to delete the file. If we delete "in_upper.txt", the file will disappear in the upper / directory. If you delete "in_lower.txt", the "in_lower.txt" file in the lower / directory will not change, but a special file is added in the upper / directory to tell OverlayFS that the "in_lower.txt" file cannot appear in merged /, which means that it has been deleted
Another operation is to modify the file, similar to if you modify "in"_ "Lower. TXT", then a new "in" will be created in the upper / directory_ Lower.txt "file, containing the updated content, while the original actual file in lower /" in_lower.txt "will not change.
Application of overlay in docker
See how overlay 2 stores image files through ubuntu:16.04
1. pull mirror
# docker pull ubuntu:16.04 16.04: Pulling from library/ubuntu 58690f9b18fc: Pull complete b51569e7c507: Pull complete da8ef40b9eca: Pull complete fb15d46c38dc: Pull complete Digest: sha256:454054f5bbd571b088db25b662099c6c7b3f0cb78536a2077d54adc48f00cd68 Status: Downloaded newer image for ubuntu:16.04 docker.io/library/ubuntu:16.04 You can see that the image is divided into four layers
2. View overlay 2 directory
# ls -l /var/lib/docker/overlay2/ total 0 drwx-----x. 3 root root 47 Oct 13 16:34 1133896ce991157dd016815c510609ce9b8dedd379a9ca9567015489bb0cbd28 drwx-----x. 4 root root 72 Oct 13 16:34 610a3087aee462a80b92b7e49e696d0cd534888c3946892bf772e107994af341 drwx-----x. 4 root root 72 Oct 13 16:34 76fc1a6fad04775ee034ea2ae03b8ad246e468ebac2fa12f842f7108918629d1 drwx-----x. 4 root root 55 Oct 13 16:34 7f59c5e2adc963612e1bf8202b91459507ce69190f6a7118eb46d39292e38937 brw-------. 1 root root 253, 0 Oct 12 02:32 backingFsBlockDev drwx-----x. 2 root root 142 Oct 13 16:34 l
You can see that there are four mirror layer directories and an l directory under the overlay 2 directory. Let's first check the contents of the l Directory:
# ls -l l total 0 lrwxrwxrwx. 1 root root 72 Oct 13 16:34 AVIMWR2WDK2V56X4FH73CQBSEU -> ../610a3087aee462a80b92b7e49e696d0cd534888c3946892bf772e107994af341/diff lrwxrwxrwx. 1 root root 72 Oct 13 16:34 Q245O4UVMNEC2JSCDOSHOMMRLE -> ../7f59c5e2adc963612e1bf8202b91459507ce69190f6a7118eb46d39292e38937/diff lrwxrwxrwx. 1 root root 72 Oct 13 16:34 RP7HB5ZQRKZG6SM33H3AZKA4LK -> ../1133896ce991157dd016815c510609ce9b8dedd379a9ca9567015489bb0cbd28/diff lrwxrwxrwx. 1 root root 72 Oct 13 16:34 WWJF2YUSDMDXPY7M4II4KCH3R2 -> ../76fc1a6fad04775ee034ea2ae03b8ad246e468ebac2fa12f842f7108918629d1/diff
You can see that the l directory is a pile of soft connections. Soft connect some short random strings to the diff folder of the image layer to avoid reaching the length limit of the mount command parameter.
Let's view the file contents under any image layer
# ls -l 7f59c5e2adc963612e1bf8202b91459507ce69190f6a7118eb46d39292e38937/ total 8 drwxr-xr-x. 3 root root 17 Oct 13 16:34 diff -rw-r--r--. 1 root root 26 Oct 13 16:34 link -rw-r--r--. 1 root root 86 Oct 13 16:34 lower drwx------. 2 root root 6 Oct 13 16:34 work
Note: the content of the link file of the mirror layer is the short ID of the mirror layer, the diff folder is the change content of the mirror layer, and the lower file is the short ID of all the parent layer images of the layer.
3. Use the docker image inspect command to view the hierarchical relationship of an image
# docker inspect ubuntu:16.04 ...... "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/76fc1a6fad04775ee034ea2ae03b8ad246e468ebac2fa12f842f7108918629d1/diff:/var/lib/docker/overlay2/610a3087aee462a80b92b7e49e696d0cd534888c3946892bf772e107994af341/diff:/var/lib/docker/overlay2/1133896ce991157dd016815c510609ce9b8dedd379a9ca9567015489bb0cbd28/diff", "MergedDir": "/var/lib/docker/overlay2/7f59c5e2adc963612e1bf8202b91459507ce69190f6a7118eb46d39292e38937/merged", "UpperDir": "/var/lib/docker/overlay2/7f59c5e2adc963612e1bf8202b91459507ce69190f6a7118eb46d39292e38937/diff", "WorkDir": "/var/lib/docker/overlay2/7f59c5e2adc963612e1bf8202b91459507ce69190f6a7118eb46d39292e38937/work" }, "Name": "overlay2" }, ......
MergedDir represents the directory of the current image layer stored in overlay 2, LowerDir represents the parent relationship of the current image, separated by a colon, and the last colon represents the lowest layer of the image.
4. Run the image as a container:
# docker run --name=ubuntu -d ubuntu:16.04 sleep 3600
5. Use the docker inspect command to view the working directory of the container
# docker inspect ubuntu ...... "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/9155d11f6bcdb7865eb9aa57f92c730ed16107912b675697396e142e7b80d155-init/diff:/var/lib/docker/overlay2/7f59c5e2adc963612e1bf8202b91459507ce69190f6a7118eb46d39292e38937/diff:/var/lib/docker/overlay2/76fc1a6fad04775ee034ea2ae03b8ad246e468ebac2fa12f842f7108918629d1/diff:/var/lib/docker/overlay2/610a3087aee462a80b92b7e49e696d0cd534888c3946892bf772e107994af341/diff:/var/lib/docker/overlay2/1133896ce991157dd016815c510609ce9b8dedd379a9ca9567015489bb0cbd28/diff", "MergedDir": "/var/lib/docker/overlay2/9155d11f6bcdb7865eb9aa57f92c730ed16107912b675697396e142e7b80d155/merged", "UpperDir": "/var/lib/docker/overlay2/9155d11f6bcdb7865eb9aa57f92c730ed16107912b675697396e142e7b80d155/diff", "WorkDir": "/var/lib/docker/overlay2/9155d11f6bcdb7865eb9aa57f92c730ed16107912b675697396e142e7b80d155/work" }, "Name": "overlay2" }, ......
The link and lower files have the same functions as the image layer. The link file content is the short ID of the container layer, and the lower file is the short ID of the image of all parent layers of the layer. The diff directory is the read-write layer of the container, and the modified files in the container will appear in the diff. The merged directory is the result of the combined mounting of hierarchical files, and it is also the working directory in the container.
Generally speaking, overlay 2 stores files in this way: overlay 2 places both the image layer and the container layer in a separate directory and has a unique ID. each layer only stores changed files. Finally, the joint mount technology is used to mount all files of the container layer and the image layer into the container, so that the complete system files can be seen in the container.