FastDFS distributed file system

Chapter 1 Introduction to FastDFS

1.1 distributed file system

Distributed file system is a software / software server, which can be used to manage files. However, the files managed by this software are usually not on one server node, but on multiple server nodes. These server nodes are connected through the network to form a huge file storage server cluster. These servers are used to store file resources and manage the files on these servers through a distributed file system.

Common distributed file systems include FastDFS, GFS, HDFS, Lustre, Ceph, GridFS, mogileFS, TFS, etc.

Comparison between distributed file system and traditional file system

Disadvantages of traditional methods

  • If the number of users is large and there are many IO operations, it will put great pressure on disk access

  • If the disk fails, it will cause data loss

  • Limited storage capacity

1.2 introduction to fastdfs

FastDFS is an open source lightweight distributed file system tailored for Internet applications. It is simple, flexible and efficient. It is developed in C language and open source by Alibaba.

FastDFS manages files. Its functions include file storage, file synchronization, file access (file upload, file download, file deletion), etc. It solves the problem of large-capacity file storage. It is especially suitable for online services with files as carriers, such as photo album websites, file websites, picture websites, video websites, etc.

FastDFS fully considers redundant backup, linear capacity expansion and other mechanisms, and pays attention to high availability, high performance and other indicators. Using FastDFS, it is easy to build a set of high-performance file server cluster to provide file upload, download and other services.

1.3 development history of fastdfs

​ The project was launched in April 2008, and the first version V1.00 was released in July. It was continuously upgraded to V1.29 within two years

​ V2.00 launched in August 2010

​ V3.00 launched in June 2011

V4.0.0 launched in October 2012

V5.0.0 launched in December 2013

Up to now, the latest version is V5.11 (released in June 2017)

The FastDFS system architecture has not been greatly adjusted since the first version was released. The high version is fully compatible with the data of the low version and can be upgraded smoothly. It is recommended to update and upgrade to the latest version

FastDFS code is hosted on github: https://github.com/happyfish100/fastdfs

1.4 FastDFS overall architecture

FastDFS file system consists of two parts, one is the client and the other is the server

The client usually refers to our program, such as our Java program to connect and operate FastDFS. Our Java program is a client. FastDFS provides proprietary API access. At present, it provides APIs of C, Java and PHP programming languages to access FastDFS file system.

The server consists of two parts: a tracker and a storage node

The tracker is mainly used for scheduling, recording the status information of storage nodes in the cluster in memory. It is the hub of front-end Client and back-end storage nodes. Because all relevant information is in memory, the performance of Tracker server is very high. Three in a large cluster (such as hundreds of group s) is enough.

The storage node is used to store files, including files and file attributes (meta data), which are saved to the storage server disk to complete all functions of file management: file storage, file synchronization and file access.

Chapter 2 Establishment of FastDFS environment

2.1 FastDFS installation

2.1.1 preparation before installation

(1) Check whether gcc, libevent and libevent devel are installed on Linux

  • yum list installed | grep gcc

  • yum list installed | grep libevent

  • yum list installed | grep libevent-devel

(2) If not, install

yum install gcc libevent libevent-devel -y

2.1.2 installing libfastcommon Library

libfastcommon library is a common C language function library required for the operation of FastDFS file system

Note: the latest version of v1.0.39 is incompatible with the latest version of FastDFS5.11. All the versions we use here are v1.0.36. Download address: https://github.com/happyfish100

(1) Upload the downloaded libfastcommon file to Linux (/ opt)

(2) Unzip the downloaded tar.gz package to the current directory

tar -zxvf libfastcommon-1.0.36.tar.gz

(3) Switch to the extracted libfastcommon directory

cd libfastcommon-1.0.36

(4) Execute the make script to compile

./make.sh

Note: if an error is reported during make compilation, make again after solving the error. Usually, the error is caused by the lack of some dependent libraries in Linux. Solve the error according to the error prompt

(5) Execute make install to install

./make.sh install

The libfastcommon library is now installed

2.1.3 installing FastDFS

FastDFS does not have a Windows version and cannot be used under windows.

FastDFS needs to be installed and deployed in Linux environment. We use fastdfs-5.11 (201901)

Download address: https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gz

(1) Upload the downloaded FastDFS file to Linux (/ opt)

(2) Unzip the downloaded tar.gz package to the current directory

tar -zxvf fastdfs-5.11.tar.gz

(3) Switch to the directory of FastDFS after decompression

cd fastdfs-5.11

(4) Execute the make script to compile

./make.sh

(5) Execute make install to install

./make.sh install

The FastDFS installation is now complete

All compiled files are stored in the / usr/bin directory

All configuration files are stored in / etc/fdfs directory

(6) View the effect after installation

A. View FastDFS related executables
ll /usr/bin/fdfs*

/usr/bin is the Linux environment variable, which can be viewed through echo $PATH

B. View the configuration file for FastDFS
ll /etc/fdfs/

(7) In addition, note that you need to copy the two files in the decompressed fastdfs-5.11/conf directory to / etc/fdfs /, otherwise there will be many strange problems to be solved later

cp http.conf /etc/fdfs/
cp mime.types /etc/fdfs/

These two files need to be used later, so copy them first

2.2 FastDFS configuration

2.2.1 remove the suffix of FastDFS configuration file in / etc/fdfs / directory

2.2.2 modify the tracker.conf file

By default, it points to the directory of FastDFS author Yu Qing. It doesn't exist on our machine. Please change it manually

base_path=/opt/fastdfs/tracker #Configure the directory where tracker stores data

2.2.3 modify the storage.conf file

base_path=/opt/fastdfs/storage          #storage data directory

store_path0=/opt/fastdfs/storage/files     #The directory where the files are actually stored

tracker_server=192.168.137.142:22122    #Registers the tracker address of the current storage node

2.2.4 create the directory specified above on the Linux server

/opt/fastdfs/tracker

/opt/fastdfs/storage

/opt/fastdfs/storage/files

2.2.5 then start FastDFS

2.3 FastDFS startup

Two scripts are required to start the FastDFS service:

2.3.1 start the tracker service of FastDFS

Execute in any directory:

fdfs_trackerd /etc/fdfs/tracker.conf

2.3.2 start the storage service of FastDFS

Execute in any directory:

fdfs_storaged /etc/fdfs/storage.conf

2.3.3 viewing the startup process

If there is an execution command to start, the start is successful

2.3.4 check whether the storage has been registered under the tracker

fdfs_monitor /etc/fdfs/storage.conf

2.3.5 after starting storage for the first time, a directory for storing files will be created under the configured path

2.4 FastDFS restart

2.4.1 restart tracker

fdfs_trackerd /etc/fdfs/tracker.conf restart

2.4.2 restart storage

fdfs_storaged /etc/fdfs/storage.conf restart

2.5 FastDFS shutdown

2.5.1 close the tracker and execute the command

Execute in any directory:

fdfs_trackerd /etc/fdfs/tracker.conf stop

2.5.2 close storage execution command

Execute in any directory:

fdfs_storaged /etc/fdfs/storage.conf stop

2.5.3 or kill to turn off fastdfs, but it is not recommended to use kill -9 to turn it off online, because it may cause file information synchronization problems

2.6 FastDFS test

After the FastDFS installation is completed, you can use fdfs_test script test file upload

2.6.1 before testing, you need to modify the client.conf configuration file and modify two configurations

  • base_path=/opt/fastdfs/client

  • tracker_server=192.168.179.128:22122

  • Create a client in the / opt/fastdfs / directory

2.6.2 test file upload

  • Prepare files to upload

  • Execute upload command

fdfs_test /etc/fdfs/client.conf upload /root/aa.txt 

This is FastDFS client test program v5.11

Copyright (C) 2008, Happy Fish / YuQing

FastDFS may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDFS source kit.
Please visit the FastDFS Home Page http://www.csource.org/ 
for more detail.

[2021-11-28 22:37:52] DEBUG - base_path=/opt/fastdfs/client, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

tracker_query_storage_store_list_without_group: 
	server 1. group_name=, ip_addr=192.168.137.142, port=23000

group_name=group1, ip_addr=192.168.137.142, port=23000
storage_upload_by_filename
group_name=group1, remote_filename=M00/00/00/wKiJjmGjlECAErA4AAAADwA12ic463.txt
source ip address: 192.168.137.142
file timestamp=2021-11-28 22:37:52
file size=15
file crc32=3529255
example file url: http://192.168.137.142/group1/M00/00/00/wKiJjmGjlECAErA4AAAADwA12ic463.txt
storage_upload_slave_by_filename
group_name=group1, remote_filename=M00/00/00/wKiJjmGjlECAErA4AAAADwA12ic463_big.txt
source ip address: 192.168.137.142
file timestamp=2021-11-28 22:37:52
file size=15
file crc32=3529255
example file url: http://192.168.137.142/group1/M00/00/00/wKiJjmGjlECAErA4AAAADwA12ic463_big.txt
  • Switch to the storage directory to view the file upload

2.6.3 example of file directory structure and name generated by fastdfs

2.6.4 deletion of test files

fdfs_delete_file /etc/fdfs/client.conf group1/File path to delete

Previously uploaded files: master file: M00 / 00 / 00 / wkijjmgjlecaera4aaadwa12ic463.txt and slave file: M00 / 00 / 00 / wkijmgjlecaera4aaadwa12ic463_ Big.txt - m that file is the attribute information file

fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKiJjmGjlECAErA4AAAADwA12ic463.txt
fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKiJjmGjlECAErA4AAAADwA12ic463_big.txt

2.6.5 notes

  • There is no cluster. By default, there is only one group, group1

  • The suffix containing - m is the attribute file (meta)

  • In Linux, no disk is virtual

Tags: Web Development FastDFS

Posted on Sun, 28 Nov 2021 15:47:03 -0500 by natasha_thomas