Experience of FastDFS cluster

FastDFS cluster

FastDFS is an open-source lightweight distributed file system, which manages files. Its functions include file storage, file synchronization, file access (file upload, file download), etc. It solves the problems of mass storage and load balancing. It is especially suitable for online services based on documents, such as album websites, video websites, etc.

FastDFS is customized for the Internet, fully considering redundant backup, load balancing, linear capacity expansion and other mechanisms, and focusing on high availability, high performance and other indicators. Using FastDFS, it is easy to build a set of high-performance file server cluster to provide file upload, download and other services.

Cluster architecture

Take a look at the cluster picture provided by the official website, as shown below:

As you can see,

There can be multiple clients on the Client side, that is, multiple clients can access the FastDFS cluster service at the same time;

Tracker is a tracker, which is responsible for coordinating the interaction between Client and Storage. In order to achieve high availability, multiple trackers need to be used as the tracker;

Storage is specially used to store things, and it is stored in groups. Each group can have multiple devices, and the contents of these devices are completely consistent. This is also for high availability. When the existing group capacity is not enough, we can expand horizontally, that is to say, add more components to achieve the purpose of expansion. In addition, it should be noted that if the capacity of A group of devices is not the same, for example, if the capacity of device A is 80G and the capacity of device B is 100G, then the capacity of the group where the two devices are located will be subject to the small capacity, that is to say, when the size of the stored things exceeds 80G, we will not be able to store them in the group.

The Client side also interacts with the Tracker cluster when interacting with Storage. The popular point is that Storage reports and registers with the Tracker cluster, telling the Tracker which location is still free and how much space is left.

File upload

See a schematic diagram of file upload

Explanation:

① . if the Client wants to upload the image, it first asks the Tracker;

② . after checking the registration information, the Tracker will tell the Client which storage is currently idle, and the Tracker will return the IP and port number to the Client;

③ . after the Client gets the IP and port number, it no longer needs to upload the image directly to Storage through Tracker. When Storage saves the image, it will report to Tracker to tell Tracker whether it still has the remaining space and the size of the remaining space.

④ . after reporting, Storage returns the address of the image stored on the server to the Client, which can be used by the Client to access the image.

File download

Take a look at the file download flowchart

Explanation:

① . the client asks the tracker for the storage of the downloaded file. The parameter is the file ID (volume name and file name);
② . tracker returns an available storage;
③ , client directly communicates with storage to complete the file download.

Set up FastDFS cluster

Take a look at the architecture of our online FastDFS cluster

We use 6 virtual machines and only use group1. If you need to add a group, please refer to the instructions at the end of the article to add it.
Operating system installation reference address https://blog.csdn.net/qq_23435961/article/details/106693857

name IP address application
tracker01 192.168.0.1 FastDFS,libfastcommon
tracker02 192.168.0.2 FastDFS,libfastcommon
storage01 192.168.0.10 FastDFS,libfastcommon,nginx,fastdfs-nginx-module
storage02 192.168.0.11 FastDFS,libfastcommon,nginx,fastdfs-nginx-module
nginx01 192.168.0.100 nginx,keepalived
nginx02 192.168.0.101 nginx,keepalived
VIP 192.168.0.200 -

Software Description:

name explain
CentOS 7.6 (installation system)
libfastcommon Some common function packages separated by FastDFS
FastDFS FastDFS Ontology
fastdfs-nginx-module FastDFS and nginx correlation module, to solve the problem of synchronization delay within the group
nginx nginx 1.16.1 (the latest version that YUM can install under CentOS 7)

1. Compiling environment preparation

#Install build environment
$ yum groups install Development Tools -y
$ yum install perl -y
$ mkdir -p /opt/fastdfs /data/fastdfs
$ cd /opt/fastdfs  #Prepare for the next download

Note: the / opt/fastdfs directory holds all installation packages. / the data/fastdfs directory is the data storage location.

2. Install libfastcommon

#Download File
$ wget https://github.com/happyfish100/libfastcommon/archive/V1.0.39.tar.gz
$ mv V1.0.39.tar.gz libfastcommon-1.0.39.tar.gz
$ tar -zxvf libfastcommon-1.0.39.tar.gz
$ cd libfastcommon-1.0.39/
$ ./make.sh
$ ./make.sh install

3. Install FastDFS

$ wget https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gz
$ mv V5.11.tar.gz fastdfs-5.11.tar.gz
$ tar -zxvf fastdfs-5.11.tar.gz
$ cd fastdfs-5.11/
$ ./make.sh
$ ./make.sh install
#Profile preparation
$ cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf #tracker node
$ cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf #storage node
$ cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf #Client files, for testing
$ cp /opt/fastdfs/fastdfs-5.11/conf/http.conf /etc/fdfs/ #For nginx access
$ cp /opt/fastdfs/fastdfs-5.11/conf/mime.types /etc/fdfs/ #For nginx access
① . tracker server configuration:
$ vim /etc/fdfs/tracker.conf
#The content to be modified is as follows
port=22122 # tracker server port (22122 by default, generally not modified)
base_path=/data/fastdfs # Root directory where logs and data are stored
#Edit startup file
$ vim /usr/lib/systemd/system/fastdfs-tracker.service
[Unit]
Description=The FastDFS File server
After=network.target remote-fs.target nss-lookup.target
 
[Service]
Type=forking
ExecStart=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start
ExecStop=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf stop
ExecRestart=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart
 
[Install]
WantedBy=multi-user.target

Start tracker server

$ systemctl daemon-reload
$ systemctl enable fastdfs-tracker.service
$ systemctl start fastdfs-tracker.service
$ netstat -tulnp #Check whether the service is started and whether the port is open
② . storage server configuration
$ vim /etc/fdfs/storage.conf
#The content to be modified is as follows
port=23000 # storage service port (23000 by default, generally not modified)
base_path=/data/fastdfs # Data and log file storage root
store_path0=/data/fastdfs # First storage directory
tracker_server=192.168.0.1:22122 # tracker server IP and port
tracker_server=192.168.0.2:22122 # tracker server IP and port
http.server_port=8888 # http access to the file port (8888 by default, modified according to the situation, consistent with nginx)
#Edit startup file
$ vim /usr/lib/systemd/system/fastdfs-storage.service
 
[Unit]
Description=The FastDFS File server
After=network.target remote-fs.target nss-lookup.target
 
[Service]
Type=forking
ExecStart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start
ExecStop=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop
ExecRestart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart
 
[Install]
WantedBy=multi-user.target

Start storage server

$ systemctl enable fastdfs-storage.service
$ systemctl start fastdfs-storage.service
$ netstat -tulnp #Check whether the service is started and whether the port is open
#View cluster status
$ fdfs_monitor /etc/fdfs/storage.conf list
[2018-11-06 00:00:00] DEBUG - base_path=/data/fastdfs/storage, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
server_count=2, server_index=0
tracker server is 192.168.0.1:22122
group count: 1
Group 1:
...
③ . Client configuration
$ vim /etc/fdfs/client.conf
#The content to be modified is as follows
base_path=/data/fastdfs
tracker_server=192.168.0.1:22122 # tracker server IP and port
tracker_server=192.168.0.2:22122 # tracker server IP and port
 
#After saving, the test returns an ID indicating success, such as group1/M00/00/00/xx.tar.gz
$ fdfs_upload_file /etc/fdfs/client.conf testfile

4. Install nginx and fastdfs nginx module

Download nginx module

$ wget https://github.com/happyfish100/fastdfs-nginx-module/archive/V1.20.tar.gz
$ mv V1.20.tar.gz fastdfs-nginx-module-1.20.tar.gz
$ tar -zxvf fastdfs-nginx-module-1.20.tar.gz
 
#Configure config file
$ vi fastdfs-nginx-module-1.20/src/config
//The modification is as follows: (replace local,:% s+/usr/local/+/usr/+g):
ngx_module_incs="/usr/include/fastdfs /usr/include/fastcommon/"
CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"
 
#Configure mod_fastdfs.conf file
$ cp /opt/fastdfs/fastdfs-nginx-module-1.20/src/mod_fastdfs.conf /etc/fdfs
$ vim /etc/fdfs/mod_fastdfs.conf
//The amendment is as follows:
connect_timeout=10  #Client connection timeout
tracker_server=192.168.0.1:22122  #Tracker, if more than one, then more lines
tracker_server=192.168.0.2:22122
url_have_group_name =true   #Add group name before visiting link
base_path=/data/fastdfs
store_path0=/data/fastdfs   #File storage path
 
#Copy part of FastDFS configuration file to / etc/fdfs directory
$ cp /opt/fastdfs/fastdfs-5.11/conf/http.conf /etc/fdfs/
$ cp /opt/fastdfs/fastdfs-5.11/conf/client.conf /etc/fdfs/

Install nginx (select yum install here, exercise to add module s to nginx)

#Install some dependencies, or the subsequent compilation process will always report errors
$ yum install gperftools pcre pcre-devel install openssl openssl-devel zlib-devel libxml2 libxml2-dev libxslt-devel gd-devel perl-devel perl-ExtUtils-Embed GeoIP GeoIP-devel GeoIP-data -y
$ yum install nginx -y
# View version and compilation parameters
$ nginx -v              #Get the corresponding version number, then wget the corresponding version
$ nginx -V              #. / configure

Add nginx module

$ wget http://nginx.org/download/nginx-1.16.1.tar.gz
#Note that the downloaded nginx version is consistent with the yum installation version
$ tar -zxvf nginx-1.16.1.tar.gz
$ cd nginx-1.16.1/
$ ./configure --prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --http-uwsgi-temp-path=/var/lib/nginx/tmp/uwsgi --http-scgi-temp-path=/var/lib/nginx/tmp/scgi --pid-path=/run/nginx.pid --lock-path=/run/lock/subsys/nginx --user=nginx --group=nginx --with-file-aio --with-ipv6 --with-http_auth_request_module --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_addition_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_degradation_module --with-http_slice_module --with-http_stub_status_module --with-http_perl_module=dynamic --with-mail=dynamic --with-mail_ssl_module --with-pcre --with-pcre-jit --with-stream=dynamic --with-stream_ssl_module --with-google_perftools_module --with-debug --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --with-ld-opt='-Wl,-z,relro -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -Wl,-E' --add-module=/opt/fastdfs/fastdfs-nginx-module-1.20/src/
$ make                            #If nginx has been installed before, you only need to compile and do not need to install it
$ cp objs/nginx /usr/sbin/nginx
$ systemctl enable nginx
$ systemctl restart nginx
#To see if the plug-in was installed successfully:
$ nginx -V

Configure nginx access

#to configure nginx.config
$ vim /etc/nginx/nginx.conf In vi /etc/nginx/conf.d/default.conf
#Add the following
server {
        listen 8888; ## The port is storage.conf In http.server_ Same port
        server_name localhost;
        location ~/group[0-9]/ {
            root /data/fastdfs;
            ngx_fastdfs_module;
        }
        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
            root html;
        }
    }
#Test download, use external browser to access the file just passed, and reference the returned ID
http://192.168.0.10:8888/group1/M00/00/00/wKgAQ1pysxmAaqhAAA76tz-dVgg.tar.gz
#For pop-up download, nginx is OK

Errors may be reported in nginx construction

An error will be reported during the make process: / usr/include/fastdfs/fdfs_define.h:15:27: fatal error: common_define.h: No such file or directory
The solution is as follows:

Modify the fastdfs-nginx-module-1.20/src/config file as follows: 
ngx_module_incs="/usr/include/fastdfs /usr/include/fastcommon/" 
CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/" 
Then re. / configure & & make, it's ok

5. Load balancing and high availability of profile access

Install nginx and keepalived on 192.168.0.100 and 101 (in the production environment, you can install nginx load balancing and keepalived on the tracker server machine)

yum install -y nginx keepalived

The configuration file of nginx is as follows:

upstream fdfs_group01 {
    server 192.168.0.10:8888 weight=1 max_fails=2 fail_timeout=30s;
    server 192.168.0.11:8888 weight=1 max_fails=2 fail_timeout=30s;
}
server {
    listen 80;
    server_name localhost;
    location /group01{
        proxy_next_upstream http_502 http_504 error timeout invalid_header;
        proxy_pass http://fdfs_group01;
        expires 30d;
    }
}

The keepalived configuration file is as follows:

global_defs {
   router_id LVS_DEVEL
}
 
vrrp_script chk_ngx {
    script "/etc/keepalived/check_nginx.sh"
    interval 2
    weight -5
    fall 3  
    rise 2
}
 
 
vrrp_instance VI_1 {
    interface eno16777984
    state MASTER
    priority 100
    virtual_router_id 11
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
 
    unicast_src_ip 192.168.0.100
 
    unicast_peer {
        192.168.0.101
    }
 
    virtual_ipaddress {
        192.168.0.200
    }
 
    track_script {
       chk_ngx
    }
    notify_master "/etc/keepalived/notify.sh master"  
    notify_backup "/etc/keepalived/notify.sh backup"  
    notify_fault "/etc/keepalived/notify.sh fault"
}

check_nginx.sh

#!/bin/bash

counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
    exit 1
    else
        exit 0
    fi

In this example, storage does not have multiple groups configured. To modify the configuration storage.conf Group in file_ Name and mod_fastdfs.conf The configuration of the group in the file, and check whether the access to the corresponding group is configured in nginx.

After all our tracker s and storage nodes are started successfully, we can view the information of the storage cluster on any storage node, command: / usr/bin/fdfs_monitor /etc/fdfs/storage.conf , you can see the cluster information.

heck_nginx.sh

#!/bin/bash

counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
    exit 1
    else
        exit 0
    fi

In this example, storage does not have multiple groups configured. To modify the configuration storage.conf Group in file_ Name and mod_fastdfs.conf The configuration of the group in the file, and check whether the access to the corresponding group is configured in nginx.

After all our tracker s and storage nodes are started successfully, we can view the information of the storage cluster on any storage node, command: / usr/bin/fdfs_monitor /etc/fdfs/storage.conf , you can see the cluster information.

Tags: Nginx yum vim github

Posted on Fri, 12 Jun 2020 02:01:27 -0400 by Pha