target
Realize a unified high availability architecture of external download access portal, in which all Nginx are only used for download.
machine | Installation instructions |
---|---|
192.168.89.101 | Libfastcommon,FastDFS |
192.168.89.102 | Libfastcommon,FastDFS |
192.168.89.103 | Libfastcommon,FastDFS,nginx,fastdfs-nginx-module |
192.168.89.104 | Libfastcommon,FastDFS,nginx,fastdfs-nginx-module |
192.168.89.105 | Libfastcommon,FastDFS,nginx,fastdfs-nginx-module |
192.168.89.106 | Libfastcommon,FastDFS,nginx,fastdfs-nginx-module |
Installation deployment
Installation dependency
[root@centos7-81 local]# yum -y install zlib zlib-devel pcre pcre-devel gcc gcc-c++ openssl openssl-devel libevent libevent-devel perl unzip
Because my local virtual machine is newly installed, I need to install it. In fact, I can install what is missing.
Document preparation
file name | explain |
---|---|
V1.0.7 | libfastcommon-1.0.7 (FastDFS all nodes) |
FastDFS_v5.05.tar.gz | (FastDFS all nodes) |
fastdfs-nginx-module_v1.16.tar.gz | Resolve intra group synchronization delays (storage nodes) |
ngx_cache_purge-2.3.tar.gz | Clear the cache for the specified url |
Note: ① the installed versions in this document correspond to the versions in this table. ② The installation files are stored in the / usr/src directory of the host by default.
Installation steps
General file installation
# For easy installation, turn off the firewall first [root@centos7-101 ~]# systemctl stop firewalld # First upload the file to 101 and copy the whole node file (102, 103, 104, 105, 106) [root@centos7-101 src]# scp V1.0.7 192.168.89.102:/usr/src [root@centos7-101 src]# scp FastDFS_v5.05.tar.gz 192.168.89.102:/usr/src
libfastcommon installation (full node installation)
[root@centos7-101 src]# tar -zxvf V1.0.7 -C /usr/local [root@centos7-101 src]# cd /usr/local/libfastcommon-1.0.7 [root@centos7-101 libfastcommon-1.0.7]# ./make.sh [root@centos7-101 libfastcommon-1.0.7]# ./make.sh install
Complete the installation of all nodes.
FastDFS_v5.05 installation (all node installation)
[root@centos7-101 src]# tar -zxvf FastDFS_v5.05.tar.gz -C /usr/local [root@centos7-101 src]# cd /usr/local/FastDFS [root@centos7-101 FastDFS]# ./make.sh [root@centos7-101 FastDFS]# ./make.sh install
Note: after the command is executed, there is no error log and there are files in the / etc/fdfs directory, the installation is successful.
Single tracking node single group storage node section
Tracker configuration
Take node 192.168.89.101 as an example for the following description.
(1) Configure / etc/fdfs/tracker.conf, copy a copy first, and then configure as follows:
base_path=/fastdfs/tracker # Configuration memory store_lookup=0
(2) Create base_path directory
[root@centos7-101 fdfs]# mkdir -p /fastdfs/tracker
(3) Test service
# Start service [root@centos7-101 src]# fdfs_trackerd /etc/fdfs/tracker.conf [root@centos7-101 src]# ps -ef | grep fdfs [root@centos7-101 src]# ss -tpln
Memory configuration
Single group single node configuration
Take node 192.168.89.103 as an example for the following description.
(1) Configure / etc/fdfs/storage.conf, copy a copy (storage.conf.sample), and then configure as follows.
group_name=group1 base_path=/fastdfs/storage store_path0=/fastdfs/storage tracker_server=192.168.89.101:22122
(2) Create base_path directory
[root@centos7-103 fdfs]# mkdir -p /fastdfs/storage
(3) Test [ start service ]
# Start storage service [root@centos7-103 fdfs]# fdfs_storaged /etc/fdfs/storage.conf # View startup log [root@centos7-103 ~]# tail -f /fastdfs/storage/logs/storaged.log [2020-05-20 13:42:09] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=1, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s [2020-05-20 13:42:09] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0 [2020-05-20 13:42:09] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.103, my_server_id_str: 192.168.89.103, g_server_id_in_filename: 1733929152 [2020-05-20 13:42:09] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.103 192.168.89.103 [2020-05-20 13:42:09] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.103 [2020-05-20 13:42:39] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.101:22122, set tracker leader: 192.168.89.101:22122 [2020-05-20 13:42:09] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=1, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s [2020-05-20 13:42:09] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0 [2020-05-20 13:42:09] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.103, my_server_id_str: 192.168.89.103, g_server_id_in_filename: 1733929152 [2020-05-20 13:42:09] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.103 192.168.89.103 [2020-05-20 13:42:09] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.103 [2020-05-20 13:42:39] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.101:22122, set tracker leader: 192.168.89.101:22122
(4) Test [ upload photos ]
# Modify the controller (192.168.89.101) client configuration file [root@centos7-101 fdfs]# vi client.conf # base_path=/fastdfs/tracker # tracker_server=192.168.89.101:22122 # Upload pictures [root@centos7-101 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-1.jpg group1/M00/00/00/wKhZZ17Ew1OAYOtBAAKLWRU1Cxw772.jpg
Single group multi node configuration
Take 192.168.89.103 and 192.168.89.105 as examples.
192.168.89.105 the node configuration can be the same as 103. After configuring the storage.conf file, perform the following operations.
(1) Start service
# Start storage service [root@centos7-103 fdfs]# fdfs_storaged /etc/fdfs/storage.conf # View startup log [root@centos7-103 ~]# tail -f /fastdfs/storage/logs/storaged.log [2020-05-20 16:05:27] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=1, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s [2020-05-20 16:05:27] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0 [2020-05-20 16:05:27] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.105, my_server_id_str: 192.168.89.105, g_server_id_in_filename: 1767483584 [2020-05-20 16:05:27] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.105 192.168.89.105 [2020-05-20 16:05:27] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.105 [2020-05-20 16:05:27] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.101:22122, set tracker leader: 192.168.89.101:22122 [2020-05-20 16:05:27] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.103:23000
After the service is started, the pictures on 103 will be automatically synchronized. You can view the log file as follows (time interval):
[2020-05-20 16:12:04] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.105:23000
(2) Testing
# Upload pictures [root@centos7-101 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg group1/M00/00/00/wKhZZ17E5S2AMZf2AAKMU4U081o805.jpg
Summary (t1-s1-s2)
192.168.89.101 [tracker-1], 192.168.89.103 [group1-1] and 192.168.89.105 [group1-2] completed the construction of a single group of multiple nodes.
Upload test:
[root@centos7-101 fdfs]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-2.jpg group1/M00/00/00/wKhZZ17EqGCAL9j8AAKZ_83sbhk602.jpg
Log analysis:
# 105 log [root@centos7-105 logs]# tail -f storaged.log mkdir data path: FD ... mkdir data path: FE ... mkdir data path: FF ... data path: /fastdfs/storage/data, mkdir sub dir done. [2020-05-20 11:38:53] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0 [2020-05-20 11:38:53] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.105, my_server_id_str: 192.168.89.105, g_server_id_in_filename: 1767483584 [2020-05-20 11:38:53] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.105 192.168.89.105 [2020-05-20 11:38:53] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.105 [2020-05-20 11:38:53] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.101:22122, set tracker leader: 192.168.89.101:22122 [2020-05-20 11:38:53] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.103:23000 # 103 log [root@centos7-103 logs]# tail -f storaged.log mkdir data path: FE ... mkdir data path: FF ... data path: /fastdfs/storage/data, mkdir sub dir done. [2020-05-20 11:38:42] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0 [2020-05-20 11:38:42] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.103, my_server_id_str: 192.168.89.103, g_server_id_in_filename: 1733929152 [2020-05-20 11:38:42] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.103 192.168.89.103 [2020-05-20 11:38:42] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.103 [2020-05-20 11:39:12] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.101:22122, set tracker leader: 192.168.89.101:22122 [2020-05-20 11:39:12] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.105:23000 [2020-05-20 11:39:42] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.105:23000
Multi tracking node multi group storage node section
Tracker configuration
Take 192.168.89.101 and 192.168.89.102 as examples. The installation is the same as "single tracking node single group storage node tracker configuration section".
Memory configuration
- [192.168.89.103] and [192.168.89.105]
Both nodes belong to group1. Modify the configuration of / etc/fdfs/storage.conf as follows:
group_name=group1 base_path=/fastdfs/storage store_path0=/fastdfs/storage tracker_server=192.168.89.101:22122 tracker_server=192.168.89.102:22122
- [192.168.89.104] and [192.168.89.106]
Both nodes belong to group2. Modify the configuration of / etc/fdfs/storage.conf as follows:
group_name=group2 base_path=/fastdfs/storage store_path0=/fastdfs/storage tracker_server=192.168.89.101:22122 tracker_server=192.168.89.102:22122
test
Upload a picture on the 192.168.89.102 (T2) tracker to see the polling effect of the tracker, as shown below:
[root@centos7-102 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg group1/M00/00/00/wKhZZ17E9TmADDyNAAKMU4U081o921.jpg [root@centos7-102 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg group2/M00/00/00/wKhZal7E9UKAeIPjAAKMU4U081o129.jpg [root@centos7-102 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg group1/M00/00/00/wKhZaV7E9UiARRYYAAKMU4U081o553.jpg [root@centos7-102 src]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg group2/M00/00/00/wKhZaF7E9UqAM212AAKMU4U081o334.jpg
Installation summary
The above part has completed the installation of FastDFS and uploaded files. In order to achieve high availability of image display, we also need to configure ngx_fastdfs_module configuration. The plug-in is described in the following chapters.
nginx deployment and installation
nginx function: storage nodes (103, 104, 105, 106) act as static resource agents; Non storage nodes (107, 108) act as reverse agents.
Fastdfs nginx module installation
All storage nodes need to install this plug-in. In this case, nodes 103, 104, 105 and 106 need to be installed.
(1) Upload and unzip fastdfs nginx module_ v1.16.tar.gz
[root@centos7-103 src]# tar -zxvf fastdfs-nginx-module_v1.16.tar.gz -C /usr/local/
(2) Modify config file
[root@centos7-103 src]# cd /usr/local/fastdfs-nginx-module/src [root@centos7-103 src]# vi config # Remove local from the path CORE_INCS="$CORE_INCS /usr/local/include/fastdfs /usr/local/include/fastcommon/"
(3) Configure mod_fastdfs.conf file
[root@centos7-103 src]# cp /usr/local/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs [root@centos7-103 src]# vi /etc/fdfs/mod_fastdfs.conf base_path=/fastdfs/storage tracker_server=192.168.89.101:22122 # Single tracking node tracker_server=192.168.89.102:22122 # Multiple tracking nodes group_name=group1 # The name of the group storage node is configured url_have_group_name = true store_path0=/fastdfs/storage # The actual storage location of the current node group_count = 2 # 0 indicates a single group. More than one group is configured according to the actual sublease quantity [group1] # The following configuration can only be performed for multiple groups group_name=group1 storage_server_port=23000 store_path_count=1 store_path0=/fastdfs/storage [group2] group_name=group2 storage_server_port=23000 store_path_count=1 store_path0=/fastdfs/storage
(4) Copy http.conf, mime.types
[root@centos7-103 src]# cd /usr/local/FastDFS/conf [root@centos7-103 conf]# cp http.conf mime.types /etc/fdfs
(5) Configure soft connection
[root@centos7-103 src]# ln -s /fastdfs/storage/data /fastdfs/storage/data/M00
At this point, the Nginx expansion module has been installed.
nginx installation
(1) Installation
[root@centos7-103 src]# tar -zxvf nginx-1.16.1.tar.gz -C /usr/local [root@centos7-103 src]# cd /usr/local/nginx-1.16.1/ [root@centos7-103 nginx-1.16.1]# ./configure --prefix=/usr/local/nginx --add-module=/usr/local/fastdfs-nginx-module/src [root@centos7-103 nginx-1.16.1]# make && make install
(2) Start service
[root@centos7-103 nginx-1.16.1]# /usr/local/nginx/sbin/nginx
(3) Browser test, enter address: http://192.168.89.103/
103 machine command line input:[ root@centos7 -102 src]# curl http://192.168.89.3
The words "Welcome to nginx" are displayed, indicating that the service is started successfully.
nginx.conf configuration
location /group1/M00 { ngx_fastdfs_module; }
Test in browser: http://192.168.89.103/group1/M00/00/00/wKhZZ17Ew1OAYOtBAAKLWRU1Cxw772.jpg
Multi storage node installation configuration
(1)mod_fastdfs.conf file [103, 104, 105, 106]
base_path=/fastdfs/storage tracker_server=192.168.89.101:22122 tracker_server=192.168.89.102:22122 group_name=group1 # 103 105 is group1104 and 106 is group2 url_have_group_name = true store_path0=/fastdfs/storage # The actual storage location of the current node group_count = 2 # There are 2 groups in this case [group1] # Multi group configuration group_name=group1 storage_server_port=23000 store_path_count=1 store_path0=/fastdfs/storage [group2] group_name=group2 storage_server_port=23000 store_path_count=1 store_path0=/fastdfs/storage
(2) nginx.conf configuration
location ~/group([1-2])/M00 { ngx_fastdfs_module; }
Others can be installed according to the single node (the above three sections).
Load test
(1) Start service
[root@centos7-101 src]# fdfs_trackerd /etc/fdfs/tracker.conf [root@centos7-102 src]# fdfs_trackerd /etc/fdfs/tracker.conf [root@centos7-103 src]# fdfs_storaged /etc/fdfs/storage.conf [root@centos7-104 src]# fdfs_storaged /etc/fdfs/storage.conf [root@centos7-105 src]# fdfs_storaged /etc/fdfs/storage.conf [root@centos7-106 src]# fdfs_storaged /etc/fdfs/storage.conf
(2) Log analysis
[root@centos7-101 src]# tail -f /fastdfs/tracker/logs/trackerd.log [2020-05-22 10:07:42] INFO - FastDFS v5.05, base_path=/fastdfs/tracker, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=22122, bind_addr=, max_connections=256, accept_threads=1, work_threads=4, store_lookup=0, store_group=, store_server=0, store_path=0, reserved_storage_space=10.00%, download_server=0, allow_ip_count=-1, sync_log_buff_interval=10s, check_active_interval=120s, thread_stack_size=64 KB, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=300s, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, use_storage_id=0, id_type_in_filename=ip, storage_id_count=0, rotate_error_log=0, error_log_rotate_time=00:00, rotate_error_log_size=0, log_file_keep_days=0, store_slave_file_use_link=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s [2020-05-22 10:07:42] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.101 192.168.89.101 [2020-05-22 10:08:08] INFO - file: tracker_relationship.c, line: 383, selecting leader... [2020-05-22 10:08:09] INFO - file: tracker_service.c, line: 969, the tracker leader is 192.168.89.102:22122 [root@centos7-102 src]# tail -f /fastdfs/tracker/logs/trackerd.log [2020-05-22 10:07:51] INFO - FastDFS v5.05, base_path=/fastdfs/tracker, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=22122, bind_addr=, max_connections=256, accept_threads=1, work_threads=4, store_lookup=0, store_group=, store_server=0, store_path=0, reserved_storage_space=10.00%, download_server=0, allow_ip_count=-1, sync_log_buff_interval=10s, check_active_interval=120s, thread_stack_size=64 KB, storage_ip_changed_auto_adjust=1, storage_sync_file_max_delay=86400s, storage_sync_file_max_time=300s, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, use_storage_id=0, id_type_in_filename=ip, storage_id_count=0, rotate_error_log=0, error_log_rotate_time=00:00, rotate_error_log_size=0, log_file_keep_days=0, store_slave_file_use_link=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s [2020-05-22 10:07:51] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.102 192.168.89.102 [2020-05-22 10:08:10] INFO - file: tracker_relationship.c, line: 383, selecting leader... [2020-05-22 10:08:10] INFO - file: tracker_relationship.c, line: 401, I am the new tracker leader 192.168.89.102:22122 [root@centos7-103 src]# tail -f /fastdfs/storage/logs/storaged.log [2020-05-22 10:08:33] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s [2020-05-22 10:08:33] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0 [2020-05-22 10:08:33] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.103, my_server_id_str: 192.168.89.103, g_server_id_in_filename: 1733929152 [2020-05-22 10:08:33] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.103 192.168.89.103 [2020-05-22 10:08:33] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.102:22122, as a tracker client, my ip is 192.168.89.103 [2020-05-22 10:08:33] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.103 [2020-05-22 10:08:33] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.102:22122, set tracker leader: 192.168.89.102:22122 [2020-05-22 10:09:04] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.105:23000 [root@centos7-104 nginx-1.16.1]# tail -f /fastdfs/storage/logs/storaged.log [2020-05-22 10:08:09] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group2, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s [2020-05-22 10:08:09] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0 [2020-05-22 10:08:09] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.104, my_server_id_str: 192.168.89.104, g_server_id_in_filename: 1750706368 [2020-05-22 10:08:09] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.104 192.168.89.104 [2020-05-22 10:08:09] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.102:22122, as a tracker client, my ip is 192.168.89.104 [2020-05-22 10:08:09] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.104 [2020-05-22 10:08:39] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.102:22122, set tracker leader: 192.168.89.102:22122 [2020-05-22 10:09:10] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.106:23000 [root@centos7-105 src]# tail -f /fastdfs/storage/logs/storaged.log [2020-05-22 10:08:35] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group1, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s [2020-05-22 10:08:35] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0 [2020-05-22 10:08:35] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.105, my_server_id_str: 192.168.89.105, g_server_id_in_filename: 1767483584 [2020-05-22 10:08:36] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.105 192.168.89.105 [2020-05-22 10:08:36] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.105 [2020-05-22 10:08:36] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.102:22122, as a tracker client, my ip is 192.168.89.105 [2020-05-22 10:08:36] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.102:22122, set tracker leader: 192.168.89.102:22122 [2020-05-22 10:08:36] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.103:23000 [root@centos7-106 nginx-1.16.1]# tail -f /fastdfs/storage/logs/storaged.log [2020-05-22 10:08:40] INFO - FastDFS v5.05, base_path=/fastdfs/storage, store_path_count=1, subdir_count_per_path=256, group_name=group2, run_by_group=, run_by_user=, connect_timeout=30s, network_timeout=60s, port=23000, bind_addr=, client_bind=1, max_connections=256, accept_threads=1, work_threads=4, disk_rw_separated=1, disk_reader_threads=1, disk_writer_threads=1, buff_size=256KB, heart_beat_interval=30s, stat_report_interval=60s, tracker_server_count=2, sync_wait_msec=50ms, sync_interval=0ms, sync_start_time=00:00, sync_end_time=23:59, write_mark_file_freq=500, allow_ip_count=-1, file_distribute_path_mode=0, file_distribute_rotate_count=100, fsync_after_written_bytes=0, sync_log_buff_interval=10s, sync_binlog_buff_interval=10s, sync_stat_file_interval=300s, thread_stack_size=512 KB, upload_priority=10, if_alias_prefix=, check_file_duplicate=0, file_signature_method=hash, FDHT group count=0, FDHT server count=0, FDHT key_namespace=, FDHT keep_alive=0, HTTP server port=8888, domain name=, use_access_log=0, rotate_access_log=0, access_log_rotate_time=00:00, rotate_error_log=0, error_log_rotate_time=00:00, rotate_access_log_size=0, rotate_error_log_size=0, log_file_keep_days=0, file_sync_skip_invalid_record=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s [2020-05-22 10:08:40] INFO - file: storage_param_getter.c, line: 191, use_storage_id=0, id_type_in_filename=ip, storage_ip_changed_auto_adjust=1, store_path=0, reserved_storage_space=10.00%, use_trunk_file=0, slot_min_size=256, slot_max_size=16 MB, trunk_file_size=64 MB, trunk_create_file_advance=0, trunk_create_file_time_base=02:00, trunk_create_file_interval=86400, trunk_create_file_space_threshold=20 GB, trunk_init_check_occupying=0, trunk_init_reload_from_binlog=0, trunk_compress_binlog_min_interval=0, store_slave_file_use_link=0 [2020-05-22 10:08:40] INFO - file: storage_func.c, line: 254, tracker_client_ip: 192.168.89.106, my_server_id_str: 192.168.89.106, g_server_id_in_filename: 1784260800 [2020-05-22 10:08:40] INFO - local_host_ip_count: 3, 127.0.0.1 10.0.2.106 192.168.89.106 [2020-05-22 10:08:40] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.102:22122, as a tracker client, my ip is 192.168.89.106 [2020-05-22 10:08:40] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 192.168.89.101:22122, as a tracker client, my ip is 192.168.89.106 [2020-05-22 10:08:40] INFO - file: tracker_client_thread.c, line: 1235, tracker server 192.168.89.102:22122, set tracker leader: 192.168.89.102:22122 [2020-05-22 10:08:40] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server 192.168.89.104:23000
(3) Upload test
[root@centos7-101 fdfs]# vi client.conf [root@centos7-101 fdfs]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-2.jpg group1/M00/00/00/wKhZZ17HNmSAFufHAAKZ_83sbhk311.jpg [root@centos7-101 fdfs]# fdfs_upload_file /etc/fdfs/client.conf /usr/src/t-3.jpg group2/M00/00/00/wKhZaF7HNmqABRnyAAKMU4U081o247.jpg
(4) Access test
http://192.168.89.103:8000/group1/M00/00/00/wKhZZ17HNmSAFufHAAKZ_83sbhk311.jpg
http://192.168.89.104:8000/group1/M00/00/00/wKhZZ17HNmSAFufHAAKZ_83sbhk311.jpg
According to the truth, 104 should not be accessible, because group1 is stored on 103 and 105, but the actual results show that there is no problem. After testing: if the four storage nodes are normal, use 103 and 105 to access the storage of group2 and redirect to 104; Use 104 and 106 to access the storage of group1 and redirect to 103. The log is as follows:
[root@centos7-104 ~]# tail -f /usr/local/nginx/logs/error.log 2020/05/22 10:21:29 [error] 5189#0: *1 upstream timed out (110: Connection timed out) while reading upstream, client: 192.168.89.1, server: localhost, request: "GET /group1/M00/00/00/wKhZZ17HNmSAFufHAAKZ_83sbhk311.jpg HTTP/1.1", upstream: "http://192.168.89.103:8000/group1/M00/00/00/wKhZZ17HNmSAFufHAAKZ_83sbhk311.jpg?redirect=1", host: "192.168.89.104:8000"
As can be seen from the log, the service redirected to 103.
Reverse proxy configuration
We use 192.168.89.107 and 192.168.89.108 nodes for reverse proxy configuration.
(1) Installing nginx
[root@centos7-107 src]# tar -zxvf /usr/src/nginx-1.16.1.tar.gz -C /usr/local [root@centos7-107 src]# cd /usr/local/nginx-1.16.1 [root@centos7-107 src]# ./configure --prefix=/usr/local/nginx [root@centos7-107 src]# cd /usr/local/nginx-1.16.1
(2) / usr/local/nginx/conf/nginx.conf configuration
upstream fdfs_group1 { server 192.168.89.103:8000 weight=1 max_fails=2 fail_timeout=30s; server 192.168.89.105:8000 weight=1 max_fails=2 fail_timeout=30s; } upstream fdfs_group2 { server 192.168.89.104:8000 weight=1 max_fails=2 fail_timeout=30s; server 192.168.89.106:8000 weight=1 max_fails=2 fail_timeout=30s; } location /group1/M00 { proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_pass http://fdfs_group1; expires 30d; } location /group2/M00 { proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_pass http://fdfs_group2; expires 30d; }
(3) Testing
http://192.168.89.107:8081/group2/M00/00/00/wKhZaF7HNmqABRnyAAKMU4U081o247.jpg
http://192.168.89.108:8081/group2/M00/00/00/wKhZaF7HNmqABRnyAAKMU4U081o247.jpg
nginx.conf reference configuration:
server_names_hash_bucket_size 128; client_header_buffer_size 32k; large_client_header_buffers 4 32k; client_max_body_size 300m; proxy_redirect off; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffer_size 16k; proxy_buffers 4 64k; proxy_busy_buffers_size 128k; proxy_temp_file_write_size 128k; proxy_cache_path /fastdfs/cache/nginx/proxy_cache levels=1:2 keys_zone=http-cache:200m max_size=1g inactive=30d; proxy_temp_path /fastdfs/cache/nginx/proxy_cache/tmp; upstream fdfs_group1 { server 192.168.89.106:8888 weight=1 max_fails=2 fail_timeout=30s; server 192.168.89.108:8888 weight=1 max_fails=2 fail_timeout=30s; } upstream fdfs_group2 { server 192.168.89.105:8888 weight=1 max_fails=2 fail_timeout=30s; server 192.168.89.107:8888 weight=1 max_fails=2 fail_timeout=30s; } location /group1/M00 { proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache http-cache; proxy_cache_valid 200 304 12h; proxy_cache_key $uri$is_args$args; proxy_pass http://fdfs_group1; expires 30d; } location /group2/M00 { #proxy_next_upstream http_502 http_504 error timeout invalid_header; #proxy_cache http-cache; #proxy_cache_valid 200 304 12h; #proxy_cache_key $uri$is_args$args; proxy_pass http://fdfs_group2; expires 30d; } #Clear cache access location ~/purge(/.*) { allow 127.0.0.1; allow 192.168.89.0/24; deny all; proxy_cache_purge http-cache $1$is_args$args; }
Cache settings
Command:
[root@centos7-81 src]# tar -zxvf ngx_cache_purge-2.3.tar.gz -C /usr/local [root@centos7-83 nginx-1.16.1]# ./configure --prefix=/usr/local/nginx --add-module=/usr/local/ngx_cache_purge-2.3 [root@centos7-83 nginx-1.16.1]# make && make install mkdir –p /fastdfs/cache/nginx/proxy_cache mkdir –p /fastdfs/cache/nginx/proxy_cache/tmp
Test:
http://192.168.89.3:8000/group2/M00/00/00/wKhZBl7CNgaAQuHFAAKZ_83sbhk190.jpg
http://192.168.89.4:8000/group2/M00/00/00/wKhZBl7CNgaAQuHFAAKZ_83sbhk190.jpg
keepalived
We use 192.168.89.109 and 192.168.89.110 nodes for keepalived configuration.
(1) Installing nginx
(2) Install keepalived
[root@centos7-109 src]# tar -zxvf keepalived-2.0.20.tar.gz [root@centos7-109 src]# cd keepalived-2.0.20 [root@centos7-109 keepalived-2.0.20]# ./configure --prefix=/usr/local/keepalived [root@centos7-109 keepalived-2.0.20]# make && make install
After completion, the generated paths and files are as follows:
/usr/local/keepalived/etc/keepalived/keepalived.conf
/usr/local/keepalived/etc/sysconfig/keepalived
/usr/local/keepalived/sbin/keepalived
(3) Initialization and startup
# The kept startup script variable references the file. The default path is / etc/sysconfig [root@centos7-109 src]# cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig # Add the keepalived main program to the environment variable [root@centos7-109 sbin]# cp /usr/local/keepalived/sbin/keepalived /usr/sbin [root@centos7-109 sbin]# cp /usr/src/keepalived-2.0.20/keepalived/etc/init.d/keepalived /etc/init.d/ # Put the configuration file in the default path [root@centos7-109 etc]# mkdir /etc/keepalived [root@centos7-109 etc]# cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
(4) 192.168.89.109 node configuration
# vi /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_109 } vrrp_script chk_nginx { script "/etc/keepalived/check_list" interval 2 weight -20 } vrrp_instance VI_1 { state MASTER interface enp0s8 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } # Track_ Add script block to instance configuration quickly track_script { chk_nginx # Perform Nginx monitoring service } virtual_ipaddress { 192.168.89.100 } }
(5) 192.168.89.110 node configuration
# vi /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { router_id LVS_110 } vrrp_script chk_nginx { script "/etc/keepalived/check_list" interval 2 weight -20 } vrrp_instance VI_1 { state BACKUP interface enp0s8 virtual_router_id 51 priority 90 advert_int 1 authentication { auth_type PASS auth_pass 1111 } # Track_ Add script block to instance configuration quickly track_script { chk_nginx # Perform Nginx monitoring service } virtual_ipaddress { 192.168.89.100 } }
(6)check_list script
The script used to detect the survival status of nginx. If nginx does not survive, turn off and shut down the keepalived on the nginx host.
In the keepalived+nginx architecture, after nginx goes down, user requests will fail, but keepalived will not switch.
#!/bin/sh nginxpid=$(ps -C nginx --no-header|wc -l) #1. Judge whether Nginx is alive. If not, try to start Nginx if [ $nginxpid -eq 0 ];then systemctl start nginx sleep 3 #2. Wait for 3 seconds and get the Nginx status again nginxpid=$(ps -C nginx --no-header|wc -l) #3. Judge again. If Nginx does not survive, stop Keepalived, drift the address, and exit the script if [ $nginxpid -eq 0 ];then systemctl stop keepalived fi fi
(7) Testing
Enter in the browser: http://192.168.89.100/
Display the page of node 109, and then stop the nginx and keepalived services of node 109. You can see that the address drifts during access, and display the page of node 110.
nginx+keepalived+fastdfs
In the chapter "kept", we configured the high availability of nginx service. In this chapter, we added the agent of fastdfs. Use the 109110 environment for configuration instructions.
(1) nginx configuration [192.168.89.109 and 192.168.89.110]
# vi /usr/local/nginx/conf/nginx.conf upstream fdfs_group1 { # Add upstream server server 192.168.89.103:8000 weight=1 max_fails=2 fail_timeout=30s; server 192.168.89.105:8000 weight=1 max_fails=2 fail_timeout=30s; } upstream fdfs_group2 { # Add upstream server server 192.168.89.104:8000 weight=1 max_fails=2 fail_timeout=30s; server 192.168.89.106:8000 weight=1 max_fails=2 fail_timeout=30s; } server { listen 80; server_name localhost; ............ location /group1/M00 { # Reverse proxy proxy_pass http://fdfs_group1; expires 30d; } location /group2/M00 { proxy_pass http://fdfs_group2; expires 30d; } ............
First modify the configuration of 109, and then synchronize to 110[ root@centos7-109 conf]# scp nginx.conf 192.168.89.110:/usr/local/nginx/conf.
(2) Start service
101, 102 start the fastdfs tracker service.
103, 104, 105 and 106 start fastdfs storage and nginx services.
109 and 110 start the keepalived and nginx services.
# 101 102 tracker service startup [root@centos7-101 ~]# fdfs_trackerd /etc/fdfs/tracker.conf [root@centos7-102 ~]# fdfs_trackerd /etc/fdfs/tracker.conf # 103 104 105 105 storage and nginx services start [root@centos7-103 ~]# fdfs_storaged /etc/fdfs/storage.conf [root@centos7-103 ~]# /usr/local/nginx/sbin/nginx ...... # 109 110 keepalived, nginx service started
(3) Testing
After the service starts normally, we test it.
First, upload a picture on the 101 or 102 server, as shown below:
# 101 upload pictures [root@centos7-101 src]# fdfs_upload_file /etc/fdfs/client.conf t-1.jpg group1/M00/00/00/wKhZZ17OEOCAMRH1AAKLWRU1Cxw036.jpg [root@centos7-101 src]# fdfs_upload_file /etc/fdfs/client.conf t-2.jpg group2/M00/00/00/wKhZaF7OETOAcwfhAAKZ_83sbhk275.jpg
Use the browser to access the following address:
http://192.168.89.100/group1/M00/00/00/wKhZZ17OEOCAMRH1AAKLWRU1Cxw036.jpg
http://192.168.89.100/group2/M00/00/00/wKhZaF7OETOAcwfhAAKZ_83sbhk275.jpg
web project integration
(1) Dependent on jar: fastdfs_client.jar
(2) Initialize the client core code and write it to the static block or constructor:
# class root file directory String classPath = new File(ImgUploadService.class.getResource("/").getFile()).getCanonicalPath(); // According to fdfs_client.conf ClientGlobal.init(classPath + File.separator + configFile); trackerClient = new TrackerClient(); trackerServer = trackerClient.getConnection(); storageServer = null; storageClient = new StorageClient(trackerServer, storageServer); log.info("init: " + storageClient.hashCode()); // Send a message to dfs, otherwise there will be abnormal information in the first transmission ProtoCommon.activeTest(trackerServer.getSocket());
(3) Upload core code
TrackerServer trackerServer = trackerClient.getConnection(); StorageClient storageClient = new StorageClient(trackerServer, storageServer); // The first element in the results array is the storage group name, and the second element is the image storage path. String[] results = storageClient.upload_file(fileContent, suffix, null);
(4) Download core code
public byte[] getFileByteBuffer(Efilesummary object) throws Exception { String classPath = new File(UploadService.class.getResource("/").getFile()).getCanonicalPath(); ClientGlobal.init(classPath + File.separator + configFile); TrackerClient trackerClient = new TrackerClient(); TrackerServer trackerServer = trackerClient.getConnection(); StorageServer storageServer = null; StorageClient storageClient = new StorageClient(trackerServer, storageServer); return storageClient.download_file(object.getPath(), object.getRemotefsid()); }
(5) Delete core code
public void deleteFile(Efilesummary object) throws Exception { String classPath = new File(UploadService.class.getResource("/").getFile()).getCanonicalPath(); ClientGlobal.init(classPath + File.separator + configFile); TrackerClient trackerClient = new TrackerClient(); TrackerServer trackerServer = trackerClient.getConnection(); StorageServer storageServer = null; StorageClient storageClient = new StorageClient(trackerServer, storageServer); int i = storageClient.delete_file(object.getPath(), object.getRemotefsid()); System.out.println(i); }
(6)demo
fastdfs-demo.rar
FastDFS data
Basic module
Tracker: tracking controller. The IP address of the server will be used in the program.
Storage: storage. Groups represent capacity expansion relationship, and blocks within groups represent backup relationship.
High availability
reference material
https://blog.51cto.com/7072753/2286604
https://blog.csdn.net/wc1695040842/article/details/89766064
Delete route:
ip route del default via 192.168.89.1 this command is added to ~ /. bashrc
- Kept data:
https://blog.csdn.net/l1028386804/article/details/72801492
https://www.jianshu.com/p/a6b5ab36292a
Log analysis
Storage monitoring
[root@centos7-104 fdfs]# fdfs_monitor /etc/fdfs/storage.conf [2020-05-20 17:10:26] DEBUG - base_path=/fastdfs/storage, connect_timeout=30, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0 server_count=2, server_index=1 tracker server is 192.168.89.102:22122 group count: 2 Group 1: group name = group1 disk total space = 8662 MB disk free space = 7544 MB trunk free space = 0 MB storage server count = 2 active server count = 2 storage server port = 23000 storage HTTP port = 8888 store path count = 1 subdir count per path = 256 current write server index = 0 current trunk file id = 0 Storage 1: id = 192.168.89.103 ip_addr = 192.168.89.103 ACTIVE http domain = version = 5.05 join time = 2020-05-20 11:38:40 up time = 2020-05-20 17:05:32 total storage = 8662 MB free storage = 7544 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 4 success_upload_count = 4 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 670980 success_upload_bytes = 670980 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 4 success_file_open_count = 4 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 4 success_file_write_count = 4 last_heart_beat_time = 2020-05-20 17:10:01 last_source_update = 2020-05-20 16:11:23 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00 Storage 2: id = 192.168.89.105 ip_addr = 192.168.89.105 ACTIVE http domain = version = 5.05 join time = 2020-05-20 11:38:50 up time = 2020-05-20 17:06:32 total storage = 8662 MB free storage = 7544 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = 192.168.89.103 if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 670980 success_sync_in_bytes = 670980 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 4 success_file_open_count = 4 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 4 success_file_write_count = 4 last_heart_beat_time = 2020-05-20 17:10:03 last_source_update = 1970-01-01 08:00:00 last_sync_update = 2020-05-20 16:12:03 last_synced_timestamp = 1970-01-01 08:00:00 (never synced) Group 2: group name = group2 disk total space = 8662 MB disk free space = 7545 MB trunk free space = 0 MB storage server count = 2 active server count = 2 storage server port = 23000 storage HTTP port = 8888 store path count = 1 subdir count per path = 256 current write server index = 1 current trunk file id = 0 Storage 1: id = 192.168.89.104 ip_addr = 192.168.89.104 ACTIVE http domain = version = 5.05 join time = 2020-05-20 16:43:25 up time = 2020-05-20 17:05:54 total storage = 8662 MB free storage = 7545 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 1 success_upload_count = 1 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 166995 success_upload_bytes = 166995 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 1 success_file_open_count = 1 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 1 success_file_write_count = 1 last_heart_beat_time = 2020-05-20 17:10:23 last_source_update = 2020-05-20 16:47:08 last_sync_update = 1970-01-01 08:00:00 last_synced_timestamp = 1970-01-01 08:00:00 Storage 2: id = 192.168.89.106 ip_addr = 192.168.89.106 ACTIVE http domain = version = 5.05 join time = 2020-05-20 16:43:33 up time = 2020-05-20 17:07:02 total storage = 8662 MB free storage = 7545 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = 192.168.89.104 if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 1 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 166995 success_sync_in_bytes = 166995 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 1 success_file_open_count = 1 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 1 success_file_write_count = 1 last_heart_beat_time = 2020-05-20 17:10:01 last_source_update = 1970-01-01 08:00:00 last_sync_update = 2020-05-20 16:47:17 last_synced_timestamp = 2020-05-20 16:47:08 (0s delay)