day42 Ceph of distributed storage

Task background

Although the distributed glusterfs storage is used, it is still unable to cope with the explosive data growth. Storage also needs to keep pace with the maturity of technologies such as big data and cloud computing, so this time we choose object storage

Task requirements

1. Build ceph cluster

2. Realize the application of object storage

Task Disassembly

1. Understand ceph

2. Build ceph cluster

3. Understand rados native data access

4. Implement ceph file storage

5. Implement ceph block storage

6. Implement ceph object storage

Learning objectives

  • The ceph cluster can be successfully deployed
  • Be able to use ceph to share file storage, block storage and object storage
  • Be able to describe the characteristics of object storage

1, Meet Ceph

Ceph is a distributed storage system that can provide file storage, block storage and object storage. It provides an infinitely scalable Ceph storage cluster.

2, ceph architecture

Reference official file:

RADOS: Ceph's high reliability, high scalability, high performance and high automation are provided by this layer, and the storage of user data is finally stored through this layer.

It can be said that RADOS is the underlying native data engine of ceph, but it is not directly used in practical application, but is used in the following four ways:

  • LIBRADOS is a library that allows applications to interact with the RADOS system by accessing the library and supports multiple programming languages. Such as Python,C,C + +, etc. in short, it is an interface for developers.
  • CEPH FS provides file systems through Linux kernel client and FUSE. (file storage)
  • RBD provides a distributed block device through Linux kernel client and QEMU/KVM driver. (block storage)
  • RADOSGW is a gateway based on the popular RESTFUL protocol, and is compatible with S3 and Swift. (object storage)

Extended NOUN

RESTFUL: RESTFUL is an architecture style that provides a set of design principles and constraints. http is a typical application of this style. The biggest features of REST are: resources, unified interface, URI and statelessness.

  • Resources: a specific information on the network: a file, a picture and a video are all resources.
  • Unified interface: meta operations of data, namely CRUD(create, read, update and delete) operations, correspond to HTTP methods respectively
    • GET (SELECT): fetch resources (one or more items) from the server.
    • POST (CREATE): CREATE a new resource on the server.
    • PUT (UPDATE): UPDATE resources on the server (the client provides complete resource data).
    • PATCH (UPDATE): UPDATE the resource on the server (the client provides the resource data to be modified).
    • DELETE (DELETE): deletes resources from the server.
  • Uri (uniform resource locator): each URI corresponds to a specific resource. To get this resource, just access its URI. The most typical URI is URL
  • Stateless: the location of a resource has nothing to do with other resources and is not affected by other resources.

S3 (Simple Storage Service): S3 can be regarded as a super large hard disk, which stores data resources (files, pictures, videos, etc.), which are collectively referred to as objects. These objects are stored in the storage segment, which is called bucket in S3

Compared with the hard disk, a bucket is equivalent to a directory and an object is equivalent to a file.

The hard disk path is similar to / root/file1.txt

The URI of S3 is similar to s3://bucket_name/object_name

swift: originally developed by Rackspace, it was a highly available distributed object storage service and contributed to the OpenStack open source community as one of its initial core sub projects in 2010

3, Ceph cluster

Cluster components

Ceph cluster includes Ceph OSD and Ceph monitor daemons.

Ceph OSD(Object Storage Device): the function is to store data, handle data replication, recovery, backfilling and rebalancing, and provide some monitoring information to Ceph Monitors by checking the heartbeat of other OSD daemons.
Ceph Monitor: it is a monitor that monitors Ceph cluster status and maintains various relationships in the cluster.

Ceph storage cluster requires at least one Ceph Monitor and two OSD daemons.

Cluster environment preparation


Prepare four servers, which need to be able to access the external network and have static fixed IP (at least one disk is added for each server except the client, with a minimum of 1G and no partition);

1. Configure host name and host name binding (all nodes must be bound)

(Note: all of them are changed to short host names for later experiments. If you insist on using a host name like or adding an alias, ceph will intercept your host name as Vm1 later, resulting in inconsistency and errors.)

# hostnamectl set-hostname --static node1
# vim /etc/hosts	node1	node2	node3	client

2. Close the firewall and SELinux (use iptables -F to clear the rules)

# systemctl stop firewalld
# systemctl disable firewalld

# iptables -F

# setenforce 0

3. Time synchronization (start ntpd service and confirm that all nodes have the same time)

# systemctl restart ntpd
# systemctl enable ntpd

4. Configure yum source (all nodes, including client)

There are 2 yum source methods for ceph:

  • Public network ceph source (centos7 default public network source + epel source + aliyun source of ceph)
# yum install epel-release -y
# vim /etc/yum.repos.d/ceph.repo


name=Ceph source packages
  • Local ceph source (centos7 default public network source + ceph local source)
    • The download network of public network source is slow, and the update of public network source may cause problems. You can use the downloaded as the local ceph source

CEPH to be shared_ Copy the soft directory to all nodes (for example, under: / root /)

# vim /etc/yum.repos.d/ceph.repo

Cluster deployment process

Step 1: configure ssh password free

Take node1 as the deployment configuration node and configure ssh equivalence on node1 (ssh node1, node2, node3 and client are required to be password free)

Note: this step is not necessary. The purpose of this step is:

  • If you use CEPH deploy to install the cluster, the key will be easy to install
  • If you do not use CEPH deploy to install, you can also facilitate the following operations: for example, synchronizing configuration files
[root@node1 ~]# ssh-keygen 
[root@node1 ~]# ssh-copy-id -i node1
[root@node1 ~]# ssh-copy-id -i node2
[root@node1 ~]# ssh-copy-id -i node3
[root@node1 ~]# ssh-copy-id -i client

Step 2: install the deployment tool on node1

(other nodes do not need to be installed)

[root@node1 ~]# yum install ceph-deploy -y

Step 3: create a cluster on node1

Create a cluster configuration directory

Note: most of the following operations will be in this directory

[root@node1 ~]# mkdir /etc/ceph
[root@node1 ~]# cd /etc/ceph

Create a ceph cluster

[root@node1 ceph]# ceph-deploy new node1

[root@node1 ceph]# ls
ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring

ceph.conf  				Cluster profile
ceph-deploy-ceph.log  	use ceph-deploy Deployed logging
ceph.mon.keyring		mon Verification of key file

Step 4: install ceph on the ceph cluster node

The yum source has been prepared when preparing the environment. Here = = all cluster nodes (excluding client) = = install the following software

# yum install ceph ceph-radosgw -y

# ceph -v
ceph version 13.2.6 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)

Supplementary notes:

  • If the public network is OK and the network speed is good, you can use the CEPH deploy install node1 node2 node3 command to install, but if the network speed is bad, it will be worse
  • So here we choose to directly use the prepared local ceph source, and then install ceph using Yum install ceph radosgw - Y.

Step 5: install CEPH common on the client

[root@client ~]# yum install ceph-common -y

Step 6: create mon (monitor)

Add public network for monitoring

stay[global]Add the following sentence to the configuration section (directly to the last line))
[root@node1 ceph]# vim /etc/ceph/ceph.conf 	
public network =			Monitoring network

Monitor node initialization and synchronously configure to all nodes (node1,node2,node3, excluding clients)

[root@node1 ceph]# ceph-deploy mon create-initial
[root@node1 ceph]# ceph health			
HEALTH_OK										state health(Health)

Synchronize profile information to all nodes
[root@node1 ceph]# ceph-deploy admin node1 node2 node3
[root@node1 ceph]# ceph -s
    id:     c05c1f28-ea78-41b7-b674-a069d90553ac
    health: HEALTH_OK							Health status is OK

    mon: 1 daemons, quorum node1				1 Monitoring
    mgr: no daemons active
    osd: 0 osds: 0 up, 0 in

    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   0 B used, 0 B / 0 B avail

In order to prevent single point of mon failure, you can add multiple mon nodes (odd number is recommended because there is quorum arbitration voting)

Review: what is quorum?

[root@node1 ceph]# ceph-deploy mon add node2	
[root@node1 ceph]# ceph-deploy mon add node3

[root@node1 ceph]# ceph -s
    id:     c05c1f28-ea78-41b7-b674-a069d90553ac
    health: HEALTH_OK								Health status is OK
    mon: 3 daemons, quorum node1,node2,node3		3 Monitoring
    mgr: no daemons active							
    osd: 0 osds: 0 up, 0 in
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   0 B used, 0 B / 0 B avail

Solution to time synchronization detected by monitoring

ceph clusters have very high requirements for time synchronization. Even if you have started the ntpd service, there may still be clock skew delayed warnings

Please try the following:

1. On all nodes of ceph cluster (node1,node2,node3), ntpd service is not used, and crontab synchronization is directly used

# systemctl stop ntpd
# systemctl disable ntpd

# crontab -e
*/10 * * * * ntpdate				Synchronize the server of the public network at any time every 5 or 10 minutes

2. Increase the time warning threshold

[root@node1 ceph]# vim ceph.conf
[global]								stay global Add the following two lines to the parameter group						
mon clock drift allowed = 2				# Clock ticks between monitor s (0.5 seconds by default)
mon clock drift warn backoff = 30		# Increase the allowable offset of the clock (5 by default)

3. Synchronize to all nodes

[root@node1 ceph]# ceph-deploy --overwrite-conf admin node1 node2 node3

The previous first synchronization does not need to be added--overwrite-conf parameter
 This modification ceph.conf Resynchronization needs to be added--overwrite-conf Parameter coverage

4. Restart service on all CEPH cluster nodes

# systemctl restart

Step 7: create Mgr (Management)

A new component is added in ceph luminous: Ceph Manager Daemon, or CEPH Mgr for short.

The main function of this component is to share and expand some functions of monitor, reduce the burden of monitor and better manage ceph storage system.

Create an mgr

[root@node1 ceph]# ceph-deploy mgr create node1

[root@node1 ceph]# ceph -s
    id:     c05c1f28-ea78-41b7-b674-a069d90553ac
    health: HEALTH_OK

    mon: 3 daemons, quorum node1,node2,node3
    mgr: node1(active)							node1 by mgr
    osd: 0 osds: 0 up, 0 in

    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   0 B used, 0 B / 0 B avail

Adding multiple MGRS enables HA

[root@node1 ceph]# ceph-deploy mgr create node2
[root@node1 ceph]# ceph-deploy mgr create node3

[root@node1 ceph]# ceph -s
    id:     c05c1f28-ea78-41b7-b674-a069d90553ac
    health: HEALTH_OK								Health status is OK
    mon: 3 daemons, quorum node1,node2,node3		3 Monitoring
    mgr: node1(active), standbys: node2, node3		notice node1 Mainly,node2,node3 For preparation
    osd: 0 osds: 0 up, 0 in							See 0 disks
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   0 B used, 0 B / 0 B avail

Step 8: create OSD (storage disk)

[root@node1 ceph]# ceph-deploy disk --help

[root@node1 ceph]# ceph-deploy osd --help

The disks of all nodes in the list include sda and sdb. sdb is the disk we want to add distributed storage

List view disks on nodes
[root@node1 ceph]# ceph-deploy disk list node1
[root@node1 ceph]# ceph-deploy disk list node2
[root@node1 ceph]# ceph-deploy disk list node3

zap Indicates that the data on the disk is killed,Equivalent to formatting
[root@node1 ceph]# ceph-deploy disk zap node1 /dev/sdb
[root@node1 ceph]# ceph-deploy disk zap node2 /dev/sdb
[root@node1 ceph]# ceph-deploy disk zap node3 /dev/sdb

Create disk as osd
[root@node1 ceph]# ceph-deploy osd create --data /dev/sdb node1
[root@node1 ceph]# ceph-deploy osd create --data /dev/sdb node2
[root@node1 ceph]# ceph-deploy osd create --data /dev/sdb node3
[root@node1 ceph]# ceph -s
    id:     c05c1f28-ea78-41b7-b674-a069d90553ac
    health: HEALTH_OK
    mon: 3 daemons, quorum node1,node2,node3
    mgr: node1(active), standbys: node2, node3
    osd: 3 osds: 3 up, 3 in									See here are three osd
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   41 MiB used, 2.9 GiB / 3.0 GiB avail			The size is the sum of 3 disks

OSDs are created, so how to access data?

Capacity expansion method of cluster node

Suppose you add a new cluster node node4

1. Host name configuration and binding

2. Install CEPH CEPH radosgw - Y on node4 using Yum or up2date

3. Synchronize the configuration file on the deployment node node1 to node4. CEPH deploy admin node4

4. Add mon or mgr or osd on node4 as required

4, RADOS native data access demo

As mentioned above, RADOS can also access data, but we generally do not use it directly, but we can first use RADOS to deeply understand the data access principle of ceph.

Access principle

To achieve data access, you need to create a pool. To create a pool, you need to allocate PG first.

If the client writes a file to a pool, how is the file distributed to the disks of multiple nodes?

The answer is through the CRUSH algorithm.

CRUSH algorithm

  • CRUSH(Controlled Scalable Decentralized Placement of Replicated Data) algorithm is the abbreviation of controllable, scalable and distributed replica data placement algorithm.
  • The process algorithm of PG to OSD mapping is called cross algorithm. (an Object needs to save three copies, that is, it needs to be saved on three OSDs).
  • Cross algorithm is a pseudo-random process. It can randomly select an OSD set from all OSDs, but the result of each random selection of the same PG is unchanged, that is, the mapped OSD set is fixed.


  • The client operates directly on the pool (but we don't do this for file storage, block storage and object storage)
  • PG should be allocated in the pool
  • Multiple objects can be stored in PG
  • An object is a separate unit of data written by the client
  • The CRUSH algorithm maps the data written by the client to the OSD and finally stores it on the physical disk (this specific process is abstract, and our operation and maintenance engineers don't have to dig deeply, because distributed storage is a big hard disk for the operation and maintenance engineers)

Create pool

Create test_pool, the specified number of pg is 128

[root@node1 ceph]# ceph osd pool create test_pool 128
pool 'test_pool' created

To view the PG quantity, you can use CEPH OSD pool set test_ pool pg_ Use commands such as num 64 to try to adjust

[root@node1 ceph]# ceph osd pool get test_pool pg_num
pg_num: 128

Note: the number of pg is related to the number of ods

  • The number of PG is a multiple of 2, generally less than 5 osd, and it can be divided into 128 PG or less (if the number of PG is more, an error will be reported, which can be adjusted down appropriately according to the error)
  • You can use CEPH OSD pool set test_ pool pg_ Use commands such as num 64 to try to adjust

Storage test

1. I upload the local / etc/fstab file to test_pool and name it newfstab

[root@node1 ceph]# rados put newfstab /etc/fstab --pool=test_pool

2. View

[root@node1 ceph]# rados -p test_pool ls

3. Delete

[root@node1 ceph]# rados rm newfstab --pool=test_pool

Delete pool

1. Add a parameter on the deployment node node1 to allow ceph to delete the pool

[root@node1 ceph]# vim /etc/ceph/ceph.conf 
mon_allow_pool_delete = true

2. The configuration is modified to synchronize to other cluster nodes

[root@node1 ceph]# ceph-deploy --overwrite-conf admin node1 node2 node3

3. Restart the monitoring service

[root@node1 ceph]# systemctl restart

4. When deleting, enter the pool name twice, followed by the -- yes-i-really-really-mean-it parameter to delete it

[root@node1 ceph]# ceph osd pool delete test_pool test_pool --yes-i-really-really-mean-it

5, Create Ceph file store

To run Ceph file system, you must first create Ceph storage cluster with at least one mds

(MDS is not used for Ceph block devices and Ceph object storage).

Ceph MDS: Ceph file storage type is used to store and manage metadata metadata

Create a file store and use

Step 1: synchronize the configuration file on the node1 deployment node and create an mds service (you can also do multiple mds to implement HA)

[root@node1 ceph]# ceph-deploy mds create node1 node2 node3
 I'll make three here mds

Step 2: a Ceph file system requires at least two RADOS storage pools, one for data and one for metadata. So we create them.

[root@node1 ceph]# ceph osd pool create cephfs_pool 128
pool 'cephfs_pool' created
[root@node1 ceph]# ceph osd pool create cephfs_metadata 64
pool 'cephfs_metadata' created

[root@node1 ceph]# ceph osd pool ls |grep cephfs

Step 3: create Ceph file system and confirm the node accessed by the client

[root@node1 ceph]# ceph fs new cephfs cephfs_metadata cephfs_pool
[root@node1 ceph]# ceph fs ls
name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_pool ]

[root@node1 ceph]# ceph mds stat
cephfs-1/1/1 up  {0=ceph_node3=up:active}, 2 up:standby		See here node3 by up state

Step 4: the client prepares to verify the key file

  • Note: ceph enables cephx authentication by default, so the mounting of the client must be verified

View the key string on any cluster node (node1,node2,node3)

[root@node1 ~]# cat /etc/ceph/ceph.client.admin.keyring
        key = AQDEKlJdiLlKAxAARx/PXR3glQqtvFFMhlhPmw==		The following string is required for verification
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *"

Create a file record key string on the client

[root@client ~]# vim admin.key			# Create a key file, copy and paste the string obtained above

Step 5: mount the client (mount the node monitored by mon in ceph cluster, and the mon monitoring is port 6789)

[root@client ~]# mount -t ceph node1:6789:/ /mnt -o name=admin,secretfile=/root/admin.key

Step 6: verify

[root@client ~]# df -h |tail -1
node1:6789:/  3.8G     0  3.8G   0% /mnt # Don't worry about the size. Different scenes will affect the number of pg and copies

If you want to verify read and write, please verify yourself

You can use two clients to mount this file storage at the same time, which can read and write at the same time

Delete file storage method

If you need to delete a file store, follow the procedure below

Step 1: delete data on the client and umount all mounts

[root@client ~]# rm /mnt/* -rf
[root@client ~]# umount /mnt/

Step 2: stop mds of all nodes (file storage can be deleted only when mds is stopped)

[root@node1 ~]# systemctl stop
[root@node2 ~]# systemctl stop
[root@node3 ~]# systemctl stop

Step 3: go back to any node of the cluster (one of node1, node2 and node3) and delete it

If you want to delete a client, you need to synchronize the configuration of CEPH deploy admin client on node1

[root@client ~]# ceph fs rm cephfs --yes-i-really-mean-it

[root@client ~]# ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it
pool 'cephfs_metadata' removed

[root@client ~]# ceph osd pool delete cephfs_pool cephfs_pool --yes-i-really-really-mean-it
pool 'cephfs_pool' removed

Step 4: start the mds service again

[root@node1 ~]# systemctl start
[root@node2 ~]# systemctl start
[root@node3 ~]# systemctl start

6, Create Ceph block store

Create block storage and use

Step 1: synchronize the configuration file to the client on node1

[root@node1 ceph]# ceph-deploy admin client

Step 2: establish storage pool and initialize

Note: operate on the client side

[root@client ~]# ceph osd pool create rbd_pool 128		
pool 'rbd_pool' created

[root@client ~]# rbd pool init rbd_pool

Step 3: create a storage volume (the volume name here is volume1 and the size is 5000M)

Note: the professional term of volume1 is image. I call it storage volume here for easy understanding

[root@client ~]# rbd create volume1 --pool rbd_pool --size 5000

[root@client ~]# rbd ls rbd_pool

[root@client ~]# rbd info volume1 -p rbd_pool
rbd image 'volume1':						Can see volume1 by rbd image
        size 4.9 GiB in 1250 objects
        order 22 (4 MiB objects)
        id: 149256b8b4567
        block_name_prefix: rbd_data.149256b8b4567
        format: 2							There are two formats: 1 and 2,Now it's 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten   characteristic
        create_timestamp: Sat Aug 17 19:47:51 2019

Step 4: map the created volume to a block device

  • Because some features of rbd image are not supported by the OS kernel, the mapping error is reported
[root@client ~]# rbd map rbd_pool/volume1
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable rbd_pool/volume1 object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
  • Solution: disable the related features
[root@client ~]# rbd feature disable rbd_pool/volume1 exclusive-lock object-map fast-diff deep-flatten
  • Remap
[root@client ~]# rbd map rbd_pool/volume1

Step 5: view the mapping (if you want to unmap, you can use rbd unmap /dev/rbd0)

[root@client ~]# rbd showmapped
id pool     image   snap device    
0  rbd_pool volume1 -    /dev/rbd0

Step 6: format and mount

[root@client ~]# mkfs.xfs /dev/rbd0

[root@client ~]# mount /dev/rbd0 /mnt/

[root@client ~]# df -h |tail -1
/dev/rbd0        4.9G   33M  4.9G   1%   /mnt

Self verifiable read and write

Note: block storage cannot be read and written at the same time. Please do not mount two clients for reading and writing at the same time

Expansion and reduction of block storage

Online capacity expansion

After testing, after partition, / dev/rbd0p1 cannot be expanded online, and / dev/rbd0 can be used directly

Capacity expansion to 8000 M
[root@client ~]# rbd resize --size 8000 rbd_pool/volume1

[root@client ~]# rbd info rbd_pool/volume1  |grep size
        size 7.8 GiB in 2000 objects

View size,No change
[root@client ~]# df -h |tail -1
/dev/rbd0        4.9G   33M  4.9G   1%   /mnt

[root@client ~]# xfs_growfs -d /mnt/
View size again,Online capacity expansion succeeded
[root@client ~]# df -h |tail -1
/dev/rbd0        7.9G   33M  7.9G   1%   /mnt

Block storage reduction

It cannot be cut online. It needs to be reformatted and mounted after cutting, so please back up the data in advance

Cut back to 5000 M
[root@client ~]# rbd resize --size 5000 rbd_pool/volume1 --allow-shrink

Reformat mount
[root@client ~]# umount /mnt/
[root@client ~]# mkfs.xfs -f /dev/rbd0
[root@client ~]# mount /dev/rbd0 /mnt/
View again,Confirm successful reduction
[root@client ~]# df -h |tail -1
/dev/rbd0        4.9G   33M  4.9G   1%   /mnt

Delete block storage method

[root@client ~]# umount /mnt/

[root@client ~]# rbd unmap /dev/rbd0

[root@client ~]# ceph osd pool delete rbd_pool rbd_pool --yes-i-really-really-mean-it
pool 'rbd_pool' removed

7, Ceph object store

Test the connection of ceph object gateway

Step 1: create rgw on node1

[root@node1 ceph]# ceph-deploy rgw create node1
[root@node1 ceph]# lsof -i:7480
radosgw 6748 ceph   40u  IPv4  49601      0t0  TCP *:7480 (LISTEN)

Step 2: test the connection object gateway on the client

Create a test user,It needs to be used on the deployment node ceph-deploy admin client Sync profile to client
[root@client ~]# radosgw-admin user create --uid="testuser" --display-name="First User"
    "user_id": "testuser",
    "display_name": "First User",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "auid": 0,
    "subusers": [],
    "keys": [
            "user": "testuser",
            "access_key": "36ROCI84S5NSP4BPYL01",
            "secret_key": "jBOKH0v6J79bn8jaAF2oaWU7JvqTxqb4gjerWOFW"
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []

The main useful section above is access_key and secret_key, used to connect to the object storage gateway

[root@client ~]# radosgw-admin user create --uid='testuser' --display-name='First User' |grep -E 'access_key|secret_key'
            "access_key": "36ROCI84S5NSP4BPYL01",
            "secret_key": "jBOKH0v6J79bn8jaAF2oaWU7JvqTxqb4gjerWOFW"

S3 connection ceph object gateway

Amazon s3 is an Internet oriented object storage service. We can use s3 tool to connect ceph's object storage for operation

Step 1: install s3cmd tool on the client and write ceph connection configuration file

[root@client ~]# yum install s3cmd

Create and write the following file,key The file corresponds to the test user created earlier key
[root@client ~]# vim /root/.s3cfg
access_key = 36ROCI84S5NSP4BPYL01
secret_key = jBOKH0v6J79bn8jaAF2oaWU7JvqTxqb4gjerWOFW 
host_base =
host_bucket =
cloudfront_host =
use_https = False

Step 2: Command test

list bucket,You can view the created by the previous test my-new-bucket
[root@client ~]# s3cmd ls
2019-01-05 23:01  s3://my-new-bucket
 Build another bucket
[root@client ~]# s3cmd mb s3://test_bucket
 Upload file to bucket
[root@client ~]# s3cmd put /etc/fstab s3://test_bucket
upload: '/etc/fstab' -> 's3://test_bucket/fstab'  [1 of 1]
 501 of 501   100% in    1s   303.34 B/s  done
 Download to current directory
[root@client ~]# s3cmd get s3://test_bucket/fstab

See reference command help for more commands
[root@client ~]# s3cmd --help

CEPH dashboard (expansion)

Visual monitoring of ceph storage system is completed through ceph dashboard.

Step 1: check the cluster status and confirm the active node of mgr

[root@node1 ~]# ceph -s
    id:     6788206c-c4ea-4465-b5d7-ef7ca3f74552
    health: HEALTH_OK

    mon: 3 daemons, quorum node1,node2,node3
    mgr: node1(active), standbys: node3, node2		 	confirm mgr of active Node is node1
    osd: 4 osds: 4 up, 4 in
    rgw: 1 daemon active

    pools:   6 pools, 48 pgs
    objects: 197  objects, 2.9 KiB
    usage:   596 MiB used, 3.4 GiB / 4.0 GiB avail
    pgs:     48 active+clean

Step 2: open the dashboard module

[root@node1 ~]# ceph mgr module enable dashboard

Step 3: create a self signed certificate

[root@node1 ~]# ceph dashboard create-self-signed-cert
Self-signed certificate created

Step 4: generate a key pair and configure it to ceph mgr

[root@node1 ~]# mkdir /etc/mgr-dashboard

[root@node1 ~]# cd /etc/mgr-dashboard/

[root@node1 mgr-dashboard]# openssl req -new -nodes -x509 -subj "/O=IT-ceph/CN=cn" -days 365 -keyout dashboard.key -out dashboard.crt -extensions v3_ca
Generating a 2048 bit RSA private key
writing new private key to 'dashboard.key'

[root@node1 mgr-dashboard]# ls
dashboard.crt  dashboard.key

Step 5: configure mgr services on the active mgr node of ceph cluster (node1 here)

The dashboard service is mainly used to configure the IP and Port used by the dashboard

[root@node1 mgr-dashboard]# ceph config set mgr mgr/dashboard/server_addr
[root@node1 mgr-dashboard]# ceph config set mgr mgr/dashboard/server_port 8080

Step 6: restart the dashboard module and check the access address

[root@node1 mgr-dashboard]# ceph mgr module disable dashboard                 
[root@node1 mgr-dashboard]# ceph mgr module enable dashboard                    

[root@node1 mgr-dashboard]# ceph mgr services
    "dashboard": ""

Step 7: set the user name and password for accessing the web page

[root@node1 mgr-dashboard]# ceph dashboard set-login-credentials daniel daniel123
Username and password updated

Step 8: access through this machine or other hosts

ceph object storage combined with owncloud to create cloud disk (expansion)

1. Prepare the bucket and related connection key s on the ceph client

[root@client ~]# s3cmd mb s3://owncloud
Bucket 's3://owncloud/' created

[root@client ~]# cat /root/.s3cfg
access_key = 36ROCI84S5NSP4BPYL01
secret_key = jBOKH0v6J79bn8jaAF2oaWU7JvqTxqb4gjerWOFW 
host_base =
host_bucket =
cloudfront_host =
use_https = False

2. Install the web environment required by the owncloud cloud disk on the client side

Owncloud requires web server and PHP support. At present, the latest version of owncloud requires php7.x version. Here we use rpm version to save time

[root@client ~]# yum install httpd mod_ssl php-mysql php php-gd php-xml php-mbstring -y

[root@client ~]# systemctl restart httpd

3. Upload the owncloud package and unzip it to the httpd home directory

[root@client ~]# tar xf owncloud-9.0.1.tar.bz2 -C /var/www/html/

[root@client ~]# chown apache.apache -R /var/www/html/
It needs to be modified to run web User of the server owner,group,Otherwise, there will be permission problems when writing later

4. Access http: through the browser for configuration

5. File upload and download test

[root@client ~]# s3cmd put /etc/fstab s3://owncloud
upload: '/etc/fstab' -> 's3://owncloud/fstab'  [1 of 1]
 501 of 501   100% in    0s     6.64 kB/s  done

Because the default owncloud upload file is limited to 2M. So it needs to be modified

[root@client ~]# vim /var/www/html/owncloud/.htaccess

<IfModule mod_php5.c>
  php_value upload_max_filesize 2000M				Modify and enlarge
  php_value post_max_size 2000M						Modify and enlarge

[root@client ~]# vim /etc/php.ini

post_max_size = 2000M								Modify and enlarge		
upload_max_filesize = 2000M							Modify and enlarge

[root@client ~]# systemctl restart httpd

Tags: Python Linux cloud computing RESTful

Posted on Wed, 06 Oct 2021 19:03:37 -0400 by mesh2005