Differences between ceph osd reweight and osd crush weight

  • osd crush weight
  • osd weight
  • crushtool modifies crushmap
  • test

When we execute the ceph osd tree command, a WEIGHT and REWEIGHT are displayed in the display. What are they?

[root@xt7 ceph]# ceph osd tree
//The second column corresponds to osd crush weight, and the last column corresponds to osd weight
ID  WEIGHT   TYPE NAME             UP/DOWN REWEIGHT PRIMARY-AFFINITY
-13  2.66554 root metadata
-14  1.00401     host xt7-metadata
 23  1.00000         osd.23             up  1.00000          1.00000
-15  1.05763     host xt6-metadata
 11  1.00000         osd.11             up  1.00000          1.00000
-16  0.60390     host xt8-metadata
 35  1.00000         osd.35             up  1.00000          1.00000
-12        0 root default
-11        0     host xt7-default
-10        0     host xt6-default
 -9        0     host xt8-default
 -8  2.90688 root ssd
 -7  0.79999     host xt7-ssd
 14  0.79999         osd.14             up  1.00000          1.00000
 -6  0.90689     host xt6-ssd
  2  1.00000         osd.2              up  1.00000          1.00000
 -5  1.20000     host xt8-ssd
 26  1.20000         osd.26             up  1.00000          1.00000
 -4 30.99991 root hdd
 -3  8.99994     host xt7-hdd
 12  0.79999         osd.12             up  1.00000          1.00000
 13  0.79999         osd.13             up  1.00000          1.00000
 15  0.79999         osd.15             up  1.00000          1.00000
 16  0.79999         osd.16             up  1.00000          1.00000
 17  0.79999         osd.17             up  0.70000          1.00000
 18  1.00000         osd.18             up  1.00000          1.00000
 19  1.00000         osd.19             up  1.00000          1.00000
 20  1.00000         osd.20             up  1.00000          1.00000
 21  1.00000         osd.21             up  1.00000          1.00000
 22  1.00000         osd.22             up  1.00000          1.00000
 -2 10.00000     host xt6-hdd
 0  1.00000         osd.0              up  1.00000          1.00000
 1  1.00000         osd.1              up  1.00000          1.00000
 3  1.00000         osd.3              up  1.00000          1.00000
 4  1.00000         osd.4              up  1.00000          1.00000
 5  1.00000         osd.5              up  1.00000          1.00000
 6  1.00000         osd.6              up  1.00000          1.00000
 7  1.00000         osd.7              up  1.00000          1.00000
 8  1.00000         osd.8              up  1.00000          1.00000
 9  1.00000         osd.9              up  1.00000          1.00000
 10  1.00000         osd.10             up  1.00000          1.00000
 -1 11.99997     host xt8-hdd
 24  1.20000         osd.24             up  1.00000          1.00000
 25  1.20000         osd.25             up  1.00000          1.00000
 27  1.20000         osd.27             up  1.00000          1.00000
 28  1.20000         osd.28             up  1.00000          1.00000
 29  1.20000         osd.29             up  1.00000          1.00000
 30  1.20000         osd.30             up  1.00000          1.00000
 31  1.20000         osd.31             up  1.00000          1.00000
 32  1.20000         osd.32             up  1.00000          1.00000
 33  1.20000         osd.33             up  1.00000          1.00000
 34  1.20000         osd.34             up  1.00000          1.00000

osd crush weight

Crush weight is actually a bucket item weight. Here is a description of the bucket item weight:

Weighting Bucket Items
Ceph expresses bucket weights as doubles, which allows for fine weighting. A weight is the relative difference between device capacities. We recommend using 1.00 as the relative weight for a 1TB storage device. In such a scenario, a weight of 0.5 would represent approximately 500GB, and a weight of 3.00 would represent approximately 3TB. Higher level buckets have a weight that is the sum total of the leaf items aggregated by the bucket.
A bucket item weight is one dimensional, but you may also calculate your item weights to reflect the performance of the storage drive. For example, if you have many 1TB drives where some have relatively low data transfer rate and the others have a relatively high data transfer rate, you may weight them differently, even though they have the same capacity (e.g., a weight of 0.80 for the first set of drives with lower total throughput, and 1.20 for the second set of drives with higher total throughput).

"ceph osd crush reweight" sets the CRUSH weight of the OSD. This weight is an arbitrary value (generally the size of the disk in TB or something) and controls how much data the system tries to allocate to the OSD.

Simply put, a bucket weight represents the capacity of a device, 1TB equals 1.00, 500G equals 0.5, and a bucket weight is the sum of all item weights. The change of item weight affects the change of bucket weight, that is, osd.X affects the host.Adjustments to it immediately redistribute the pg and migrate the data, which is typically set based on the osd's capacity when the init10,000 OSD is just started.

Command:
ceph osd crush reweight osd.1 1.2

osd weight

osd weight values are 0-1.osd reweight does not affect host.When osd is kicked out of the cluster, osd weight is set to 0 and 1 when joining the cluster.

"ceph osd reweight" sets an override weight on the OSD. This value is in the range 0 to 1, and forces CRUSH to re-place (1-weight) of the data that would otherwise live on this drive. It does *not* change the weights assigned to the buckets above the OSD, and is a corrective measure in case the normal CRUSH distribution isn't working out quite right. (For instance, if one of your OSDs is at 90% and the others are at 50%, you could reduce this weight to try and compensate for it.)

Note that 'ceph osd reweight' is not a persistent setting. When an OSD gets marked out, the osd weight will be set to 0. When it gets marked in again, the weight will be changed to 1.
Because of this 'ceph osd reweight' is a temporary solution. You should only use it to keep your cluster running while you're ordering more hardware.

osd weight also immediately reassigns the pg and reassigns (USE_DATA * (1-weight)) data locally for online data migration, which is typically temporarily lowered for osd near full or osd full, adding disk extensions to clusters (deleting unused data is also a common way).
Command:
ceph osd reweight 1 0.7

Use crushtool to modify crushmap:

Get the current crushmap:
ceph osd getcrushmap -o crushmap.bin

List the usage and number of copies of a pool:
ceph osd dump | grep '^pool 0'

Decompilate crushmap:
curshtool -d crushmap.bin -o crushmap.txt

Modify:
vim crushmap.txt

host xt1-hdd {
    id -1       # do not change unnecessarily
    # weight 2.500
    alg straw
    hash 0  # rjenkins1
    item osd.1 weight 1.500  //Modify these two values, noting that only item weight can be modified here
    item osd.0 weight 1.500
}

Recompile the new crushmap:
crushtool -c crushmap.txt -o crushmap-new.bin

Apply the new CRUSH map to the ceph cluster:
ceph osd setcrushmap -i crushmap-new.bin

ceph osd df --cluster xtao
ID WEIGHT  REWEIGHT SIZE   USE   AVAIL  %USE  VAR  PGS
 2 1.00000  1.00000  7668M  959M  6709M 12.52 1.29 166
 3 1.00000  0.70000  7668M  528M  7140M  6.89 0.71  90
 1 1.50000  1.00000  7668M  920M  6748M 12.00 1.23 141
 0 1.50000  1.00000  7668M  572M  7096M  7.46 0.77 132
              TOTAL 30675M 2980M 27695M  9.72
MIN/MAX VAR: 0.71/1.29  STDDEV: 2.53

Effective indeed

My test 1: ceph osd crush reweight

Before testing:

ceph health detail --cluster xtao
HEALTH_OK

ceph pg dump pgs_brief --cluster xtao | less
pg_stat     state        up   up_primary   acting  acting_primary
9.6b     active+clean    [3,0]     3         [3,0]       3
10.68    active+clean    [0,3]     0         [0,3]       0
10.1f   active+clean  [2,1]    2       [2,1]   2

ceph osd df --cluster xtao
ID WEIGHT  REWEIGHT SIZE   USE   AVAIL  %USE  VAR  PGS
 2 1.00000  1.00000  7668M  719M  6949M  9.38 0.97 119
 3 1.00000  1.00000  7668M  754M  6914M  9.84 1.02 137
 1 1.00000  1.00000  7668M  646M  7022M  8.42 0.87 121
 0 1.00000  1.00000  7668M  836M  6832M 10.91 1.13 135
              TOTAL 30675M 2956M 27719M  9.64
MIN/MAX VAR: 0.87/1.13  STDDEV: 0.89

sh /ws/dump_pg.sh
pool :    9    10    | SUM
--------------------------------
osd.0    71    64    | 135
osd.1    57    64    | 121
osd.2    63    56    | 119
osd.3    65    72    | 137
--------------------------------
SUM :    256    256    |

Let's take osd.1 as an example. It has 121 PGs on it, and we'll look at its changes with pg 10.1f. It was above [2,1] before, and there's an object below it:

ceph osd map cephfs_data 10000002213.00000015 --cluster xtao
osdmap e1148 pool 'cephfs_data' (10) object '10000002213.00000015' -> pg 10.553f4b1f (10.1f) -> up ([2,1], p2) acting ([2,1], p2)

ls /var/lib/ceph/osd/xtao-1/current/10.1f_head/10000002213.00000015__head_553F4B1F__a
-rw-r--r-- 1 root root 4.0M 7month  19 21:36 10000002213.00000015__head_553F4B1F__a

Next, start the experiment:
Modify crush weight s of osd.1 to 1.5

ceph osd crush reweight osd.1 1.5 --cluster xtao
reweighted item id 1 name 'osd.1' to 1.5 in crush map

ceph health detail --cluster xtao
HEALTH_ERR 35 pgs are stuck inactive for more than 300 seconds; 1 pgs backfill_wait; 14 pgs degraded; 9 pgs peering; 35 pgs stuck inactive; 38 pgs stuck unclean; recovery 5/912 objects degraded (0.548%); recovery 4/912 objects misplaced (0.439%)
pg 10.9 is stuck inactive for 371.069851, current state activating+remapped, last acting [3,0]

pg 9.4d is stuck inactive for 424.559393, current state activating+remapped, last acting [1,2]
pg 10.1f is stuck inactive for 371.326352, current state activating+remapped, last acting [2,1]
pg 9.25 is stuck inactive for 583.209478, current state activating, last acting [1,2]
pg 10.1f is stuck unclean for 371.326495, current state activating+remapped, last acting [2,1]
pg 9.4d is stuck unclean for 424.559540, current state activating+remapped, last acting [1,2]
pg 9.7d is stuck unclean for 387.550744, current state activating+remapped, last acting [3,1]
pg 9.4c is stuck unclean for 378.600928, current state activating+remapped, last acting [3,0]
...
pg 10.2a is peering, acting [1,3]
pg 9.29 is activating+degraded, acting [3,1]
pg 10.2e is peering, acting [1,2]
pg 10.32 is activating+degraded, acting [1,3]
pg 10.39 is peering, acting [1,2]
pg 9.3a is activating+degraded, acting [2,1]
pg 10.6f is peering, acting [1,2]
pg 10.7d is active+degraded, acting [0,2]
pg 9.f is activating+degraded, acting [2,1]
pg 9.c is activating+degraded, acting [3,1]
recovery 5/912 objects degraded (0.548%)
recovery 4/912 objects misplaced (0.439%)

pg 10.1f status changed to activating+remapped
Wait until remap + peering is complete:

ceph pg dump pgs_brief --cluster xtao | less
pg_stat     state         up   up_primary   acting  acting_primary
10.1f   active+clean  [0,2]    0       [0,2]      0

pg 10.1f Of actiing Changed:[2,1] -> [0,2]

ceph osd df --cluster xtao
ID WEIGHT  REWEIGHT SIZE   USE   AVAIL  %USE  VAR  PGS
 2 1.00000  1.00000  7668M  716M  6952M  9.34 0.97 118
 3 1.00000  1.00000  7668M  768M  6900M 10.02 1.04 138
 1 1.50000  1.00000  7668M  912M  6756M 11.90 1.23 162
 0 1.00000  1.00000  7668M  568M  7100M  7.41 0.77  94
              TOTAL 30675M 2964M 27711M  9.66
MIN/MAX VAR: 0.77/1.23  STDDEV: 1.60

osd 1There's more data on it: 646M —> 912M


sh /ws/dump_pg.sh
pool :    9    10    | SUM
--------------------------------
osd.0    48    46    | 94
osd.1    80    82    | 162
osd.2    62    56    | 118
osd.3    66    72    | 138
--------------------------------
SUM :    256    256    |

osd 1Above pg Much more:121 --> 162

My test 2: ceph osd reweight

Continue with the above results:

ceph osd reweight 3 0.7 --cluster xtao

eph pg dump pgs_brief --cluster xtao |less
pg_stat     state         up      up_primary      acting  acting_primary
 9.6b    active+clean    [3,0]        3            [3,0]        3

ceph health detail --cluster xtao
HEALTH_ERR 48 pgs are stuck inactive for more than 300 seconds; 5 pgs degraded; 34 pgs peering; 48 pgs stuck inactive
pg 9.39 is stuck inactive for 46841.599332, current state remapped+peering, last acting [3,1]
pg 10.55 is stuck inactive for 47017.191571, current state activating+degraded, last acting [0,2]
pg 9.3e is stuck inactive for 46817.170840, current state remapped+peering, last acting [1,3]
pg 9.7 is stuck inactive for 46841.604976, current state activating+remapped, last acting [0,3]
pg 9.43 is stuck inactive for 46774.034149, current state remapped+peering, last acting [3,1]
pg 10.73 is stuck inactive for 46970.796402, current state remapped+peering, last acting [1,3]
pg 10.46 is stuck inactive for 47000.825895, current state remapped+peering, last acting [1,3]
pg 10.17 is stuck inactive for 47057.230433, current state activating+degraded, last acting [0,2]
pg 10.3c is stuck inactive for 47007.838808, current state remapped+peering, last acting [1,3]
pg 9.44 is stuck inactive for 46841.600278, current state remapped+peering, last acting [3,1]
pg 9.75 is stuck inactive for 46770.993348, current state remapped+peering, last acting [3,1]
pg 9.4b is stuck inactive for 1194759.272957, current state activating+degraded, last acting [0,2]
pg 9.48 is stuck inactive for 46841.600050, current state activating, last acting [0,2]
pg 10.4b is stuck inactive for 46817.057872, current state remapped+peering, last acting [3,1]
pg 9.8 is stuck inactive for 46841.602502, current state remapped+peering, last acting [1,3]
pg 9.6b is stuck inactive for 935534.199493, current state activating+remapped, last acting [0,3]
pg 9.61 is stuck inactive for 46841.598176, current state activating+remapped, last acting [0,3]

ceph osd df --cluster xtao
ID WEIGHT  REWEIGHT SIZE   USE   AVAIL  %USE  VAR  PGS
 2 1.00000  1.00000  7668M  962M  6706M 12.56 1.29 166
 3 1.00000  0.70000  7668M  528M  7140M  6.89 0.71  90
 1 1.50000  1.00000  7668M  917M  6751M 11.97 1.23 159
 0 1.00000  1.00000  7668M  566M  7102M  7.39 0.76  97
              TOTAL 30675M 2975M 27700M  9.70
MIN/MAX VAR: 0.71/1.29  STDDEV: 2.56

osd.3Data volume reduction: 768 M —> 528M,Data volume changed to original68.75% 

sh /ws/dump_pg.sh
dumped all in format plain

pool :    9    10    | SUM
--------------------------------
osd.0    49    48    | 97
osd.1    79    80    | 159
osd.2    88    78    | 166
osd.3    40    50    | 90
--------------------------------
SUM :    256    256    |

//You can see osd.3Above pg Reduced number:138 --> 90


Author: Ivan Guan
Link: https://www.jianshu.com/p/bfcf91fd10da
Source: Short Book
Copyright belongs to the author.For commercial reprinting, please contact the author for authorization. For non-commercial reprinting, please indicate the source.

Tags: Operation & Maintenance osd Ceph less vim

Posted on Mon, 23 Mar 2020 12:44:28 -0400 by draco2317