Ceph: Too Many PGs on OSD

Question

I configured Ceph with the recommended values ​​(using the formula from the docs). I have 3 on-screen menus, and my configuration (which I installed on the monitor node and all 3 on-screen menus) includes the following:

osd pool default size = 2 osd pool default min size = 1 osd pool default pg num = 150 osd pool default pgp num = 150 

When I run ceph status , I get:

  health HEALTH_WARN too many PGs per OSD (1042 > max 300) 

This is confusing for two reasons. Firstly, because the recommended formula did not satisfy Ceph. Secondly, and most perplexing, is that it says I have 1042 PGs on OSD when my configuration says 150.

What am I doing wrong?

Answer 1

Before setting up a PG account, you need to know 3 things.

1. Number of on-screen menus

 ceph osd ls Sample Output: 0 1 2 Here Total number of osd is three. 

2. The number of pools

ceph osd pool ls or rados lspools

 Sample Output: rbd images vms volumes backups Here Total number of pool is five. 

3. The amount of replication

 ceph osd dump | grep repli Sample Output: pool 0 'rbd' replicated size 2 min_size 2 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 38 flags hashpspool stripe_width 0 pool 1 'images' replicated size 2 min_size 2 crush_ruleset 1 object_hash rjenkins pg_num 30 pgp_num 30 last_change 40 flags hashpspool stripe_width 0 pool 2 'vms' replicated size 2 min_size 2 crush_ruleset 1 object_hash rjenkins pg_num 30 pgp_num 30 last_change 42 flags hashpspool stripe_width 0 pool 3 'volumes' replicated size 2 min_size 2 crush_ruleset 1 object_hash rjenkins pg_num 30 pgp_num 30 last_change 36 flags hashpspool stripe_width 0 pool 4 'backups' replicated size 2 min_size 2 crush_ruleset 1 object_hash rjenkins pg_num 30 pgp_num 30 last_change 44 flags hashpspool stripe_width 0 You can see each pool has replication count two. 

Now let's move on to calculating

Calculations:

General PG calculation:

 Total PGs = (Total_number_of_OSD * 100) / max_replication_count This result must be rounded up to the nearest power of 2. 

Example:

No OSD: 3
Number of replication: 2

Total PGs = (3 * 100) / 2 = 150. The closest power from 150 to 2 is 256.

Therefore, the maximum recommended PG is 256

You can set PG for each pool

Total PG per pool Calculation:

 Total PGs = ((Total_number_of_OSD * 100) / max_replication_count) / pool count This result must be rounded up to the nearest power of 2. 

Example:

No OSD: 3
Number of replication: 2
No pools: 5

Total PGs = ((3 * 100) / 2) / 5 = 150/5 = 30. The nearest power from 30 to 2 is 32.

Thus, the total number of PGs per pool is 32.

Power 2 tables:

 2^0 1 2^1 2 2^2 4 2^3 8 2^4 16 2^5 32 2^6 64 2^7 128 2^8 256 2^9 512 2^10 1024 

Useful Commands

 ceph osd pool create <pool-name> <pg-number> <pgp-number> - To create a new pool ceph osd pool get <pool-name> <pg_num> - To get number of PG in a pool ceph osd pool get <pool-name> <pgp_num> - To get number of PGP in a pool ceph osd pool set <pool-name> <pg_num number> - To increase number of PG in a pool ceph osd pool set <pool-name> <pgp_num number> - To increase number of PGP in a poo *usually pg and pgp number is same 

Answer 2

As I fixed it in 12.2.4 with light:

Too many PGs on the OSD (380> 200 maximum) can lead to multiple lock requests.

First you need to install:

 [global] mon_max_pg_per_osd = 800 # < depends on you amount of PGs osd max pg per osd hard ratio = 10 # < default is 2, try to set at least 5. It will be mon allow pool delete = true # without it you can't remove a pool 

Then restart all MON and OSDs one by one.

Check the value:

 ceph --admin-daemon /var/run/ceph/ceph-mon.ceph2.asok config get mon_max_pg_per_osd ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config get osd_max_pg_per_osd_hard_ratio 

Now look here:

 rados lspools ceph osd pool get .users.email pg_num 

In my case, the default pg_num was 128 or something like this (my cluster was 4 years old, there were a lot of changes, a lot of changes). You can reduce it this way.

Be careful:

 ceph osd pool create .users.email.new 8 rados cppool .users.email default.rgw.lc.new ceph osd pool delete .users.email .users.email --yes-i-really-really-mean-it ceph osd pool rename .users.email.new .users.email ceph osd pool application enable .users.email rgw 

If this was not enough, try to find another pool that you can cut.