User Tools

Site Tools


linux:ceph:troubleshooting:too_few_pgs_per_osd

[TROUBLESHOOT] Ceph too few pgs per osd

Documentation
Name: [TROUBLESHOOT] Ceph too few pgs per osd
Description: how to solve this “issue”
Modification date :13/06/2019
Owner:dodger
Notify changes to:Owner
Tags:ceph, object storage
Scalate to:The_fucking_bofh

WARNING

This documents cover the TOO FEW PGs per OSD, not too many (documented here)

The error

ceph@avmlp-osm-001 ~/ceph-deploy $ sudo ceph health
HEALTH_WARN too few PGs per OSD (4 < min 30)

Official Documentation: http://docs.ceph.com/docs/master/rados/operations/health-checks/#many-objects-per-pg
So “the problem” is that there's any pool with less PGs (pg_num) that the configured in the warning threshold.

The solutions

There are 2 solutions for this warning:

  1. Increase the pg_num or pg_num_min in autoscale mode.
  2. Change the warning threshold.

Increase pg_num/pg_num_min

pg_num

See which pool is the one with less PG's than expected:

ceph osd pool ls detail

Just change the pg_num, pgp_num values:

ceph osd pool set ${POOL_NAME} pg_num 500
ceph osd pool set ${POOL_NAME} pgp_num 500

All pool at once (TAKE CARE with this, I've wrote an echo to avoid problems, think twice before executing):

for i in $(ceph osd pool ls) ; do echo ceph osd pool set ${i} pg_num 60 ; done

WARNING a huge number of PGs in all the pools can lead in error:

Error ERANGE: pool id 4 pg_num 500 size 3 would mean 6000 total pgs, which exceeds max 5000 (mon_max_pg_per_osd 250 * num_in_osds 20)

See [TROUBLESHOOT] Error ERANGE: pool id X pg_num Y size Z would mean W total pgs, which exceeds

pg_num_min

That's for pg_autoscale_mode on:

ceph osd pool set ${POOL_NAME} pg_num_min ${MINPGS}

If pool actual pg_num is lower than pg_num_min you'll have to change pg_num also:

ceph osd pool set ${POOL_NAME} pg_num ${MINPGS}
ceph osd pool set ${POOL_NAME} pg_num_min ${MINPGS}

Change the warning threshold

Add the following to ceph.conf under [global] section:

mon pg warn min per osd = 8

and restart mon and mgr (mon to apply the changes, mgr for the monitoring).

linux/ceph/troubleshooting/too_few_pgs_per_osd.txt · Last modified: 2022/02/11 11:36 by 127.0.0.1