Documentation | |
---|---|
Name: | [TROUBLESHOOT] Ceph too few pgs per osd |
Description: | how to solve this “issue” |
Modification date : | 13/06/2019 |
Owner: | dodger |
Notify changes to: | Owner |
Tags: | ceph, object storage |
Scalate to: | The_fucking_bofh |
This documents cover the TOO FEW PGs per OSD, not too many (documented here)
ceph@avmlp-osm-001 ~/ceph-deploy $ sudo ceph health HEALTH_WARN too few PGs per OSD (4 < min 30)
Official Documentation: http://docs.ceph.com/docs/master/rados/operations/health-checks/#many-objects-per-pg
So “the problem” is that there's any pool with less PGs (pg_num
) that the configured in the warning threshold.
There are 2 solutions for this warning:
pg_num
or pg_num_min
in autoscale mode.See which pool is the one with less PG's than expected:
ceph osd pool ls detail
Just change the pg_num
, pgp_num
values:
ceph osd pool set ${POOL_NAME} pg_num 500 ceph osd pool set ${POOL_NAME} pgp_num 500
All pool at once (TAKE CARE with this, I've wrote an echo
to avoid problems, think twice before executing):
for i in $(ceph osd pool ls) ; do echo ceph osd pool set ${i} pg_num 60 ; done
WARNING a huge number of PGs in all the pools can lead in error:
Error ERANGE: pool id 4 pg_num 500 size 3 would mean 6000 total pgs, which exceeds max 5000 (mon_max_pg_per_osd 250 * num_in_osds 20)
See [TROUBLESHOOT] Error ERANGE: pool id X pg_num Y size Z would mean W total pgs, which exceeds
That's for pg_autoscale_mode on
:
ceph osd pool set ${POOL_NAME} pg_num_min ${MINPGS}
If pool actual pg_num
is lower than pg_num_min
you'll have to change pg_num
also:
ceph osd pool set ${POOL_NAME} pg_num ${MINPGS} ceph osd pool set ${POOL_NAME} pg_num_min ${MINPGS}
Add the following to ceph.conf
under [global]
section:
mon pg warn min per osd = 8
and restart mon
and mgr
(mon to apply the changes, mgr for the monitoring).