====== [TROUBLESHOOT] Ceph too few pgs per osd ======
^ Documentation ^|
^Name:| [TROUBLESHOOT] Ceph too few pgs per osd |
^Description:| how to solve this "issue" |
^Modification date :|13/06/2019|
^Owner:|dodger|
^Notify changes to:|Owner |
^Tags:|ceph, object storage |
^Scalate to:|The_fucking_bofh|
====== WARNING ======
This documents cover the **TOO FEW** PGs per OSD, not //too many// (documented [[linux:ceph:troubleshooting:too_many_pgs_per_osd|here]])
====== The error ======
ceph@avmlp-osm-001 ~/ceph-deploy $ sudo ceph health
HEALTH_WARN too few PGs per OSD (4 < min 30)
Official Documentation: [[http://docs.ceph.com/docs/master/rados/operations/health-checks/#many-objects-per-pg]]
\\
So "the problem" is that there's any pool with less PGs (''pg_num'') that the configured in the warning threshold.
====== The solutions ======
There are 2 solutions for this warning:
- Increase the ''pg_num'' or ''pg_num_min'' in //autoscale// mode.
- Change the warning threshold.
===== Increase pg_num/pg_num_min =====
==== pg_num ====
See which pool is the one with less PG's than expected:
ceph osd pool ls detail
Just change the ''pg_num'', ''pgp_num'' values:
ceph osd pool set ${POOL_NAME} pg_num 500
ceph osd pool set ${POOL_NAME} pgp_num 500
All pool at once (**TAKE CARE** with this, I've wrote an ''echo'' to avoid problems, think twice before executing):
for i in $(ceph osd pool ls) ; do echo ceph osd pool set ${i} pg_num 60 ; done
**WARNING** a huge number of PGs in all the pools can lead in error:\\
Error ERANGE: pool id 4 pg_num 500 size 3 would mean 6000 total pgs, which exceeds max 5000 (mon_max_pg_per_osd 250 * num_in_osds 20)
See [[linux:ceph:troubleshooting:erange_total_pg_exceeded|[TROUBLESHOOT] Error ERANGE: pool id X pg_num Y size Z would mean W total pgs, which exceeds]]
==== pg_num_min ====
That's for ''pg_autoscale_mode on'':
ceph osd pool set ${POOL_NAME} pg_num_min ${MINPGS}
If pool actual ''pg_num'' is lower than ''pg_num_min'' you'll have to change ''pg_num'' also:
ceph osd pool set ${POOL_NAME} pg_num ${MINPGS}
ceph osd pool set ${POOL_NAME} pg_num_min ${MINPGS}
===== Change the warning threshold =====
Add the following to ''ceph.conf'' under ''[global]'' section:
mon pg warn min per osd = 8
and restart ''mon'' **and** ''mgr'' (mon to apply the changes, mgr for the monitoring).