User Tools

Site Tools


linux:ceph:troubleshooting:pg_degraded_redundancy

[TROUBLESHOOT] PG_DEGRADED Degraded data redundancy

Documentation
Name: [TROUBLESHOOT] PG_DEGRADED Degraded data redundancy
Description: how to solve this “issue”
Modification date :13/06/2019
Owner:dodger
Notify changes to:Owner
Tags:ceph, object storage
Scalate to:The_fucking_bofh

The error

avmlp-osm-001 /etc/ceph # ceph health 
HEALTH_WARN Degraded data redundancy: 1329 pgs undersized
avmlp-osm-001 /etc/ceph # ceph health detail
HEALTH_WARN Degraded data redundancy: 1329 pgs undersized
PG_DEGRADED Degraded data redundancy: 1329 pgs undersized
    pg 1.168 is stuck undersized for 72516.102692, current state active+undersized+remapped, last acting [14,4,2]
    pg 1.169 is stuck undersized for 72516.105401, current state active+undersized+remapped, last acting [18,2,6]
    pg 1.16a is stuck undersized for 72516.098682, current state active+undersized+remapped, last acting [8,0,10]
    pg 1.16b is stuck undersized for 72516.102384, current state active+undersized+remapped, last acting [8,12,6]
    pg 1.16d is stuck undersized for 72516.105526, current state active+undersized+remapped, last acting [18,2,10]
    pg 1.16e is stuck undersized for 72516.097173, current state active+undersized+remapped, last acting [4,8,2]
    pg 1.16f is stuck undersized for 72516.105602, current state active+undersized+remapped, last acting [6,18,2]
    pg 1.170 is stuck undersized for 72516.102068, current state active+undersized+remapped, last acting [14,4,2]
    pg 1.171 is stuck undersized for 72516.101080, current state active+undersized+remapped, last acting [14,2,6]
    pg 1.172 is stuck undersized for 72516.108720, current state active+undersized+remapped, last acting [16,0,10]
    pg 1.173 is stuck undersized for 72516.102557, current state active+undersized+remapped, last acting [12,6,14]
    pg 1.174 is stuck undersized for 72516.103496, current state active+undersized+remapped, last acting [18,4,2]
    pg 1.175 is stuck undersized for 72516.101506, current state active+undersized+remapped, last acting [4,2,10]
    pg 1.176 is stuck undersized for 72516.098652, current state active+undersized+remapped, last acting [2,8,4]
    pg 1.177 is stuck undersized for 72516.104113, current state active+undersized+remapped, last acting [0,18,2]
    pg 1.1f0 is stuck undersized for 72516.100203, current state active+undersized+remapped, last acting [8,4,2]
    pg 1.1f1 is stuck undersized for 72516.104306, current state active+undersized+remapped, last acting [18,2,6]
    pg 1.1f2 is stuck undersized for 72516.101514, current state active+undersized+remapped, last acting [14,0,10]
    pg 1.1f3 is stuck undersized for 72516.101139, current state active+undersized+remapped, last acting [12,6,14]
    pg 2.168 is stuck undersized for 72491.414882, current state active+undersized+remapped, last acting [18,14,16]
    pg 2.169 is stuck undersized for 72491.416090, current state active+undersized+remapped, last acting [6,0,10]
    pg 2.16a is stuck undersized for 72491.408729, current state active+undersized+remapped, last acting [10,0,14]
    pg 2.16b is stuck undersized for 72491.409007, current state active+undersized+remapped, last acting [10,2,18]
    pg 2.16c is stuck undersized for 72491.413090, current state active+undersized+remapped, last acting [0,12,18]
    pg 2.16d is stuck undersized for 72491.415706, current state active+undersized+remapped, last acting [16,0,12]
    pg 2.16e is stuck undersized for 72491.411571, current state active+undersized+remapped, last acting [8,2,12]
    pg 2.16f is stuck undersized for 72491.414464, current state active+undersized+remapped, last acting [10,16,18]
    pg 2.170 is stuck undersized for 72491.405008, current state active+undersized+remapped, last acting [18,14,16]
    pg 2.171 is stuck undersized for 72491.411614, current state active+undersized+remapped, last acting [6,0,10]
    pg 2.172 is stuck undersized for 72491.411492, current state active+undersized+remapped, last acting [10,0,14]
    pg 2.173 is stuck undersized for 72491.413998, current state active+undersized+remapped, last acting [10,2,18]
    pg 2.174 is stuck undersized for 72491.406730, current state active+undersized+remapped, last acting [0,12,18]
    pg 2.175 is stuck undersized for 72491.416080, current state active+undersized+remapped, last acting [16,0,12]
    pg 2.176 is stuck undersized for 72491.409336, current state active+undersized+remapped, last acting [8,2,12]
    pg 2.177 is stuck undersized for 72491.415159, current state active+undersized+remapped, last acting [10,16,18]
    pg 3.168 is stuck undersized for 72490.394675, current state active+undersized+remapped, last acting [2,4,6]
    pg 3.169 is stuck undersized for 72490.399679, current state active+undersized+remapped, last acting [6,2,18]
    pg 3.16a is stuck undersized for 72490.381105, current state active+undersized+remapped, last acting [12,2,14]
    pg 3.16b is stuck undersized for 72490.386668, current state active+undersized+remapped, last acting [6,0,18]
    pg 3.16c is stuck undersized for 72490.396841, current state active+undersized+remapped, last acting [0,12,14]
    pg 3.16d is stuck undersized for 72490.393868, current state active+undersized+remapped, last acting [12,10,18]
    pg 3.16e is stuck undersized for 72490.396822, current state active+undersized+remapped, last acting [12,4,6]
    pg 3.16f is stuck undersized for 72490.393344, current state active+undersized+remapped, last acting [6,2,8]
    pg 3.170 is stuck undersized for 72490.391844, current state active+undersized+remapped, last acting [2,4,6]
    pg 3.171 is stuck undersized for 72490.398862, current state active+undersized+remapped, last acting [6,2,18]
    pg 3.172 is stuck undersized for 72490.393152, current state active+undersized+remapped, last acting [12,2,14]
    pg 3.173 is stuck undersized for 72490.398763, current state active+undersized+remapped, last acting [6,0,18]
    pg 3.174 is stuck undersized for 72490.397356, current state active+undersized+remapped, last acting [0,12,14]
    pg 3.175 is stuck undersized for 72490.398161, current state active+undersized+remapped, last acting [12,10,18]
    pg 3.176 is stuck undersized for 72490.393288, current state active+undersized+remapped, last acting [12,4,6]
    pg 3.177 is stuck undersized for 72490.397813, current state active+undersized+remapped, last acting [6,2,8]

Official Documentation: http://docs.ceph.com/docs/master/rados/operations/health-checks/#pg-degraded

The solution

The problem is that the pool redundancy is higher than available, reducing pool redundancy will remove the alert.
THIS MUST BE STUDIED!!

ceph@avmlp-osm-001 ~/ceph-deploy $ sudo ceph osd pool ls detail
pool 1 '.rgw.root' replicated size 4 min_size 2 crush_rule 1 object_hash rjenkins pg_num 500 pgp_num 500 autoscale_mode warn last_change 665 lfor 0/0/220 flags hashpspool stripe_width 0 application rgw
pool 2 'default.rgw.control' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 390 pgp_num 8 pg_num_target 60 pgp_num_target 60 pg_num_pending 389 autoscale_mode warn last_change 668 lfor 0/668/668 flags hashpspool stripe_width 0 application rgw
pool 3 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 390 pgp_num 8 pg_num_target 60 pgp_num_target 60 pg_num_pending 389 autoscale_mode warn last_change 668 lfor 0/668/668 flags hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 60 pgp_num 8 pgp_num_target 60 autoscale_mode warn last_change 269 lfor 0/0/243 flags hashpspool stripe_width 0 application rgw
avmlp-osm-001 /home/ceph # ceph health
HEALTH_WARN Degraded data redundancy: 1329 pgs undersized
avmlp-osm-001 /home/ceph # for i in $(ceph osd pool ls) ; do  ceph osd pool set ${i} size 3 ; done                                                                                                                                                                                     
set pool 1 size to 3
set pool 2 size to 3
set pool 3 size to 3
set pool 4 size to 3
avmlp-osm-001 /home/ceph # ceph health
HEALTH_WARN Reduced data availability: 197 pgs peering; Degraded data redundancy: 196 pgs undersized
avmlp-osm-001 /home/ceph # ceph health
HEALTH_OK
avmlp-osm-001 /home/ceph # ceph osd pool ls detail
pool 1 '.rgw.root' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 500 pgp_num 500 autoscale_mode warn last_change 696 lfor 0/0/220 flags hashpspool stripe_width 0 application rgw
pool 2 'default.rgw.control' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 339 pgp_num 8 pg_num_target 60 pgp_num_target 60 pg_num_pending 338 autoscale_mode warn last_change 880 lfor 0/880/880 flags hashpspool stripe_width 0 application rgw
pool 3 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 339 pgp_num 8 pg_num_target 60 pgp_num_target 60 pg_num_pending 338 autoscale_mode warn last_change 880 lfor 0/880/880 flags hashpspool stripe_width 0 application rgw
pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 60 pgp_num 8 pgp_num_target 60 autoscale_mode warn last_change 699 lfor 0/0/243 flags hashpspool stripe_width 0 application rgw

Maybe also ceph osd repair will work:

INCIDENT #185269 : Degraded data redundancy: 2 pgs degraded, 8 pgs undersized

DON'T use syntax highlight in the following block or will make this page terrible slow!!!

General troubleshoot:

avmlp-osm-001 /home/ceph # ceph -s
  cluster:
    id:     a3a799ce-f1d3-4230-a915-06e988fee767
    health: HEALTH_WARN
            Degraded data redundancy: 2 pgs degraded, 8 pgs undersized
 
  services:
    mon: 6 daemons, quorum avmlp-osm-001,avmlp-osm-002,avmlp-osm-003,avmlp-osm-004,avmlp-osm-005,avmlp-osm-006 (age 5d)
    mgr: avmlp-osm-005.ciberterminal.net(active, since 6w), standbys: avmlp-osm-002.ciberterminal.net, avmlp-osm-006.ciberterminal.net, avmlp-osm-004.ciberterminal.net, avmlp-osm-001.ciberterminal.net, avmlp-osm-003.ciberterminal.net
    mds: cephfs:1 {0=avmlp-osfs-002.ciberterminal.net=up:active} 3 up:standby
    osd: 20 osds: 20 up (since 7w), 20 in (since 12M)
    rgw: 4 daemons active (avmlp-osgw-001.ciberterminal.net, avmlp-osgw-002.ciberterminal.net, avmlp-osgw-003.ciberterminal.net, avmlp-osgw-004.ciberterminal.net)
 
  data:
    pools:   9 pools, 96 pgs
    objects: 1.26M objects, 102 GiB
    usage:   659 GiB used, 39 TiB / 40 TiB avail
    pgs:     88 active+clean
             6  active+undersized
             2  active+undersized+degraded
 
  io:
    client:   11 MiB/s rd, 36 KiB/s wr, 128 op/s rd, 0 op/s wr
 
avmlp-osm-001 /home/ceph # ceph health detail
HEALTH_WARN Degraded data redundancy: 2 pgs degraded, 8 pgs undersized
PG_DEGRADED Degraded data redundancy: 2 pgs degraded, 8 pgs undersized
    pg 14.0 is stuck undersized for 510298.054479, current state active+undersized, last acting [5,12]
    pg 14.1 is stuck undersized for 510298.091712, current state active+undersized, last acting [18,7]
    pg 14.2 is stuck undersized for 510298.007891, current state active+undersized+degraded, last acting [7,18]
    pg 14.3 is stuck undersized for 510298.086409, current state active+undersized, last acting [8,5]
    pg 14.4 is stuck undersized for 510298.054479, current state active+undersized+degraded, last acting [5,18]
    pg 14.5 is stuck undersized for 510298.033776, current state active+undersized, last acting [16,1]
    pg 14.6 is stuck undersized for 510298.086409, current state active+undersized, last acting [8,3]
    pg 14.7 is stuck undersized for 510298.091649, current state active+undersized, last acting [18,3]
    

Why pg 14 is the only one that is failing???

avmlp-osm-001 /home/ceph # ceph tell 14.0 query | jq .
{
  "state": "active+undersized",
  "snap_trimq": "[]",
  "snap_trimq_len": 0,
  "epoch": 3629,
  "up": [
    5,
    12
  ],
  "acting": [
    5,
    12
  ],
  "acting_recovery_backfill": [
    "5",
    "12"
  ],
  "info": {
    "pgid": "14.0",
    "last_update": "0'0",
    "last_complete": "0'0",
    "log_tail": "0'0",
    "last_user_version": 0,
    "last_backfill": "MAX",
    "last_backfill_bitwise": 0,
    "purged_snaps": [],
    "history": {
      "epoch_created": 3627,
      "epoch_pool_created": 3627,
      "last_epoch_started": 3628,
      "last_interval_started": 3627,
      "last_epoch_clean": 0,
      "last_interval_clean": 0,
      "last_epoch_split": 0,
      "last_epoch_marked_full": 0,
      "same_up_since": 3627,
      "same_interval_since": 3627,
      "same_primary_since": 3627,
      "last_scrub": "0'0",
      "last_scrub_stamp": "2020-08-11 11:50:37.135596",
      "last_deep_scrub": "0'0",
      "last_deep_scrub_stamp": "2020-08-11 11:50:37.135596",
      "last_clean_scrub_stamp": "2020-08-11 11:50:37.135596"
    },
    "stats": {
      "version": "0'0",
      "reported_seq": "9",
      "reported_epoch": "3629",
      "state": "active+undersized",
      "last_fresh": "2020-08-11 11:50:39.473996",
      "last_change": "2020-08-11 11:50:38.238074",
      "last_active": "2020-08-11 11:50:39.473996",
      "last_peered": "2020-08-11 11:50:39.473996",
      "last_clean": "2020-08-11 11:50:37.135596",
      "last_became_active": "2020-08-11 11:50:38.238074",
      "last_became_peered": "2020-08-11 11:50:38.238074",
      "last_unstale": "2020-08-11 11:50:39.473996",
      "last_undegraded": "2020-08-11 11:50:39.473996",
      "last_fullsized": "2020-08-11 11:50:38.233209",
      "mapping_epoch": 3627,
      "log_start": "0'0",
      "ondisk_log_start": "0'0",
      "created": 3627,
      "last_epoch_clean": 0,
      "parent": "0.0",
      "parent_split_bits": 0,
      "last_scrub": "0'0",
      "last_scrub_stamp": "2020-08-11 11:50:37.135596",
      "last_deep_scrub": "0'0",
      "last_deep_scrub_stamp": "2020-08-11 11:50:37.135596",
      "last_clean_scrub_stamp": "2020-08-11 11:50:37.135596",
      "log_size": 0,
      "ondisk_log_size": 0,
      "stats_invalid": false,
      "dirty_stats_invalid": false,
      "omap_stats_invalid": false,
      "hitset_stats_invalid": false,
      "hitset_bytes_stats_invalid": false,
      "pin_stats_invalid": false,
      "manifest_stats_invalid": false,
      "snaptrimq_len": 0,
      "stat_sum": {
        "num_bytes": 0,
        "num_objects": 0,
        "num_object_clones": 0,
        "num_object_copies": 0,
        "num_objects_missing_on_primary": 0,
        "num_objects_missing": 0,
        "num_objects_degraded": 0,
        "num_objects_misplaced": 0,
        "num_objects_unfound": 0,
        "num_objects_dirty": 0,
        "num_whiteouts": 0,
        "num_read": 0,
        "num_read_kb": 0,
        "num_write": 0,
        "num_write_kb": 0,
        "num_scrub_errors": 0,
        "num_shallow_scrub_errors": 0,
        "num_deep_scrub_errors": 0,
        "num_objects_recovered": 0,
        "num_bytes_recovered": 0,
        "num_keys_recovered": 0,
        "num_objects_omap": 0,
        "num_objects_hit_set_archive": 0,
        "num_bytes_hit_set_archive": 0,
        "num_flush": 0,
        "num_flush_kb": 0,
        "num_evict": 0,
        "num_evict_kb": 0,
        "num_promote": 0,
        "num_flush_mode_high": 0,
        "num_flush_mode_low": 0,
        "num_evict_mode_some": 0,
        "num_evict_mode_full": 0,
        "num_objects_pinned": 0,
        "num_legacy_snapsets": 0,
        "num_large_omap_objects": 0,
        "num_objects_manifest": 0,
        "num_omap_bytes": 0,
        "num_omap_keys": 0,
        "num_objects_repaired": 0
      },
      "up": [
        5,
        12
      ],
      "acting": [
        5,
        12
      ],
      "avail_no_missing": [
        "5",
        "12"
      ],
      "object_location_counts": [],
      "blocked_by": [],
      "up_primary": 5,
      "acting_primary": 5,
      "purged_snaps": []
    },
    "empty": 1,
    "dne": 0,
    "incomplete": 0,
    "last_epoch_started": 3628,
    "hit_set_history": {
      "current_last_update": "0'0",
      "history": []
    }
  },
  "peer_info": [
    {
      "peer": "12",
      "pgid": "14.0",
      "last_update": "0'0",
      "last_complete": "0'0",
      "log_tail": "0'0",
      "last_user_version": 0,
      "last_backfill": "MAX",
      "last_backfill_bitwise": 0,
      "purged_snaps": [],
      "history": {
        "epoch_created": 3627,
        "epoch_pool_created": 3627,
        "last_epoch_started": 3628,
        "last_interval_started": 3627,
        "last_epoch_clean": 0,
        "last_interval_clean": 0,
        "last_epoch_split": 0,
        "last_epoch_marked_full": 0,
        "same_up_since": 0,
        "same_interval_since": 0,
        "same_primary_since": 0,
        "last_scrub": "0'0",
        "last_scrub_stamp": "2020-08-11 11:50:37.135596",
        "last_deep_scrub": "0'0",
        "last_deep_scrub_stamp": "2020-08-11 11:50:37.135596",
        "last_clean_scrub_stamp": "2020-08-11 11:50:37.135596"
      },
      "stats": {
        "version": "0'0",
        "reported_seq": "0",
        "reported_epoch": "0",
        "state": "unknown",
        "last_fresh": "0.000000",
        "last_change": "0.000000",
        "last_active": "0.000000",
        "last_peered": "0.000000",
        "last_clean": "0.000000",
        "last_became_active": "0.000000",
        "last_became_peered": "0.000000",
        "last_unstale": "0.000000",
        "last_undegraded": "0.000000",
        "last_fullsized": "0.000000",
        "mapping_epoch": 0,
        "log_start": "0'0",
        "ondisk_log_start": "0'0",
        "created": 0,
        "last_epoch_clean": 0,
        "parent": "0.0",
        "parent_split_bits": 0,
        "last_scrub": "0'0",
        "last_scrub_stamp": "0.000000",
        "last_deep_scrub": "0'0",
        "last_deep_scrub_stamp": "0.000000",
        "last_clean_scrub_stamp": "0.000000",
        "log_size": 0,
        "ondisk_log_size": 0,
        "stats_invalid": false,
        "dirty_stats_invalid": false,
        "omap_stats_invalid": false,
        "hitset_stats_invalid": false,
        "hitset_bytes_stats_invalid": false,
        "pin_stats_invalid": false,
        "manifest_stats_invalid": false,
        "snaptrimq_len": 0,
        "stat_sum": {
          "num_bytes": 0,
          "num_objects": 0,
          "num_object_clones": 0,
          "num_object_copies": 0,
          "num_objects_missing_on_primary": 0,
          "num_objects_missing": 0,
          "num_objects_degraded": 0,
          "num_objects_misplaced": 0,
          "num_objects_unfound": 0,
          "num_objects_dirty": 0,
          "num_whiteouts": 0,
          "num_read": 0,
          "num_read_kb": 0,
          "num_write": 0,
          "num_write_kb": 0,
          "num_scrub_errors": 0,
          "num_shallow_scrub_errors": 0,
          "num_deep_scrub_errors": 0,
          "num_objects_recovered": 0,
          "num_bytes_recovered": 0,
          "num_keys_recovered": 0,
          "num_objects_omap": 0,
          "num_objects_hit_set_archive": 0,
          "num_bytes_hit_set_archive": 0,
          "num_flush": 0,
          "num_flush_kb": 0,
          "num_evict": 0,
          "num_evict_kb": 0,
          "num_promote": 0,
          "num_flush_mode_high": 0,
          "num_flush_mode_low": 0,
          "num_evict_mode_some": 0,
          "num_evict_mode_full": 0,
          "num_objects_pinned": 0,
          "num_legacy_snapsets": 0,
          "num_large_omap_objects": 0,
          "num_objects_manifest": 0,
          "num_omap_bytes": 0,
          "num_omap_keys": 0,
          "num_objects_repaired": 0
        },
        "up": [],
        "acting": [],
        "avail_no_missing": [],
        "object_location_counts": [],
        "blocked_by": [],
        "up_primary": -1,
        "acting_primary": -1,
        "purged_snaps": []
      },
      "empty": 1,
      "dne": 0,
      "incomplete": 0,
      "last_epoch_started": 3628,
      "hit_set_history": {
        "current_last_update": "0'0",
        "history": []
      }
    }
  ],
  "recovery_state": [
    {
      "name": "Started/Primary/Active",
      "enter_time": "2020-08-11 11:50:38.233290",
      "might_have_unfound": [],
      "recovery_progress": {
        "backfill_targets": [],
        "waiting_on_backfill": [],
        "last_backfill_started": "MIN",
        "backfill_info": {
          "begin": "MIN",
          "end": "MIN",
          "objects": []
        },
        "peer_backfill_info": [],
        "backfills_in_flight": [],
        "recovering": [],
        "pg_backend": {
          "pull_from_peer": [],
          "pushing": []
        }
      },
      "scrub": {
        "scrubber.epoch_start": "0",
        "scrubber.active": false,
        "scrubber.state": "INACTIVE",
        "scrubber.start": "MIN",
        "scrubber.end": "MIN",
        "scrubber.max_end": "MIN",
        "scrubber.subset_last_update": "0'0",
        "scrubber.deep": false,
        "scrubber.waiting_on_whom": []
      }
    },
    {
      "name": "Started",
      "enter_time": "2020-08-11 11:50:37.502984"
    }
  ],
  "agent_state": {}
}

pg 14 does not have data nor activity!!!!

avmlp-osm-001 /home/ceph # ceph pg dump
version 2063560
stamp 2020-08-17 09:46:06.557196
last_osdmap_epoch 0
last_pg_scan 0
PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES      OMAP_BYTES* OMAP_KEYS* LOG  DISK_LOG STATE                      STATE_STAMP                VERSION      REPORTED       UP            UP_PRIMARY ACTING        ACTING_PRIMARY LAST_SCRUB   SCRUB_STAMP                LAST_DEEP_SCRUB DEEP_SCRUB_STAMP           SNAPTRIMQ_LEN 
12.1f     37234                  0        0         0       0 2448717330           0          0 3074     3074               active+clean 2020-08-16 23:01:57.976378  3629'342755    3629:445975  [4,10,19,17]          4  [4,10,19,17]              4  3629'342725 2020-08-16 23:01:57.976326     3629'342409 2020-08-14 13:40:26.247849             0 
12.1e     36870                  0        0         0       0 2452671420           0          0 3074     3074               active+clean 2020-08-16 05:41:59.725318  3629'342325    3629:605224    [2,14,5,1]          2    [2,14,5,1]              2  3629'342214 2020-08-16 05:41:59.725253     3629'342075 2020-08-15 01:28:05.207465             0 
12.1d     36549                  0        0         0       0 2255784538           0          0 3065     3065               active+clean 2020-08-17 03:55:29.331054  3629'342855    3629:507767     [6,0,7,5]          6     [6,0,7,5]              6  3629'342842 2020-08-17 03:55:29.330992     3629'342585 2020-08-14 21:29:33.393053             0 
12.1c     36839                  0        0         0       0 2492477640           0          0 3013     3013               active+clean 2020-08-17 07:42:13.522039  3629'343720    3629:445046     [3,1,6,0]          3     [3,1,6,0]              3  3629'343709 2020-08-17 07:42:13.521977     3629'343568 2020-08-15 22:22:48.911933             0 
12.1b     36940                  0        0         0       0 2470626204           0          0 3048     3048               active+clean 2020-08-16 13:35:08.123034  3629'344727    3629:501600   [1,13,12,6]          1   [1,13,12,6]              1  3629'344655 2020-08-16 13:35:08.122944     3629'344655 2020-08-16 13:35:08.122944             0 
12.1a     37215                  0        0         0       0 2632273586           0          0 3070     3070               active+clean 2020-08-17 07:44:07.798470  3629'351695    3629:494814   [5,15,14,6]          5   [5,15,14,6]              5  3629'351679 2020-08-17 07:44:07.798401     3629'351587 2020-08-16 04:51:26.455475             0 
12.19     37311                  0        0         0       0 2597837981           0          0 3085     3085               active+clean 2020-08-16 07:01:40.593214  3629'346755    3629:474433  [12,16,17,1]         12  [12,16,17,1]             12  3629'346674 2020-08-16 07:01:40.593159     3629'346466 2020-08-15 04:48:07.588319             0 
12.18     36731                  0        0         0       0 2458805578           0          0 3092     3092               active+clean 2020-08-16 03:13:11.863075  3629'342802    3629:468666   [14,2,13,3]         14   [14,2,13,3]             14  3629'342683 2020-08-16 03:13:11.863020     3629'342449 2020-08-14 19:08:54.971752             0 
12.17     37234                  0        0         0       0 2411367145           0          0 3084     3084               active+clean 2020-08-17 08:47:51.571373  3629'344472    3629:496481    [5,9,8,14]          5    [5,9,8,14]              5  3629'344468 2020-08-17 08:47:51.571280     3629'343706 2020-08-13 01:47:55.688179             0 
12.16     37146                  0        0         0       0 2517689370           0          0 3090     3090               active+clean 2020-08-16 13:44:35.341646  3629'343210    3629:472908     [7,9,8,4]          7     [7,9,8,4]              7  3629'343143 2020-08-16 13:44:35.341577     3629'342810 2020-08-14 00:43:42.538689             0 
12.15     36841                  0        0         0       0 2550933263           0          0 3040     3040               active+clean 2020-08-17 03:52:55.184968  3629'345294    3629:502420   [1,15,18,4]          1   [1,15,18,4]              1  3629'345270 2020-08-17 03:52:55.184909     3629'345169 2020-08-16 02:39:11.303006             0 
12.14     36936                  0        0         0       0 2436184387           0          0 3076     3076               active+clean 2020-08-16 15:16:55.057652  3629'334510    3629:491995  [8,12,17,15]          8  [8,12,17,15]              8  3629'334418 2020-08-16 15:16:55.057592     3629'334259 2020-08-15 08:21:42.290408             0 
12.13     36989                  0        0         0       0 2452289072           0          0 3095     3095               active+clean 2020-08-17 06:11:39.821708  3629'330241    3629:459043     [4,8,7,5]          4     [4,8,7,5]              4  3629'330217 2020-08-17 06:11:39.821627     3629'330217 2020-08-17 06:11:39.821627             0 
12.12     36937                  0        0         0       0 2454977301           0          0 3057     3057               active+clean 2020-08-16 16:35:25.451333  3629'344686    3629:473960  [17,13,0,10]         17  [17,13,0,10]             17  3629'344608 2020-08-16 16:35:25.451262     3629'344608 2020-08-16 16:35:25.451262             0 
12.11     37165                  0        0         0       0 2639030037           0          0 3010     3010               active+clean 2020-08-16 12:21:25.240115  3629'344750    3629:504301   [2,18,1,15]          2   [2,18,1,15]              2  3629'344663 2020-08-16 12:21:25.239991     3629'343787 2020-08-12 10:45:52.034024             0 
12.10     37063                  0        0         0       0 2584770038           0          0 3046     3046               active+clean 2020-08-17 08:51:07.093579  3629'340698    3629:498656  [5,19,12,10]          5  [5,19,12,10]              5  3629'340689 2020-08-17 08:51:07.093497     3629'339693 2020-08-12 03:45:50.319811             0 
12.a      37207                  0        0         0       0 2594481924           0          0 3094     3094               active+clean 2020-08-16 16:26:16.525831  3629'345493    3629:347039  [12,0,17,19]         12  [12,0,17,19]             12  3629'345385 2020-08-16 16:26:16.525778     3629'345385 2020-08-16 16:26:16.525778             0 
7.1          11                  0        0         0       0       1560        1416          8 3076     3076               active+clean 2020-08-16 13:47:36.599722    3620'6878    3629:438065    [6,14,1,3]          6    [6,14,1,3]              6    3620'6878 2020-08-16 13:47:36.599648       3620'6878 2020-08-16 13:47:36.599648             0 
5.3           1                  0        0         0       0        805           0          0    1        1               active+clean 2020-08-16 13:52:05.527723       3605'3      3629:4525   [3,13,0,18]          3   [3,13,0,18]              3       3605'3 2020-08-16 13:52:05.527692          3605'3 2020-08-13 05:28:18.436690             0 
6.0           1                  0        0         0       0          0           0          0  346      346               active+clean 2020-08-16 12:20:48.361592     3629'535  3629:26442589   [4,14,9,17]          4   [4,14,9,17]              4     3629'535 2020-08-16 12:20:48.361537        3629'535 2020-08-16 12:20:48.361537             0 
12.b      36624                  0        0         0       0 2542103794           0          0 3055     3055               active+clean 2020-08-17 06:38:51.532937  3629'337923    3629:353370     [3,5,0,4]          3     [3,5,0,4]              3  3629'337893 2020-08-17 06:38:51.532865     3629'337089 2020-08-12 13:17:48.634712             0 
7.0          11                  0        0         0       0       1658        2912         16  975      975               active+clean 2020-08-16 09:04:51.825199     3629'976    3629:272919  [16,14,7,17]         16  [16,14,7,17]             16     3629'972 2020-08-16 09:04:51.825161        3629'972 2020-08-16 09:04:51.825161             0 
5.2           0                  0        0         0       0          0           0          0    0        0               active+clean 2020-08-16 09:24:09.871395          0'0      3629:4488  [1,15,18,10]          1  [1,15,18,10]              1          0'0 2020-08-16 09:24:09.871348             0'0 2020-08-16 09:24:09.871348             0 
6.1           1                  0        0         0       0          0           0          0  341      341               active+clean 2020-08-17 09:32:47.922139     3629'613  3629:26496283   [6,12,13,3]          6   [6,12,13,3]              6     3629'613 2020-08-17 09:32:47.922094        3629'613 2020-08-16 01:49:21.114037             0 
12.8      36745                  0        0         0       0 2453736305           0          0 3046     3046               active+clean 2020-08-16 08:20:55.871458  3629'341424    3629:352763  [19,5,14,12]         19  [19,5,14,12]             19  3629'341323 2020-08-16 08:20:55.871363     3626'340243 2020-08-11 00:28:58.843200             0 
7.3           6                  0        0         0       0       1730           0          0   34       34               active+clean 2020-08-16 18:17:48.970705      3626'35    3629:226711  [17,15,6,12]         17  [17,15,6,12]             17      3626'35 2020-08-16 18:17:48.970621         3626'35 2020-08-14 08:40:31.413876             0 
6.2           0                  0        0         0       0          0           0          0    4        4               active+clean 2020-08-16 14:09:46.658487     2078'251     3629:58666  [10,6,13,11]         10  [10,6,13,11]             10     2078'251 2020-08-16 14:09:46.658432        2078'251 2020-08-11 16:12:50.795649             0 
5.1           0                  0        0         0       0          0           0          0    0        0               active+clean 2020-08-16 19:33:01.145513          0'0      3629:4511   [19,7,16,0]         19   [19,7,16,0]             19          0'0 2020-08-16 19:33:01.145470             0'0 2020-08-16 19:33:01.145470             0 
12.9      36762                  0        0         0       0 2574357560           0          0 3088     3088               active+clean 2020-08-17 08:13:42.940483  3629'342464    3629:357530   [15,3,6,16]         15   [15,3,6,16]             15  3629'342450 2020-08-17 08:13:42.940428     3629'342179 2020-08-15 03:51:12.708397             0 
7.2           7                  0        0         0       0       1857        1532          8 1858     1858               active+clean 2020-08-16 21:11:17.394904    3629'1862    3629:357192     [3,7,0,2]          3     [3,7,0,2]              3    3629'1847 2020-08-16 21:11:17.394835       3629'1481 2020-08-11 22:28:46.467550             0 
6.3           1                  0        0         0       0          0           0          0  346      346               active+clean 2020-08-16 23:47:33.943659     3629'571  3629:26442615 [19,13,18,14]         19 [19,13,18,14]             19     3629'571 2020-08-16 23:47:33.943612        3629'571 2020-08-13 13:20:52.435651             0 
5.0           1                  0        0         0       0        348           0          0    1        1               active+clean 2020-08-16 15:09:36.160089       3605'2      3629:4694     [7,9,4,0]          7     [7,9,4,0]              7       3605'2 2020-08-16 15:09:36.159945          3605'2 2020-08-16 15:09:36.159945             0 
12.d      36881                  0        0         0       0 2473024184           0          0 3060     3060               active+clean 2020-08-16 22:07:39.858423  3629'343047    3629:358430    [9,7,10,6]          9    [9,7,10,6]              9  3629'343010 2020-08-16 22:07:39.858356     3626'341844 2020-08-10 15:03:18.641927             0 
6.7           1                  0        0         0       0          0           0          0  354      354               active+clean 2020-08-16 07:35:25.498839     3629'579  3629:26322313    [14,4,9,5]         14    [14,4,9,5]             14     3629'579 2020-08-16 07:35:25.498808        3629'579 2020-08-16 07:35:25.498808             0 
5.4           0                  0        0         0       0          0           0          0    0        0               active+clean 2020-08-17 06:52:48.025888        169'1      3629:2390    [12,0,5,9]         12    [12,0,5,9]             12        169'1 2020-08-17 06:52:48.025854           169'1 2020-08-13 17:30:10.364425             0 
7.6          15                  0        0         0       0       3862           0          0   48       48               active+clean 2020-08-16 06:56:31.300185      3629'52    3629:913522   [13,1,4,16]         13   [13,1,4,16]             13      3629'52 2020-08-16 06:56:31.300134         3629'52 2020-08-13 21:44:05.147193             0 
12.c      37146                  0        0         0       0 2542888837           0          0 3042     3042               active+clean 2020-08-17 01:47:38.880451  3629'341293    3629:356990    [17,9,0,6]         17    [17,9,0,6]             17  3629'341259 2020-08-17 01:47:38.880347     3629'340318 2020-08-11 18:13:24.694080             0 
6.6           1                  0        0         0       0          0           0          0  334      334               active+clean 2020-08-16 09:44:42.924292     3629'581  3629:26238773  [12,16,13,1]         12  [12,16,13,1]             12     3629'581 2020-08-16 09:44:42.924234        3629'581 2020-08-12 14:42:47.482884             0 
5.5           0                  0        0         0       0          0           0          0    0        0               active+clean 2020-08-16 23:24:09.575998          0'0      3629:1051    [15,5,0,4]         15    [15,5,0,4]             15          0'0 2020-08-16 23:24:09.575954             0'0 2020-08-16 23:24:09.575954             0 
7.7           6                  0        0         0       0        427         704          4 3075     3075               active+clean 2020-08-16 21:21:50.448714   3629'18477 3629:135290250  [11,17,14,4]         11  [11,17,14,4]             11   3629'18424 2020-08-16 21:21:50.448676      3626'17489 2020-08-10 20:15:44.650232             0 
12.f      37170                  0        0         0       0 2421662736           0          0 3088     3088               active+clean 2020-08-16 22:46:18.321473  3629'339853    3629:355474    [12,6,7,1]         12    [12,6,7,1]             12  3629'339826 2020-08-16 22:46:18.321426     3629'339234 2020-08-13 10:34:11.712596             0 
7.4           6                  0        0         0       0       1066           0          0   27       27               active+clean 2020-08-16 02:18:10.503010      3626'28    3629:320651   [3,7,16,18]          3   [3,7,16,18]              3      3626'28 2020-08-16 02:18:10.502933         3626'28 2020-08-16 02:18:10.502933             0 
6.5           2                  0        0         0       0          0           0          0  668      668               active+clean 2020-08-16 16:40:36.754757     3629'940  3629:52661388   [19,1,4,18]         19   [19,1,4,18]             19     3629'940 2020-08-16 16:40:36.754710        3626'904 2020-08-10 04:26:29.538486             0 
5.6           0                  0        0         0       0          0           0          0    0        0               active+clean 2020-08-16 02:46:27.887880          0'0      3629:4516  [18,14,13,5]         18  [18,14,13,5]             18          0'0 2020-08-16 02:46:27.887837             0'0 2020-08-10 06:09:08.174831             0 
12.e      36977                  0        0         0       0 2513728716           0          0 3089     3089               active+clean 2020-08-16 10:19:08.917759  3629'342525    3629:471605  [16,18,7,11]         16  [16,18,7,11]             16  3629'342435 2020-08-16 10:19:08.917672     3626'341192 2020-08-10 05:51:27.175188             0 
7.5          14                  0        0         0       0       2214        4484         24 3004     3004               active+clean 2020-08-16 15:54:32.903243    3629'7806    3629:239698  [15,19,8,12]         15  [15,19,8,12]             15    3629'7789 2020-08-16 15:54:32.903170       3629'7641 2020-08-14 02:43:54.992932             0 
6.4           1                  0        0         0       0          0           0          0  354      354               active+clean 2020-08-16 12:49:44.792839     3629'543  3629:26332861   [17,9,14,8]         17   [17,9,14,8]             17     3629'543 2020-08-16 12:49:44.792781        3626'525 2020-08-11 09:11:27.786481             0 
5.7           3                  0        0         0       0        897           0          0    3        3               active+clean 2020-08-17 00:25:41.809330       3605'5      3629:3491  [5,15,14,18]          5  [5,15,14,18]              5       3605'5 2020-08-17 00:25:41.809266          3605'5 2020-08-13 05:37:52.184231             0 
14.6          0                  0        0         0       0          0           0          0    0        0          active+undersized 2020-08-11 11:50:38.208103          0'0         3628:8         [8,3]          8         [8,3]              8          0'0 2020-08-11 11:50:37.135596             0'0 2020-08-11 11:50:37.135596             0 
13.5        580                  0        0         0       0   20971680   292948688     634088 3091     3091               active+clean 2020-08-17 02:30:59.655119  3629'587509    3629:615647  [11,7,12,10]         11  [11,7,12,10]             11  3629'587291 2020-08-17 02:30:59.655047     3629'583707 2020-08-14 15:10:59.726903             0 
12.4      37167                  0        0         0       0 2532685297           0          0 3062     3062               active+clean 2020-08-17 07:58:58.470571  3629'344228    3629:360198  [9,19,18,14]          9  [9,19,18,14]              9  3629'344222 2020-08-17 07:58:58.470487     3626'343061 2020-08-11 07:51:26.283423             0 
8.0          26                  0        0         0       0          0           0          0 3098     3098               active+clean 2020-08-16 13:17:58.821382 3629'7807401  3629:11594756  [15,1,18,10]         15  [15,1,18,10]             15 3629'7789704 2020-08-16 13:17:58.821333    3629'7766649 2020-08-15 10:37:00.441663             0 
10.2       8709                  0        0         0       0 3677669925           0          0 3066     3066               active+clean 2020-08-16 12:00:52.962103 3629'5527685  3629:33958241    [8,6,9,17]          8    [8,6,9,17]              8 3629'5527659 2020-08-16 12:00:52.962034    3629'5527659 2020-08-16 12:00:52.962034             0 
11.3          0                  0        0         0       0          0           0          0 3011     3011               active+clean 2020-08-16 18:46:04.571928 3605'7544511   3629:7551165    [4,2,5,19]          4    [4,2,5,19]              4 3605'7544511 2020-08-16 18:46:04.571891    3605'7544511 2020-08-16 18:46:04.571891             0 
14.7          0                  0        0         0       0          0           0          0    0        0          active+undersized 2020-08-11 11:50:38.205413          0'0         3628:8        [18,3]         18        [18,3]             18          0'0 2020-08-11 11:50:37.135596             0'0 2020-08-11 11:50:37.135596             0 
13.4        553                  0        0         0       0   20972154   240751960     521108 3084     3084               active+clean 2020-08-16 08:22:47.112467  3629'852149    3629:888128  [9,11,18,16]          9  [9,11,18,16]              9  3629'850960 2020-08-16 08:22:47.112414     3629'850960 2020-08-16 08:22:47.112414             0 
12.5      37005                  0        0         0       0 2461994487           0          0 3066     3066               active+clean 2020-08-16 05:29:22.304563  3629'343950    3629:359515   [17,11,8,2]         17   [17,11,8,2]             17  3629'343834 2020-08-16 05:29:22.304452     3629'343085 2020-08-12 21:39:51.184699             0 
8.1          32                  0        0         0       0          0           0          0 3080     3080               active+clean 2020-08-16 18:52:17.365345 3629'9099181  3629:13533326  [18,10,7,13]         18  [18,10,7,13]             18 3629'9084191 2020-08-16 18:52:17.365270    3626'8917520 2020-08-09 21:24:16.115253             0 
10.3       8686                  0        0         0       0 3736113818           0          0 3051     3051               active+clean 2020-08-16 14:27:13.422714 3629'4468071  3629:27962874   [12,6,13,1]         12   [12,6,13,1]             12 3629'4468045 2020-08-16 14:27:13.422638    3629'4467930 2020-08-14 10:56:15.273485             0 
11.2          4                  0        0         0       0          0      977588       3472 3068     3068               active+clean 2020-08-16 07:39:36.931627 3629'3773773   3629:3781491  [9,11,10,12]          9  [9,11,10,12]              9 3629'3773611 2020-08-16 07:39:36.931571    3629'3773091 2020-08-13 19:56:13.839973             0 
14.4          0                  0        0         0       0          0           0          0    7        7 active+undersized+degraded 2020-08-11 11:52:11.578155       3629'7        3629:21        [5,18]          5        [5,18]              5          0'0 2020-08-11 11:50:37.135596             0'0 2020-08-11 11:50:37.135596             0 
13.7        559                  0        0         0       0   25166390   290987928     629844 3070     3070               active+clean 2020-08-16 00:16:42.372445  3629'373238    3629:428637   [16,2,17,1]         16   [16,2,17,1]             16  3629'372495 2020-08-16 00:16:42.372384     3629'370804 2020-08-13 15:45:28.525122             0 
12.6      37092                  0        0         0       0 2326680636           0          0 3021     3021               active+clean 2020-08-16 17:24:46.808260  3629'342824    3629:358166   [0,14,7,15]          0   [0,14,7,15]              0  3629'342734 2020-08-16 17:24:46.808204     3629'342111 2020-08-13 01:50:02.075255             0 
8.2          26                  0        0         0       0          0           0          0 3100     3100               active+clean 2020-08-16 06:06:19.967891 3629'8514055  3629:12656747   [9,11,10,0]          9   [9,11,10,0]              9 3629'8487923 2020-08-16 06:06:19.967832    3629'8407817 2020-08-12 17:11:15.659708             0 
10.0       8727                  0        0         0       0 3670473841           0          0 3059     3059               active+clean 2020-08-16 23:18:06.938023 3629'4319994  3629:27876556   [17,9,4,14]         17   [17,9,4,14]             17 3629'4319978 2020-08-16 23:18:06.937879    3626'4319557 2020-08-10 19:02:28.103491             0 
11.1          5                  0        0         0       0          0      842872       2620 3090     3090               active+clean 2020-08-16 09:24:39.030517 3629'7553373   3629:7563678  [13,15,16,4]         13  [13,15,16,4]             13 3629'7553355 2020-08-16 09:24:39.030471    3629'7553347 2020-08-15 05:17:56.146025             0 
14.5          0                  0        0         0       0          0           0          0    0        0          active+undersized 2020-08-11 11:50:38.299784          0'0         3628:8        [16,1]         16        [16,1]             16          0'0 2020-08-11 11:50:37.135596             0'0 2020-08-11 11:50:37.135596             0 
13.6        603                  0        0         0       0   29360128   264796356     579740 3056     3056               active+clean 2020-08-16 22:00:12.848404 3629'1228207   3629:1267959   [10,6,15,9]         10   [10,6,15,9]             10 3629'1227293 2020-08-16 22:00:12.848337    3629'1220689 2020-08-14 07:04:49.213009             0 
12.7      36975                  0        0         0       0 2496638778           0          0 3095     3095               active+clean 2020-08-16 11:28:41.495579  3629'340113    3629:352605 [17,19,12,18]         17 [17,19,12,18]             17  3629'339994 2020-08-16 11:28:41.495468     3629'339838 2020-08-15 10:54:22.325510             0 
8.3          22                  0        0         0       0          0           0          0 3026     3026               active+clean 2020-08-16 08:10:35.990042 3629'6260906   3629:9249769     [5,7,8,0]          5     [5,7,8,0]              5 3629'6243224 2020-08-16 08:10:35.989978    3629'6224128 2020-08-15 04:47:58.354067             0 
10.1       8733                  0        0         0       0 3662648021           0          0 3077     3077               active+clean 2020-08-16 07:56:03.255010 3629'5147909  3629:31740280   [8,12,17,3]          8   [8,12,17,3]              8 3629'5147877 2020-08-16 07:56:03.254946    3629'5147596 2020-08-12 14:38:09.703348             0 
11.0          1                  0        0         0       0          0      175636        484  264      264               active+clean 2020-08-16 10:55:15.816476     3626'264      3629:5205   [5,13,0,10]          5   [5,13,0,10]              5     3626'264 2020-08-16 10:55:15.816425        3626'264 2020-08-11 19:47:25.967648             0 
14.2          0                  0        0         0       0          0           0          0    7        7 active+undersized+degraded 2020-08-11 11:50:38.288292       3629'7        3629:21        [7,18]          7        [7,18]              7          0'0 2020-08-11 11:50:37.135596             0'0 2020-08-11 11:50:37.135596             0 
13.1        578                  0        0         0       0   41943040   332682568     720092 3055     3055               active+clean 2020-08-16 09:55:04.810989  3629'366770    3629:422291    [2,8,17,9]          2    [2,8,17,9]              2  3629'366748 2020-08-16 09:55:04.810883     3629'360522 2020-08-12 11:49:08.440021             0 
12.0      36724                  0        0         0       0 2658760797           0          0 3019     3019               active+clean 2020-08-16 16:46:26.833362  3629'338198    3629:353649     [1,3,8,0]          1     [1,3,8,0]              1  3629'338111 2020-08-16 16:46:26.833264     3626'336976 2020-08-10 18:00:59.920643             0 
11.7          0                  0        0         0       0          0           0          0 3041     3041               active+clean 2020-08-16 13:08:43.122360 3611'3775141   3629:3777308   [12,6,19,9]         12   [12,6,19,9]             12 3611'3775141 2020-08-16 13:08:43.122319    3611'3775141 2020-08-16 13:08:43.122319             0 
8.4          29                  0        0         0       0          0           0          0 3026     3026               active+clean 2020-08-17 09:38:30.847123 3629'9980713  3629:14831818     [8,4,5,7]          8     [8,4,5,7]              8 3629'9980537 2020-08-17 09:38:30.847075    3629'9913610 2020-08-14 20:58:23.413276             0 
10.6       8722                  0        0         0       0 3650908162           0          0 3041     3041               active+clean 2020-08-16 03:36:55.650758 3629'5111161  3629:33506602 [13,19,16,12]         13 [13,19,16,12]             13 3629'5111126 2020-08-16 03:36:55.650689    3629'5110776 2020-08-11 22:33:13.472993             0 
14.3          0                  0        0         0       0          0           0          0    0        0          active+undersized 2020-08-11 11:50:38.240473          0'0         3628:8         [8,5]          8         [8,5]              8          0'0 2020-08-11 11:50:37.135596             0'0 2020-08-11 11:50:37.135596             0 
13.0        581                  0        0         0       0   50331648   297297128     643500 3089     3089               active+clean 2020-08-16 22:22:54.073570  3629'332670    3629:358920  [17,19,2,12]         17  [17,19,2,12]             17  3629'332663 2020-08-16 22:22:54.073482     3629'331971 2020-08-14 05:44:32.910314             0 
12.1      36559                  0        0         0       0 2693058955           0          0 3042     3042               active+clean 2020-08-16 19:50:56.588358  3629'340741    3629:356295   [6,0,17,13]          6   [6,0,17,13]              6  3629'340666 2020-08-16 19:50:56.588301     3629'340168 2020-08-13 07:29:49.033271             0 
11.6          2                  0        0         0       0          0    16804608      59704 3067     3067               active+clean 2020-08-16 15:16:17.271048 3629'7579572   3629:7600332   [18,4,7,15]         18   [18,4,7,15]             18 3629'7579572 2020-08-16 15:16:17.271006    3629'7579526 2020-08-15 13:57:43.665912             0 
8.5          25                  0        0         0       0          0           0          0 3088     3088               active+clean 2020-08-16 19:36:31.891091 3629'8629693  3629:12815670   [2,8,19,15]          2   [2,8,19,15]              2 3629'8616157 2020-08-16 19:36:31.891044    3629'8501363 2020-08-11 19:35:10.376406             0 
10.7       8623                  0        0         0       0 3593613124           0          0 3053     3053               active+clean 2020-08-16 06:02:32.648106 3629'4433774  3629:25394677  [12,6,13,15]         12  [12,6,13,15]             12 3629'4433731 2020-08-16 06:02:32.648055    3629'4433683 2020-08-15 00:48:32.278486             0 
14.0          0                  0        0         0       0          0           0          0    0        0          active+undersized 2020-08-11 11:50:38.238074          0'0         3628:8        [5,12]          5        [5,12]              5          0'0 2020-08-11 11:50:37.135596             0'0 2020-08-11 11:50:37.135596             0 
13.3        606                  0        0         0       0   34047265   207078008     448224 3050     3050               active+clean 2020-08-17 04:22:26.814711  3629'397666    3629:456073   [17,3,0,14]         17   [17,3,0,14]             17  3629'397480 2020-08-17 04:22:26.814605     3629'396120 2020-08-15 23:05:06.296695             0 
12.2      36744                  0        0         0       0 2550191501           0          0 3072     3072               active+clean 2020-08-16 03:08:42.253250  3629'347207    3629:362427    [11,5,6,0]         11    [11,5,6,0]             11  3629'347074 2020-08-16 03:08:42.253178     3629'347074 2020-08-16 03:08:42.253178             0 
8.6          24                  0        0         0       0          0           0          0 3062     3062               active+clean 2020-08-16 12:37:44.415819 3629'7978201  3629:11874913   [5,15,2,16]          5   [5,15,2,16]              5 3629'7959487 2020-08-16 12:37:44.415756    3629'7881700 2020-08-12 20:39:57.383255             0 
10.4       8773                  0        0         0       0 3672511308           0          0 3054     3054               active+clean 2020-08-17 06:16:12.995209 3629'4131987  3629:27722587    [7,9,18,6]          7    [7,9,18,6]              7 3629'4131972 2020-08-17 06:16:12.995135    3629'4131953 2020-08-15 20:25:05.273445             0 
11.5          1                  0        0         0       0          0        7596         32 3024     3024               active+clean 2020-08-15 23:27:34.148382 3620'3774307   3629:3779273  [11,1,16,14]         11  [11,1,16,14]             11 3620'3774307 2020-08-15 23:27:34.148298    3620'3774307 2020-08-15 23:27:34.148298             0 
14.1          0                  0        0         0       0          0           0          0    0        0          active+undersized 2020-08-11 11:50:38.283647          0'0         3628:8        [18,7]         18        [18,7]             18          0'0 2020-08-11 11:50:37.135596             0'0 2020-08-11 11:50:37.135596             0 
13.2        592                  0        0         0       0   36805387   242490960     524872 3051     3051               active+clean 2020-08-16 12:34:53.339785  3629'360774    3629:410030   [11,7,12,6]         11   [11,7,12,6]             11  3629'359198 2020-08-16 12:34:53.339730     3629'358452 2020-08-15 03:14:11.641874             0 
12.3      36975                  0        0         0       0 2452554919           0          0 3077     3077               active+clean 2020-08-16 00:06:59.430694  3629'340108    3629:355298   [5,19,2,16]          5   [5,19,2,16]              5  3629'339995 2020-08-16 00:06:59.430645     3629'339599 2020-08-13 16:13:15.935250             0 
8.7          23                  0        0         0       0          0           0          0 3052     3052               active+clean 2020-08-16 13:49:17.117923 3629'7044994  3629:10436162   [0,16,9,19]          0   [0,16,9,19]              0 3629'7029459 2020-08-16 13:49:17.117870    3626'6924487 2020-08-10 23:04:58.110588             0 
10.5       8547                  0        0         0       0 3634533507           0          0 3045     3045               active+clean 2020-08-16 16:27:34.848239 3629'4734275  3629:27342785  [18,14,7,15]         18  [18,14,7,15]             18 3629'4734243 2020-08-16 16:27:34.848142    3629'4734099 2020-08-14 01:02:12.320648             0 
11.4          2                  0        0         0       0          0    56754276     190332 3095     3095               active+clean 2020-08-17 09:39:41.814938 3629'3871595   3629:3921886    [7,9,10,4]          7    [7,9,10,4]              7 3629'3871573 2020-08-17 09:39:41.814879    3629'3868263 2020-08-12 14:00:41.440724             0 
                                                              
14       0 0 0 0 0           0          0       0    14    14 
13    4652 0 0 0 0   259597692 2169033596 4701468 24546 24546 
12 1182753 0 0 0 0 80144984316          0       0 98035 98035 
11      15 0 0 0 0           0   75562576  256644 21660 21660 
10   69520 0 0 0 0 29298471706          0       0 24446 24446 
5        5 0 0 0 0        2050          0       0     5     5 
6        8 0 0 0 0           0          0       0  2747  2747 
7       76 0 0 0 0       14374      11048      60 12097 12097 
8      207 0 0 0 0           0          0       0 24532 24532 
                                                                  
sum 1257236 0 0 0 0 109703070138 2244607220 4958172 208082 208082 
OSD_STAT USED    AVAIL   USED_RAW TOTAL   HB_PEERS                                          PG_SUM PRIMARY_PG_SUM 
19        32 GiB 2.0 TiB   33 GiB 2.0 TiB  [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]     17              4 
18        28 GiB 2.0 TiB   29 GiB 2.0 TiB  [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,19]     22              6 
17        48 GiB 2.0 TiB   49 GiB 2.0 TiB  [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,18,19]     21              9 
16        20 GiB 2.0 TiB   21 GiB 2.0 TiB  [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,17,18,19]     17              4 
15        32 GiB 2.0 TiB   33 GiB 2.0 TiB  [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,16,17,18,19]     19              4 
14        36 GiB 2.0 TiB   37 GiB 2.0 TiB  [0,1,2,3,4,5,6,7,8,9,10,11,12,13,15,16,17,18,19]     20              2 
13        27 GiB 2.0 TiB   28 GiB 2.0 TiB  [0,1,2,3,4,5,6,7,8,9,10,11,12,14,15,16,17,18,19]     17              3 
12        47 GiB 2.0 TiB   48 GiB 2.0 TiB  [0,1,2,3,4,5,6,7,8,9,10,11,13,14,15,16,17,18,19]     23              8 
11        12 GiB 2.0 TiB   13 GiB 2.0 TiB  [0,1,2,3,4,5,6,7,8,9,10,12,13,14,15,16,17,18,19]     11              5 
10        17 GiB 2.0 TiB   18 GiB 2.0 TiB  [0,1,2,3,4,5,6,7,8,9,11,12,13,14,15,16,17,18,19]     14              2 
3         24 GiB 2.0 TiB   25 GiB 2.0 TiB [0,1,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]     14              5 
2         20 GiB 2.0 TiB   21 GiB 2.0 TiB [0,1,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]     12              4 
1         36 GiB 2.0 TiB   37 GiB 2.0 TiB [0,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]     18              4 
0         41 GiB 2.0 TiB   42 GiB 2.0 TiB [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]     21              2 
4         24 GiB 2.0 TiB   25 GiB 2.0 TiB [0,1,2,3,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]     18              4 
5         40 GiB 2.0 TiB   42 GiB 2.0 TiB [0,1,2,3,4,6,7,8,9,10,11,12,13,14,15,16,17,18,19]     23             10 
6         55 GiB 1.9 TiB   56 GiB 2.0 TiB [0,1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,18,19]     21              4 
7         35 GiB 2.0 TiB   36 GiB 2.0 TiB [0,1,2,3,4,5,6,8,9,10,11,12,13,14,15,16,17,18,19]     23              5 
8         32 GiB 2.0 TiB   33 GiB 2.0 TiB [0,1,2,3,4,5,6,7,9,10,11,12,13,14,15,16,17,18,19]     16              6 
9         31 GiB 2.0 TiB   33 GiB 2.0 TiB [0,1,2,3,4,5,6,7,8,10,11,12,13,14,15,16,17,18,19]     21              5 
sum      636 GiB  39 TiB  659 GiB  40 TiB                                                                         

* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details.

pg 14 is the only one that don't have a high availability (3 replicas or more)… why?

avmlp-osm-001 /home/ceph # ceph pg dump_stuck inactive
ok
avmlp-osm-001 /home/ceph # ceph pg dump_stuck stale
ok
avmlp-osm-001 /home/ceph # ceph pg dump_stuck undersized
ok
PG_STAT STATE                      UP     UP_PRIMARY ACTING ACTING_PRIMARY 
14.1             active+undersized [18,7]         18 [18,7]             18 
14.0             active+undersized [5,12]          5 [5,12]              5 
14.3             active+undersized  [8,5]          8  [8,5]              8 
14.2    active+undersized+degraded [7,18]          7 [7,18]              7 
14.6             active+undersized  [8,3]          8  [8,3]              8 
14.7             active+undersized [18,3]         18 [18,3]             18 
14.4    active+undersized+degraded [5,18]          5 [5,18]              5 
14.5             active+undersized [16,1]         16 [16,1]             16 
avmlp-osm-001 /home/ceph # ceph pg force-recovery 14.0
pg 14.0 doesn't require recovery; 

avmlp-osm-001 /home/ceph # ceph pg force-backfill 14.0
pg 14.0 doesn't require backfilling; 

avmlp-osm-001 /home/ceph # ceph pg force-recovery 14.4
instructing pg(s) [14.4] on osd.5 to force-recovery; 

avmlp-osm-001 /home/ceph # ceph pg force-backfill 14.4
instructing pg(s) [14.4] on osd.5 to force-backfill; 

avmlp-osm-001 /home/ceph # ceph pg force-recovery 14.2
instructing pg(s) [14.2] on osd.7 to force-recovery; 

avmlp-osm-001 /home/ceph # ceph pg force-backfill 14.2
instructing pg(s) [14.2] on osd.7 to force-backfill; 

avmlp-osm-001 /home/ceph # ceph pg ls
PG    OBJECTS DEGRADED MISPLACED UNFOUND BYTES      OMAP_BYTES* OMAP_KEYS* LOG  STATE                      SINCE VERSION      REPORTED       UP               ACTING           SCRUB_STAMP                DEEP_SCRUB_STAMP           
5.0         1        0         0       0        348           0          0    1               active+clean   18h       3605'2      3629:4694      [7,9,4,0]p7      [7,9,4,0]p7 2020-08-16 15:09:36.159945 2020-08-16 15:09:36.159945 
5.1         0        0         0       0          0           0          0    0               active+clean   14h          0'0      3629:4511   [19,7,16,0]p19   [19,7,16,0]p19 2020-08-16 19:33:01.145470 2020-08-16 19:33:01.145470 
5.2         0        0         0       0          0           0          0    0               active+clean   24h          0'0      3629:4488   [1,15,18,10]p1   [1,15,18,10]p1 2020-08-16 09:24:09.871348 2020-08-16 09:24:09.871348 
5.3         1        0         0       0        805           0          0    1               active+clean   20h       3605'3      3629:4525    [3,13,0,18]p3    [3,13,0,18]p3 2020-08-16 13:52:05.527692 2020-08-13 05:28:18.436690 
5.4         0        0         0       0          0           0          0    0               active+clean    3h        169'1      3629:2390    [12,0,5,9]p12    [12,0,5,9]p12 2020-08-17 06:52:48.025854 2020-08-13 17:30:10.364425 
5.5         0        0         0       0          0           0          0    0               active+clean   10h          0'0      3629:1051    [15,5,0,4]p15    [15,5,0,4]p15 2020-08-16 23:24:09.575954 2020-08-16 23:24:09.575954 
5.6         0        0         0       0          0           0          0    0               active+clean   31h          0'0      3629:4516  [18,14,13,5]p18  [18,14,13,5]p18 2020-08-16 02:46:27.887837 2020-08-10 06:09:08.174831 
5.7         3        0         0       0        897           0          0    3               active+clean    9h       3605'5      3629:3491   [5,15,14,18]p5   [5,15,14,18]p5 2020-08-17 00:25:41.809266 2020-08-13 05:37:52.184231 
6.0         1        0         0       0          0           0          0  346               active+clean   21h     3629'535  3629:26443405    [4,14,9,17]p4    [4,14,9,17]p4 2020-08-16 12:20:48.361537 2020-08-16 12:20:48.361537 
6.1         1        0         0       0          0           0          0  341               active+clean   30m     3629'613  3629:26497099    [6,12,13,3]p6    [6,12,13,3]p6 2020-08-17 09:32:47.922094 2020-08-16 01:49:21.114037 
6.2         0        0         0       0          0           0          0    4               active+clean   19h     2078'251     3629:58666  [10,6,13,11]p10  [10,6,13,11]p10 2020-08-16 14:09:46.658432 2020-08-11 16:12:50.795649 
6.3         1        0         0       0          0           0          0  346               active+clean   10h     3629'571  3629:26443431 [19,13,18,14]p19 [19,13,18,14]p19 2020-08-16 23:47:33.943612 2020-08-13 13:20:52.435651 
6.4         1        0         0       0          0           0          0  354               active+clean   21h     3629'543  3629:26333677   [17,9,14,8]p17   [17,9,14,8]p17 2020-08-16 12:49:44.792781 2020-08-11 09:11:27.786481 
6.5         2        0         0       0          0           0          0  668               active+clean   17h     3629'940  3629:52663020   [19,1,4,18]p19   [19,1,4,18]p19 2020-08-16 16:40:36.754710 2020-08-10 04:26:29.538486 
6.6         1        0         0       0          0           0          0  334               active+clean   24h     3629'581  3629:26239589  [12,16,13,1]p12  [12,16,13,1]p12 2020-08-16 09:44:42.924234 2020-08-12 14:42:47.482884 
6.7         1        0         0       0          0           0          0  354               active+clean   26h     3629'579  3629:26323129    [14,4,9,5]p14    [14,4,9,5]p14 2020-08-16 07:35:25.498808 2020-08-16 07:35:25.498808 
7.0        11        0         0       0       1658        2912         16  975               active+clean   24h     3629'976    3629:272928  [16,14,7,17]p16  [16,14,7,17]p16 2020-08-16 09:04:51.825161 2020-08-16 09:04:51.825161 
7.1        11        0         0       0       1560        1416          8 3076               active+clean   20h    3620'6878    3629:438089     [6,14,1,3]p6     [6,14,1,3]p6 2020-08-16 13:47:36.599648 2020-08-16 13:47:36.599648 
7.2         7        0         0       0       1857        1532          8 1858               active+clean   12h    3629'1862    3629:357200      [3,7,0,2]p3      [3,7,0,2]p3 2020-08-16 21:11:17.394835 2020-08-11 22:28:46.467550 
7.3         6        0         0       0       1730           0          0   34               active+clean   15h      3626'35    3629:226724  [17,15,6,12]p17  [17,15,6,12]p17 2020-08-16 18:17:48.970621 2020-08-14 08:40:31.413876 
7.4         6        0         0       0       1066           0          0   27               active+clean   31h      3626'28    3629:320679    [3,7,16,18]p3    [3,7,16,18]p3 2020-08-16 02:18:10.502933 2020-08-16 02:18:10.502933 
7.5        14        0         0       0       2214        4484         24 3009               active+clean   18h    3629'7811    3629:239723  [15,19,8,12]p15  [15,19,8,12]p15 2020-08-16 15:54:32.903170 2020-08-14 02:43:54.992932 
7.6        15        0         0       0       3862           0          0   48               active+clean   27h      3629'52    3629:913582   [13,1,4,16]p13   [13,1,4,16]p13 2020-08-16 06:56:31.300134 2020-08-13 21:44:05.147193 
7.7         6        0         0       0        427         704          4 3076               active+clean   12h   3629'18478 3629:135300003  [11,17,14,4]p11  [11,17,14,4]p11 2020-08-16 21:21:50.448676 2020-08-10 20:15:44.650232 
8.0        26        0         0       0          0           0          0 3050               active+clean   20h 3629'7807653  3629:11595132  [15,1,18,10]p15  [15,1,18,10]p15 2020-08-16 13:17:58.821333 2020-08-15 10:37:00.441663 
8.1        32        0         0       0          0           0          0 3076               active+clean   15h 3629'9099477  3629:13533768  [18,10,7,13]p18  [18,10,7,13]p18 2020-08-16 18:52:17.365270 2020-08-09 21:24:16.115253 
8.2        26        0         0       0          0           0          0 3081               active+clean   27h 3629'8514336  3629:12657167    [9,11,10,0]p9    [9,11,10,0]p9 2020-08-16 06:06:19.967832 2020-08-12 17:11:15.659708 
8.3        22        0         0       0          0           0          0 3030               active+clean   25h 3629'6261110   3629:9250072      [5,7,8,0]p5      [5,7,8,0]p5 2020-08-16 08:10:35.989978 2020-08-15 04:47:58.354067 
8.4        29        0         0       0          0           0          0 3048               active+clean   24m 3629'9981035  3629:14832299      [8,4,5,7]p8      [8,4,5,7]p8 2020-08-17 09:38:30.847075 2020-08-14 20:58:23.413276 
8.5        25        0         0       0          0           0          0 3070               active+clean   14h 3629'8629975  3629:12816091    [2,8,19,15]p2    [2,8,19,15]p2 2020-08-16 19:36:31.891044 2020-08-11 19:35:10.376406 
8.6        24        0         0       0          0           0          0 3026               active+clean   21h 3629'7978465  3629:11875308    [5,15,2,16]p5    [5,15,2,16]p5 2020-08-16 12:37:44.415756 2020-08-12 20:39:57.383255 
8.7        23        0         0       0          0           0          0 3081               active+clean   20h 3629'7045223  3629:10436503    [0,16,9,19]p0    [0,16,9,19]p0 2020-08-16 13:49:17.117870 2020-08-10 23:04:58.110588 
10.0     8727        0         0       0 3670473841           0          0 3059               active+clean   10h 3629'4319994  3629:27878140   [17,9,4,14]p17   [17,9,4,14]p17 2020-08-16 23:18:06.937879 2020-08-10 19:02:28.103491 
10.1     8733        0         0       0 3662648021           0          0 3077               active+clean   26h 3629'5147909  3629:31741562    [8,12,17,3]p8    [8,12,17,3]p8 2020-08-16 07:56:03.254946 2020-08-12 14:38:09.703348 
10.2     8710        0         0       0 3677817145           0          0 3067               active+clean   22h 3629'5527686  3629:33959754     [8,6,9,17]p8     [8,6,9,17]p8 2020-08-16 12:00:52.962034 2020-08-16 12:00:52.962034 
10.3     8688        0         0       0 3736528078           0          0 3053               active+clean   19h 3629'4468073  3629:27964068   [12,6,13,1]p12   [12,6,13,1]p12 2020-08-16 14:27:13.422638 2020-08-14 10:56:15.273485 
10.4     8773        0         0       0 3672511308           0          0 3054               active+clean    3h 3629'4131987  3629:27723907     [7,9,18,6]p7     [7,9,18,6]p7 2020-08-17 06:16:12.995135 2020-08-15 20:25:05.273445 
10.5     8549        0         0       0 3635015958           0          0 3047               active+clean   17h 3629'4734277  3629:27344475  [18,14,7,15]p18  [18,14,7,15]p18 2020-08-16 16:27:34.848142 2020-08-14 01:02:12.320648 
10.6     8723        0         0       0 3651236257           0          0 3042               active+clean   30h 3629'5111162  3629:33508238 [13,19,16,12]p13 [13,19,16,12]p13 2020-08-16 03:36:55.650689 2020-08-11 22:33:13.472993 
10.7     8623        0         0       0 3593613124           0          0 3053               active+clean   28h 3629'4433774  3629:25395614  [12,6,13,15]p12  [12,6,13,15]p12 2020-08-16 06:02:32.648055 2020-08-15 00:48:32.278486 
11.0        1        0         0       0          0      175636        484  264               active+clean   23h     3626'264      3629:5205    [5,13,0,10]p5    [5,13,0,10]p5 2020-08-16 10:55:15.816425 2020-08-11 19:47:25.967648 
11.1        5        0         0       0          0      842872       2620 3090               active+clean   24h 3629'7553373   3629:7563678  [13,15,16,4]p13  [13,15,16,4]p13 2020-08-16 09:24:39.030471 2020-08-15 05:17:56.146025 
11.2        4        0         0       0          0      977588       3472 3068               active+clean   26h 3629'3773773   3629:3781491   [9,11,10,12]p9   [9,11,10,12]p9 2020-08-16 07:39:36.931571 2020-08-13 19:56:13.839973 
11.3        0        0         0       0          0           0          0 3011               active+clean   15h 3605'7544511   3629:7551165     [4,2,5,19]p4     [4,2,5,19]p4 2020-08-16 18:46:04.571891 2020-08-16 18:46:04.571891 
11.4        2        0         0       0          0    56754276     190332 3007               active+clean   23m 3629'3871607   3629:3921914     [7,9,10,4]p7     [7,9,10,4]p7 2020-08-17 09:39:41.814879 2020-08-12 14:00:41.440724 
11.5        1        0         0       0          0        7596         32 3024               active+clean   34h 3620'3774307   3629:3779273  [11,1,16,14]p11  [11,1,16,14]p11 2020-08-15 23:27:34.148298 2020-08-15 23:27:34.148298 
11.6        2        0         0       0          0    16804608      59704 3067               active+clean   18h 3629'7579572   3629:7600332   [18,4,7,15]p18   [18,4,7,15]p18 2020-08-16 15:16:17.271006 2020-08-15 13:57:43.665912 
11.7        0        0         0       0          0           0          0 3041               active+clean   20h 3611'3775141   3629:3777308   [12,6,19,9]p12   [12,6,19,9]p12 2020-08-16 13:08:43.122319 2020-08-16 13:08:43.122319 
12.0    36724        0         0       0 2658760797           0          0 3021               active+clean   17h  3629'338200    3629:353651      [1,3,8,0]p1      [1,3,8,0]p1 2020-08-16 16:46:26.833264 2020-08-10 18:00:59.920643 
12.1    36560        0         0       0 2693060474           0          0 3044               active+clean   14h  3629'340743    3629:356297    [6,0,17,13]p6    [6,0,17,13]p6 2020-08-16 19:50:56.588301 2020-08-13 07:29:49.033271 
12.2    36744        0         0       0 2550191501           0          0 3076               active+clean   30h  3629'347211    3629:362431    [11,5,6,0]p11    [11,5,6,0]p11 2020-08-16 03:08:42.253178 2020-08-16 03:08:42.253178 
12.3    36976        0         0       0 2452558556           0          0 3083               active+clean   33h  3629'340114    3629:355304    [5,19,2,16]p5    [5,19,2,16]p5 2020-08-16 00:06:59.430645 2020-08-13 16:13:15.935250 
12.4    37167        0         0       0 2532685297           0          0 3065               active+clean    2h  3629'344231    3629:360201   [9,19,18,14]p9   [9,19,18,14]p9 2020-08-17 07:58:58.470487 2020-08-11 07:51:26.283423 
12.5    37005        0         0       0 2461994487           0          0 3071               active+clean   12m  3629'343955    3629:359523   [17,11,8,2]p17   [17,11,8,2]p17 2020-08-17 09:50:29.837769 2020-08-12 21:39:51.184699 
12.6    37091        0         0       0 2326678284           0          0 3029               active+clean   16h  3629'342832    3629:358174    [0,14,7,15]p0    [0,14,7,15]p0 2020-08-16 17:24:46.808204 2020-08-13 01:50:02.075255 
12.7    36975        0         0       0 2496638778           0          0 3100               active+clean   22h  3629'340118    3629:352610 [17,19,12,18]p17 [17,19,12,18]p17 2020-08-16 11:28:41.495468 2020-08-15 10:54:22.325510 
12.8    36745        0         0       0 2453736305           0          0 3047               active+clean   25h  3629'341425    3629:352764  [19,5,14,12]p19  [19,5,14,12]p19 2020-08-16 08:20:55.871363 2020-08-11 00:28:58.843200 
12.9    36761        0         0       0 2574355922           0          0 3092               active+clean  109m  3629'342468    3629:357534   [15,3,6,16]p15   [15,3,6,16]p15 2020-08-17 08:13:42.940428 2020-08-15 03:51:12.708397 
12.a    37207        0         0       0 2594481924           0          0 3096               active+clean   17h  3629'345495    3629:347041  [12,0,17,19]p12  [12,0,17,19]p12 2020-08-16 16:26:16.525778 2020-08-16 16:26:16.525778 
12.b    36624        0         0       0 2542103794           0          0 3058               active+clean    3h  3629'337926    3629:353373      [3,5,0,4]p3      [3,5,0,4]p3 2020-08-17 06:38:51.532865 2020-08-12 13:17:48.634712 
12.c    37146        0         0       0 2542888837           0          0 3046               active+clean    8h  3629'341297    3629:356994    [17,9,0,6]p17    [17,9,0,6]p17 2020-08-17 01:47:38.880347 2020-08-11 18:13:24.694080 
12.d    36882        0         0       0 2473040310           0          0 3063               active+clean   11h  3629'343050    3629:358433     [9,7,10,6]p9     [9,7,10,6]p9 2020-08-16 22:07:39.858356 2020-08-10 15:03:18.641927 
12.e    36977        0         0       0 2513728716           0          0 3099               active+clean   23h  3629'342535    3629:471615  [16,18,7,11]p16  [16,18,7,11]p16 2020-08-16 10:19:08.917672 2020-08-10 05:51:27.175188 
12.f    37170        0         0       0 2421662736           0          0 3091               active+clean   11h  3629'339856    3629:355477    [12,6,7,1]p12    [12,6,7,1]p12 2020-08-16 22:46:18.321426 2020-08-13 10:34:11.712596 
12.10   37063        0         0       0 2584770038           0          0 3050               active+clean   72m  3629'340702    3629:498660   [5,19,12,10]p5   [5,19,12,10]p5 2020-08-17 08:51:07.093497 2020-08-12 03:45:50.319811 
12.11   37166        0         0       0 2639362033           0          0 3013               active+clean   21h  3629'344753    3629:504304    [2,18,1,15]p2    [2,18,1,15]p2 2020-08-16 12:21:25.239991 2020-08-12 10:45:52.034024 
12.12   36937        0         0       0 2454977301           0          0 3061               active+clean   17h  3629'344690    3629:473964  [17,13,0,10]p17  [17,13,0,10]p17 2020-08-16 16:35:25.451262 2020-08-16 16:35:25.451262 
12.13   36989        0         0       0 2452289072           0          0 3001               active+clean    3h  3629'330247    3629:459049      [4,8,7,5]p4      [4,8,7,5]p4 2020-08-17 06:11:39.821627 2020-08-17 06:11:39.821627 
12.14   36936        0         0       0 2436184387           0          0 3077               active+clean   18h  3629'334511    3629:491996   [8,12,17,15]p8   [8,12,17,15]p8 2020-08-16 15:16:55.057592 2020-08-15 08:21:42.290408 
12.15   36841        0         0       0 2550933263           0          0 3042               active+clean    6h  3629'345296    3629:502422    [1,15,18,4]p1    [1,15,18,4]p1 2020-08-17 03:52:55.184909 2020-08-16 02:39:11.303006 
12.16   37146        0         0       0 2517689370           0          0 3094               active+clean   20h  3629'343214    3629:472912      [7,9,8,4]p7      [7,9,8,4]p7 2020-08-16 13:44:35.341577 2020-08-14 00:43:42.538689 
12.17   37235        0         0       0 2411367900           0          0 3086               active+clean   75m  3629'344474    3629:496483     [5,9,8,14]p5     [5,9,8,14]p5 2020-08-17 08:47:51.571280 2020-08-13 01:47:55.688179 
12.18   36729        0         0       0 2458800671           0          0 3098               active+clean   30h  3629'342808    3629:468672   [14,2,13,3]p14   [14,2,13,3]p14 2020-08-16 03:13:11.863020 2020-08-14 19:08:54.971752 
12.19   37311        0         0       0 2597837981           0          0 3088               active+clean   27h  3629'346758    3629:474436  [12,16,17,1]p12  [12,16,17,1]p12 2020-08-16 07:01:40.593159 2020-08-15 04:48:07.588319 
12.1a   37214        0         0       0 2632270936           0          0 3071               active+clean    2h  3629'351696    3629:494815    [5,15,14,6]p5    [5,15,14,6]p5 2020-08-17 07:44:07.798401 2020-08-16 04:51:26.455475 
12.1b   36940        0         0       0 2470626204           0          0 3053               active+clean   20h  3629'344732    3629:501605    [1,13,12,6]p1    [1,13,12,6]p1 2020-08-16 13:35:08.122944 2020-08-16 13:35:08.122944 
12.1c   36839        0         0       0 2492477640           0          0 3015               active+clean    2h  3629'343722    3629:445048      [3,1,6,0]p3      [3,1,6,0]p3 2020-08-17 07:42:13.521977 2020-08-15 22:22:48.911933 
12.1d   36549        0         0       0 2255784538           0          0 3067               active+clean    6h  3629'342857    3629:507769      [6,0,7,5]p6      [6,0,7,5]p6 2020-08-17 03:55:29.330992 2020-08-14 21:29:33.393053 
12.1e   36871        0         0       0 2452671913           0          0 3077               active+clean   28h  3629'342328    3629:605227     [2,14,5,1]p2     [2,14,5,1]p2 2020-08-16 05:41:59.725253 2020-08-15 01:28:05.207465 
12.1f   37234        0         0       0 2448717330           0          0 3077               active+clean   11h  3629'342758    3629:445978   [4,10,19,17]p4   [4,10,19,17]p4 2020-08-16 23:01:57.976326 2020-08-14 13:40:26.247849 
13.0      581        0         0       0   50331648   297297128     643500 3089               active+clean   11h  3629'332670    3629:358920  [17,19,2,12]p17  [17,19,2,12]p17 2020-08-16 22:22:54.073482 2020-08-14 05:44:32.910314 
13.1      578        0         0       0   41943040   332682568     720092 3056               active+clean   24h  3629'366771    3629:422292     [2,8,17,9]p2     [2,8,17,9]p2 2020-08-16 09:55:04.810883 2020-08-12 11:49:08.440021 
13.2      592        0         0       0   37402092   242490960     524872 3049               active+clean   21h  3629'360872    3629:410128   [11,7,12,6]p11   [11,7,12,6]p11 2020-08-16 12:34:53.339730 2020-08-15 03:14:11.641874 
13.3      606        0         0       0   34049588   207078008     448224 3067               active+clean    5h  3629'397683    3629:456105   [17,3,0,14]p17   [17,3,0,14]p17 2020-08-17 04:22:26.814605 2020-08-15 23:05:06.296695 
13.4      553        0         0       0   20972154   240751960     521108 3015               active+clean   25h  3629'852180    3629:888159   [9,11,18,16]p9   [9,11,18,16]p9 2020-08-16 08:22:47.112414 2020-08-16 08:22:47.112414 
13.5      580        0         0       0   20971680   292948688     634088 3009               active+clean    7h  3629'587527    3629:615665  [11,7,12,10]p11  [11,7,12,10]p11 2020-08-17 02:30:59.655047 2020-08-14 15:10:59.726903 
13.6      603        0         0       0   29360128   264796356     579740 3100               active+clean   12h 3629'1228251   3629:1268003   [10,6,15,9]p10   [10,6,15,9]p10 2020-08-16 22:00:12.848337 2020-08-14 07:04:49.213009 
13.7      559        0         0       0   25166390   290987928     629844 3072               active+clean   33h  3629'373240    3629:428639   [16,2,17,1]p16   [16,2,17,1]p16 2020-08-16 00:16:42.372384 2020-08-13 15:45:28.525122 
14.0        0        0         0       0          0           0          0    0          active+undersized    5d          0'0         3628:8         [5,12]p5         [5,12]p5 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 
14.1        0        0         0       0          0           0          0    0          active+undersized    5d          0'0         3628:8        [18,7]p18        [18,7]p18 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 
14.2        0        0         0       0          0           0          0    7 active+undersized+degraded    5d       3629'7        3629:21         [7,18]p7         [7,18]p7 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 
14.3        0        0         0       0          0           0          0    0          active+undersized    5d          0'0         3628:8          [8,5]p8          [8,5]p8 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 
14.4        0        0         0       0          0           0          0    7 active+undersized+degraded    5d       3629'7        3629:21         [5,18]p5         [5,18]p5 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 
14.5        0        0         0       0          0           0          0    0          active+undersized    5d          0'0         3628:8        [16,1]p16        [16,1]p16 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 
14.6        0        0         0       0          0           0          0    0          active+undersized    5d          0'0         3628:8          [8,3]p8          [8,3]p8 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 
14.7        0        0         0       0          0           0          0    0          active+undersized    5d          0'0         3628:8        [18,3]p18        [18,3]p18 2020-08-11 11:50:37.135596 2020-08-11 11:50:37.135596 

* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details.

Thoughts:

  • PG 14 is empty !!!
  • PG 14 is not scrub'in !!!
  • PG 14 does not backfill!!!!!


WHY????????

avmlp-osm-001 /home/ceph # ceph osd lspools
5 .rgw.root
6 default.rgw.control
7 default.rgw.meta
8 default.rgw.log
10 default.rgw.buckets.data
11 default.rgw.buckets.index
12 cephfs_data-ftp
13 cephfs_metadata-ftp
14 default.rgw.buckets.non-ec

Maybe PG 14 is EMPTY cause pool 14 is unused!!!!!

Pool is empty, but there are more empty pools and the warning comes only from this.

avmlp-osm-001 /home/ceph # ceph osd pool autoscale-status
 POOL                          SIZE  TARGET SIZE  RATE  RAW CAPACITY   RATIO  TARGET RATIO  BIAS  PG_NUM  NEW PG_NUM  AUTOSCALE 
 default.rgw.buckets.non-ec      0                 3.0        40940G  0.0000                 1.0       8              warn      
 cephfs_metadata-ftp          3049M                4.0        40940G  0.0003                 1.0       8              on        
 default.rgw.meta            16906k                4.0        40940G  0.0000                 1.0       8              on        
 default.rgw.buckets.index   73791k                4.0        40940G  0.0000                 1.0       8              on        
 default.rgw.control             0                 4.0        40940G  0.0000                 1.0       8              on        
 default.rgw.buckets.data    113.7G                4.0        40940G  0.0111                 1.0       8              on        
 .rgw.root                    1280k                4.0        40940G  0.0000                 1.0       8              on        
 cephfs_data-ftp             520.1G                4.0        40940G  0.0508                 1.0      32              on        
 default.rgw.log                 0                 4.0        40940G  0.0000                 1.0       8              on      

This pool does not have autoscale!!!!
Turning it on:

avmlp-osm-001 /home/ceph # ceph osd pool set default.rgw.buckets.non-ec pg_autoscale_mode on
set pool 14 pg_autoscale_mode to on
avmlp-osm-001 /home/ceph # ceph osd pool autoscale-status
 POOL                          SIZE  TARGET SIZE  RATE  RAW CAPACITY   RATIO  TARGET RATIO  BIAS  PG_NUM  NEW PG_NUM  AUTOSCALE 
 default.rgw.buckets.non-ec      0                 3.0        40940G  0.0000                 1.0       8              on        
 cephfs_metadata-ftp          3049M                4.0        40940G  0.0003                 1.0       8              on        
 default.rgw.meta            16906k                4.0        40940G  0.0000                 1.0       8              on        
 default.rgw.buckets.index   73791k                4.0        40940G  0.0000                 1.0       8              on        
 default.rgw.control             0                 4.0        40940G  0.0000                 1.0       8              on        
 default.rgw.buckets.data    113.7G                4.0        40940G  0.0111                 1.0       8              on        
 .rgw.root                    1280k                4.0        40940G  0.0000                 1.0       8              on        
 cephfs_data-ftp             520.1G                4.0        40940G  0.0508                 1.0      32              on        
 default.rgw.log                 0                 4.0        40940G  0.0000                 1.0       8              on       

check:

avmlp-osm-001 /home/ceph # ceph  -s
  cluster:
    id:     a3a799ce-f1d3-4230-a915-06e988fee767
    health: HEALTH_WARN
            Degraded data redundancy: 8 pgs undersized
 
  services:
    mon: 6 daemons, quorum avmlp-osm-001,avmlp-osm-002,avmlp-osm-003,avmlp-osm-004,avmlp-osm-005,avmlp-osm-006 (age 5d)
    mgr: avmlp-osm-005.ciberterminal.net(active, since 6w), standbys: avmlp-osm-002.ciberterminal.net, avmlp-osm-006.ciberterminal.net, avmlp-osm-004.ciberterminal.net, avmlp-osm-001.ciberterminal.net, avmlp-osm-003.ciberterminal.net
    mds: cephfs:1 {0=avmlp-osfs-002.ciberterminal.net=up:active} 3 up:standby
    osd: 20 osds: 20 up (since 7w), 20 in (since 12M)
    rgw: 4 daemons active (avmlp-osgw-001.ciberterminal.net, avmlp-osgw-002.ciberterminal.net, avmlp-osgw-003.ciberterminal.net, avmlp-osgw-004.ciberterminal.net)
 
  data:
    pools:   9 pools, 96 pgs
    objects: 1.26M objects, 102 GiB
    usage:   659 GiB used, 39 TiB / 40 TiB avail
    pgs:     88 active+clean
             8  active+undersized
 
  io:
    client:   1.1 MiB/s rd, 170 B/s wr, 16 op/s rd, 0 op/s wr
 
avmlp-osm-001 /home/ceph # ceph health detail 
HEALTH_WARN Degraded data redundancy: 8 pgs undersized
PG_DEGRADED Degraded data redundancy: 8 pgs undersized
    pg 14.0 is stuck undersized for 513135.338453, current state active+undersized, last acting [5,12]
    pg 14.1 is stuck undersized for 513135.375686, current state active+undersized, last acting [18,7]
    pg 14.2 is stuck undersized for 513135.291865, current state active+undersized, last acting [7,18]
    pg 14.3 is stuck undersized for 513135.370383, current state active+undersized, last acting [8,5]
    pg 14.4 is stuck undersized for 513135.338453, current state active+undersized, last acting [5,18]
    pg 14.5 is stuck undersized for 513135.317750, current state active+undersized, last acting [16,1]
    pg 14.6 is stuck undersized for 513135.370383, current state active+undersized, last acting [8,3]
    pg 14.7 is stuck undersized for 513135.375623, current state active+undersized, last acting [18,3]

ceph is moving!!! But still has warning, looking for placement rule:

avmlp-osm-001 /home/ceph # ceph osd pool ls detail | grep "non-ec"
pool 14 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 3630 flags hashpspool stripe_width 0 application rgw

pool 14 has the default placement rule, switching to ciberterminalRule:

avmlp-osm-001 /home/ceph # ceph osd pool set default.rgw.buckets.non-ec crush_rule ciberterminalRule
set pool 14 crush_rule to ciberterminalRule

avmlp-osm-001 /home/ceph # ceph osd pool ls detail | grep "non-ec"
pool 14 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 3631 flags hashpspool stripe_width 0 application rgw

Check:

avmlp-osm-001 /home/ceph # ceph health detail
HEALTH_OK
avmlp-osm-001 /home/ceph # ceph -s
  cluster:
    id:     a3a799ce-f1d3-4230-a915-06e988fee767
    health: HEALTH_OK
 
  services:
    mon: 6 daemons, quorum avmlp-osm-001,avmlp-osm-002,avmlp-osm-003,avmlp-osm-004,avmlp-osm-005,avmlp-osm-006 (age 5d)
    mgr: avmlp-osm-005.ciberterminal.net(active, since 6w), standbys: avmlp-osm-002.ciberterminal.net, avmlp-osm-006.ciberterminal.net, avmlp-osm-004.ciberterminal.net, avmlp-osm-001.ciberterminal.net, avmlp-osm-003.ciberterminalgro
up.net
    mds: cephfs:1 {0=avmlp-osfs-002.ciberterminal.net=up:active} 3 up:standby
    osd: 20 osds: 20 up (since 7w), 20 in (since 12M)
    rgw: 4 daemons active (avmlp-osgw-001.ciberterminal.net, avmlp-osgw-002.ciberterminal.net, avmlp-osgw-003.ciberterminal.net, avmlp-osgw-004.ciberterminal.net)
 
  data:
    pools:   9 pools, 96 pgs
    objects: 1.26M objects, 102 GiB
    usage:   659 GiB used, 39 TiB / 40 TiB avail
    pgs:     96 active+clean
 
  io:
    client:   15 KiB/s rd, 2.2 KiB/s wr, 1 op/s rd, 0 op/s wr

OUUUUU YEAHHHHHHHHH

linux/ceph/troubleshooting/pg_degraded_redundancy.txt · Last modified: 2022/02/11 11:36 by 127.0.0.1