nearly 1MM empty directories under /var/vcap/store/shared

[ae-govau]

Issue

When looking into why some backups were running slowly we noted that our CF blobstore (using singleton-blobstore) has an extraordinary amount of empty directories.

find /var/vcap/store/shared -empty -type d | wc -l
883167

I note we actually only have 161 files!

find /var/vcap/store/shared -type f | wc -l
161

Most of the empty-dirs appear to be in cc-droplets:

...
/var/vcap/store/shared/cc-droplets/ae/ac/aeacaa07-929d-4c4e-b799-bcd7e799fb4d
/var/vcap/store/shared/cc-droplets/ae/ac/aeac8d1c-5e02-4146-a4ab-ac8e67a271af
/var/vcap/store/shared/cc-droplets/ae/ac/aeac3842-fd0b-499f-aded-2b320f396a1d
/var/vcap/store/shared/cc-droplets/ae/ac/aeac2392-2b04-479c-854f-0ed8b0c5b446
/var/vcap/store/shared/cc-droplets/ae/ac/aeac178b-529c-4b9b-b8c1-ecf7dbe6633c
...

Context

We deploy set of apps daily to our Cloudfoundry.

Steps to Reproduce

Run a Cloudfoundry for a long time with regular apps being pushed.

Expected result

Empty directories to not grow unbounded.

Current result

Empty directories being left on disk.

Possible Fix

We will probably run a pre-start script to install a cronjob or similar to delete the empty dirs, something like: https://github.com/cloudfoundry/cloud_controller_ng/issues/1602#issuecomment-663345978