Nested /data/files/files/files/files/files

[glensc]

Version

21.12.0

Steps to Reproduce

The problem seems to be identical to https://github.com/getsentry/self-hosted/issues/1417 (which is closed and locked without resolution), that the nested is created with each upgrade as it seems to match the update count made in this installation.

Expected Result

1. find the cause
2. offer instructions to recover
3. fix the structure

Actual Result

so currently my data is in /data/files/files/files/files:

root@2e6a3366ea93:/data# find /data/|grep -v /data/files/files/files/files/
/data/
/data/files
/data/files/files
/data/files/files/files
/data/files/files/files/files
/data/files/files/files/custom-packages
/data/files/files/files/custom-packages/.checksum
/data/files/files/custom-packages
/data/files/files/custom-packages/.checksum
/data/files/custom-packages
/data/files/custom-packages/.checksum
/data/custom-packages
/data/custom-packages/.checksum

root@2e6a3366ea93:/data# du /data/files/files/files/files/ -sh
7.1G    /data/files/files/files/files/

and it contains one empty dir files:

root@2e6a3366ea93:/data# ls -la  /data/files/files/files/files/files
total 8
drwxr-xr-x   2 sentry sentry 4096 Mar  4 20:44 .
drwxr-xr-x 260 sentry sentry 4096 Mar  4 21:03 ..
root@2e6a3366ea93:/data#

[glensc]

Also, what are the files in files, can they be deleted? and how to nicely delete them?

[glensc]

I'm absolutely sure this is the offending line making files/files nesting:

• https://github.com/getsentry/self-hosted/blob/6c88b78de6f15a19d8cf99e16d21517eed16461d/install/migrate-file-storage.sh#L8

[glensc]

The first time this was attempted to be fixed:

• https://github.com/getsentry/self-hosted/pull/297

[glensc]

Perhaps that data migration step can be removed, as due the hard stops in installation:

• https://develop.sentry.dev/self-hosted/releases/#hard-stops

this migration is already completed, as seems, https://github.com/getsentry/self-hosted/commit/3c9e8c28ff52d1a0dce1603de5a514915f545adc already appeared first in 10.0.0

[glensc]

also, arent -1 and -x options to ls conflicting or something?

• https://github.com/getsentry/self-hosted/blob/3c9e8c28ff52d1a0dce1603de5a514915f545adc/install.sh#L133

as according to my tests, the ls options don't work as expected (print number of files for wc):

       -A, --almost-all
               do not list implied . and ..
       -x     list entries by lines instead of by columns
       -1     list one file per line

# docker run --rm -v sentry-data:/data alpine ash -c "ls -A1x /data"
custom-packages  files

if the code wanted to count entries it should just use ls -1, -A is redundant without -a:

# docker run --rm -v sentry-data:/data alpine ash -c "ls -1 /data"
custom-packages
files

[glensc]

This commit introduced the files migration:

• https://github.com/getsentry/self-hosted/commit/89e8053c4089f2bdc8b8ae5b7c20792fbe824828

[glensc]

Also moving to /tmp from /data is bad: /data is a volume but /tmp is overlayfs

1. it's different filesystem so the move is slow
2. duplicate disk space will be used on /tmp

so the rename should have happened inside /data directory:

cd /data
set -- *
mkdir -p files
mv "$@" files

this way move is fast and doesn't involve copying between partitions.

[chadwhitacre]

I'm absolutely sure this is the offending line making files/files nesting:

Oooh, nice find! Putting this on the backlog to stop the triage clock.

[chadwhitacre]

My hypothesis is that /data/files already exists when we do mv /data/* /tmp/files/. That would result in /tmp/files/files/.

[glensc]

Please read the further notes. I think that migration can be just dropped now:

• https://github.com/getsentry/self-hosted/issues/1510#issuecomment-1153257519

[roock]

This issue has been fixed in #1710 by removing the offending script.

[glensc]

so, fixed in 22.10.0

[chadwhitacre]

Good catch, @roock. Best code is no code. :)