Storage ONLY inside SynologyDrive/repo instead of regular folder structure

[patrickwolf]

Our server currently uses about 5TB and BTDU shows all of them allocated within SynologyDrive instead of within the folder structure:

Synology Drive uses a numbered repo structure so no file structure or names are visible within it which means I can't use BTDU currently.

I was wondering if anyone has some pointers?

Thanks Patrick

[CyberShadow]

Can you try --expert mode and some of the other size metrics (press m)?

[patrickwolf]

@CyberShadow thank you --expert and m helped a bit... and its still confusing where the space went as the cc folder is the biggest folder but it doesn't appear that way

[CyberShadow]

Some observations:

• Shared size is always >= represented size. In your original screenshot (which I assume is without --expert - therefore showing represented size) @synologydrive is ~5.129 TiB (represented), but in the second screenshot it is ~1.495 TiB (shared), so things have moved around in between when those screenshots were taken.

• Just from looking at the second screenshot there is little data that is exclusive to outside of the @sharesnap directory, so the sought 5TiB are covered by snapshots.

• Your terminal emulator doesn't seem to be able to render bold text correctly, which may make using btdu confusing.

• I'm not sure what "cc folder" refers to here:

  as the cc folder is the biggest folder but it doesn't appear that way

• If you are able to test a patch, I can prepare one which will allow deprioritizing @synologydrive and make btdu prefer allocating representative samples to other paths, which should help understand space usage if space is shared with an opaque filesystem layout.

[patrickwolf]

Hey @CyberShadow here are 4 size metrics taken in expert mode. What my actual desire is to visualize:

1. exclusive/shared data usage inside the /convoy (sub)folder while ignoring snapshots

• this would allow us to see the impact of duplication/deduplication which snapshots obscure as they make every file shared every 1 hour (while it doesn't actually mean that data has been deduplicated)

2. exclusive/shared in /sharesnap subfolders that isn't shared with /convoy

• ie we had data in convoy and deleted it and now its taking up exclusive in snapshots

So it seems like the feature I need is to focus exclusively on a folder and subfolders while ignoring other parts of the file system.

If you are up to creating a patch I'm happy to try it out. I've been using "btdu-static-x86_64"

Here are screenshots

[CyberShadow]

That "Shared" size cannot possibly be correct, though the sample count seems right. I must have introduced a regression in the latest version for the calculation of the shared size.

[CyberShadow]

Please try the build from the https://github.com/CyberShadow/btdu/actions/runs/14282531849 artifacts.

• I fixed the calculation of the shared size.
• I added --prefer and --ignore options as discussed above.

exclusive/shared data usage inside the /convoy (sub)folder while ignoring snapshots

Try with: --ignore /@syno/@sharesnap, or: --prefer /@syno/convoy

exclusive/shared in /sharesnap subfolders that isn't shared with /convoy

Try with: --prefer /@syno/@sharesnap, or: --ignore /@syno/convoy

[patrickwolf]

• added --prefer and --ignore options as discussed above.

thank you! looks very promising! i uploaded more screenshots as the size got reported as 770TB now. I wish we had that much :) but maybe thats meant to be the accumulated snapshot size

[CyberShadow]

thank you! looks very promising! i uploaded more screenshots

Looks good, you should now be able to go into the convoy directory and see what's using up space while ignoring snapshots.

the size got reported as 770TB now. I wish we had that much :) but maybe thats meant to be the accumulated snapshot size

Indeed, that's what the "shared" mode does - it's the size of all data if you were to read everything including every snapshot one by one, whether or not it changed between snapshots.

[patrickwolf]

Try with: --ignore /@syno/@sharesnap, or: --prefer /@syno/convoy

can i use --ignore and --prefer multiple times in one command line? also i find these very useful. do you plan to add them to a general release?

[CyberShadow]

can i use --ignore and --prefer multiple times in one command line?

Yes.

also i find these very useful. do you plan to add them to a general release?

Yes, they will be in the next release.

[CyberShadow]

This is now in 0.6.1.