Folders not visible inside asset containers via Digital Ocean Spaces

[taylorcammack]

Bug description

I'm transferring assets from local to Digital Ocean Spaces. If it helps, I used s3cmd to upload them in bulk, not sure if it's different than uploading them directly. The site itself runs completely fine, but when I go to view the assets, the folders are not visible.

Spaces interface showing folders

Local cp interface just showing assets

If I go to an assets field on an entry where the field is restricted to a single folder, I can view assets within that folder. So everything else seems to work, it just doesn't show the folders in the UI.

BTW, for the main asset container, I have the root set to a subfolder at /main. However, it doesn't seem to matter what the root is set to, none of the folders are visible.

    'do_spaces_assets' => [
      'driver' => 's3',
      'key' => env('DO_SPACES_KEY'),
      'secret' => env('DO_SPACES_SECRET'),
      'endpoint' => env('DO_SPACES_ENDPOINT'),
      'region' => env('DO_SPACES_REGION'),
      'bucket' => env('DO_SPACES_BUCKET'),
      'root' => 'main',
      'url' => env('DO_SPACES_URL'),
      'visibility' => 'public',
    ],

How to reproduce

• Create Digital Ocean space and upload files
• Hook up asset container
• View Assets in CP

Logs

No response

Environment

Environment
Laravel Version: 9.31.0
PHP Version: 8.1.10
Composer Version: 2.4.2
Environment: local
Debug Mode: ENABLED
Maintenance Mode: OFF

Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: NOT CACHED
Views: CACHED

Drivers
Broadcasting: log
Cache: statamic
Database: mysql
Logs: stack / single
Mail: log
Queue: sync
Session: file

Statamic
Addons: 5
Antlers: regex
Version: 3.3.37 PRO

Statamic Addons
aryehraber/statamic-captcha: 1.8.0
simonhamp/statamic-heroicons: 1.0.1
spatie/statamic-responsive-images: 2.13.1

Installation

Other (please explain)

Antlers Parser

regex (default)

Additional details

Custom starter kit

[taylorcammack]

Here's my assets.yaml configuration

title: Assets
disk: do_spaces_assets
allow_uploads: true
allow_downloading: true
allow_renaming: true
allow_moving: true
create_folders: true

[taylorcammack]

Can confirm when I create the folder inside the Space using the "New Folder" interface, it shows up in the Control Panel. So I guess there's something different happening when using s3cmd?

[jasonvarga]

Have you cleared your cache since syncing all the files?

[taylorcammack]

Yep, I've run cache:clear and glide:clear

[jasonvarga]

Just to clarify, those folders definitely have files in them?

[taylorcammack]

Yes, they do have files in them. However, I noticed that you can create a folder using the Spaces interface and it will continue to show up even when it's empty. Makes sense.

But say I uploaded a folder named "Test" with images in them. I can create a folder called "Test 2" in the interface and when I move all the images over, rather than just having an empty folder named "Test", the original folder no longer appears.

So it's like the system thinks it's empty even when it has files in it, but that's only apparent when you remove the files.

[taylorcammack]

So as a workaround, we can create temporary folders using the interface, move the assets over and then rename to original, but seems like it should just work 🤔

[jasonvarga]

I'm a little confused between which interface you're talking about. The Statamic control panel vs. the Digital Ocean spaces interface.

Statamic won't notice any changes you make directly on Digital Ocean. Only if you enable the Stache watcher. This is intentional otherwise there would be a ton of overhead to continually perform API calls to DO.

[taylorcammack]

Here's a short screencast of the workflow, hope this helps. Had to split into two parts for upload size

https://user-images.githubusercontent.com/2514935/191604852-12212e19-b577-4d9d-8e0d-a4a20dfa6797.mov

[taylorcammack]

Part 2

https://user-images.githubusercontent.com/2514935/191605247-00db5343-c356-4fbe-b278-b66e4e807d0c.mov

[taylorcammack]

So, if you have the Stache watcher on, Statamic will recognize if you add files and directories through the Digital Ocean interface.

But if you add those files & directories to the Space using s3cmd, Statamic will recognize those files and will serve them correctly, but the uploaded directories just won't show up in the Assets panel. Interesting.

[ahinkle]

Hiya,

I'm able to replicate this issue with S3. Files show but interior folders do not.

1. I migrated from Local storage to S3.
2. Used AWS CLI to migrate assets to respective folders
3. Folders do not display in the assets manager. However, they do appear if accessed directly (via URL)

[ahinkle]

https://user-images.githubusercontent.com/17038330/194974113-b3161bd9-b2c1-4e69-ab8a-304e23e684e3.mov

[ahinkle]

Small Update:

• I noticed that uploading a file in the folder that does not appear makes the folder appear again.
• I compared file permissions between the two files; they are the same.
• In investigating, I noticed that the missing folders do not appear in the AssetContainer listing; the other folders that were manually created in Statamic do. https://github.com/statamic/cms/blob/3.3/src/Assets/AssetContainer.php#L272-L277
• Moving up, I saw that the missing folders are not returning from the AssetContainerContents filesystem API response. https://github.com/statamic/cms/blob/3.3/src/Assets/AssetContainerContents.php#L27

Switching recursive to false displays the missing folders:

$files = $this->filesystem()->listContents('/', false);

However, files within these interior folders do not appear (because it's expecting the recursive search + find within the directory)

• Appears that it could be solved by switching recursive to false and then tweaking the API call to fetch files from the API under the requested list contents directory (instead of all, then recursively filtering by the requested directory)

[aerni]

I'm running into the same issue with S3 on UpCloud. If I create a folder in the Statamic CP it is uploaded to the bucket. I can see it in the UpCloud dashboard. However, when reloading the Statamic CP, the folder is not there anymore. Adding files to the root without a folder works as expected. Only folders are an issue.

I tried on a fresh Statamic installation and I'm running league/flysystem-aws-s3-v3: 3.10.0

Environment
Application Name: Statamic
Laravel Version: 9.37.0
PHP Version: 8.1.11
Composer Version: 2.4.3
Environment: local
Debug Mode: ENABLED
URL: s3.test
Maintenance Mode: OFF

Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: NOT CACHED
Views: CACHED

Drivers
Broadcasting: log
Cache: statamic
Database: mysql
Logs: stack / single
Mail: smtp
Queue: sync
Session: file

Statamic
Addons: 0
Antlers: regex
Version: 3.3.49 Solo

[artemverbo]

Same problem with s3, appears a recent issue. Our website go-live process involves syncing s3 buckets (staging -> prod), anything that was synched and has folders will not be displayed. Assets are searchable via Statamic search interface by names etc, but no folders are showing. If the folder was created via Statamic assets UI — it will show. But it also would not allow to create the existing folder names (says they exist obviously). Tried to clear cache/stache in all possible ways etc, but has no effect and "existing" bucket folders are not displaying on the new production instance.

Somehow even when locally I then change the AWS credentials to use the prod one, it will only show one "new" folder that I created via Statamic interface post-go-live, but none of the old/existing/synced folders are showing.

UPD: What's interesting, even if I create new folders via AWS S3 console, these new folders are then appearing in Statamic.

UPD2: Ok, so after a few hour of debugging this one, I have found that it has nothing to do with Statamic.

If anyone else needs a workaround: It is related to the way S3 works and they don't have a concept of "folders", everything there is an object with a path.

When you create new Myfolder "folder object" via Statamic UI (league/flysystem-aws-s3-v3 under the hood) or via AWS console, it actually creates an object with a path like Myfolder/ and this gets returned/listed for Statamic by AwsS3V3Adapter::listContents.

Then you use aws s3 sync (aws cli) in my case or @taylorcammack's mentioned s3cmd, that is probably based on aws cli too, those "folder objects" are for some reason ignored (assumed as useless?). And then only actual files/"real objects" are synced.

So none of the synced folders or subfolders are visible in Statamic, but in AWS console all files are displayed as being in folders, because they contain "real objects" with paths like Myfolder/file.txt

When attempting to create a new/exisitng folder in Statamic, it attempts to check if the directory exists by calling AwsS3V3Adapter::directoryExists and that simply checks if there are any objects with paths prefixed by Myfolder/. As it finds our Myfolder/file.txt it assumes the folder "exists" and would not allow to create one with the same name.

However, in AWS console, once you have your sync done, although you "see folders", you can use "Create Folder" button and create an "existing"/same name folder without any errors. This creates that "folder object" (Myfolder/) that we need in Statamic for folders to be "visible". Basically I had to manually create all folder and subfolder "folder objects", luckily I didn't have a lot. Might be a script opportunity in the future :)

[barnabas-szekeres]

Same problem can be reproduce with Minio CLI client. The mirrored or copied folders wouldn't be visible in CP. :(

Workaround: On the CP you can jump to the folders and create dummy files within it. After that, the folder will be visible in the CP again.

[jasonvarga]

I'm trying to figure this out but it's just so weird.

The whole "there are no directories" concept in S3 doesn't quite track either.

(I say S3, but I'm using Digital Ocean spaces. This issue happens to any "S3 compatible" filesystem like S3, DO, Minio, etc)

I can create a completely empty directory using the dashboard (the Digital Ocean Spaces dashboard, not Statamic) and it shows up just fine in Statamic. There's no "file" inside the directory at all.

I can't figure out what's different about those directories that aren't appearing.

Also, this is happening on the AWS level. It's not a Statamic thing. ie. if I make a request to list the contents using the S3 api and completely bypass Statamic, the folders are missing too.

🤯