rm followed by exists returns true

[sshleifer]

Why does fs.exists() sometimes return True even after deleting directories using fsspec/s3fs? Is there a reliable way to confirm directory deletion on Cloudflare R2 in python?

If I rclone lsf r2://bucket-name/test from command line, contents are as I expect (deletion happened).

import fsspec
import s3fs

# Initialize filesystem pointing to Cloudflare R2 (or another S3-compatible service)
fs = s3fs.S3FileSystem(
    key='YOUR_ACCESS_KEY',
    secret='YOUR_SECRET_KEY',
    endpoint_url='https://YOUR_ACCOUNT_ID.r2.cloudflarestorage.com',
)

# Path to test
test_dir = 'bucket-name/test/checkpoint_dir'

# Create a test file
fs.touch(f'{test_dir}/file.txt')

# Verify existence
print("Exists before deletion:", fs.exists(test_dir))  # True expected

# Delete the directory
fs.rm(test_dir, recursive=True)

# Verify deletion (issue occurs here)
exists_after_delete = fs.exists(test_dir)
print(f'Exists after deletion: {exists_after_delete}')  # Expected False, but always true

[martindurant]

I create the following test, which closely matches your code:

def test_exist_after_delete(s3):
    test_dir = f'{test_bucket_name}/test/checkpoint_dir'
    s3.touch(f'{test_dir}/file.txt')
    assert s3.exists(test_dir)
    s3.rm(test_dir, recursive=True)
    assert not s3.exists(test_dir)

The test passes, so your situation doesn't seem to be reproduced. Aside from running no R2, can you think of anything that might be different in your case?