Active Storage Variant causing Aws::S3::Errors::NoSuchKey error when deleting an attachment

[basiszwo]

I am not sure if this is a bug or something is not configured correctly.

Steps to reproduce

Using Active Storage with a S3-compatible object storage from digital ocean or scaleway configured along the rails guide for active storage.

Active Storage configuration

digitalocean:
  service: S3
  access_key_id: <%= ENV.fetch("DIGITALOCEAN_S3_ACCESS_KEY_ID") %>
  secret_access_key: <%= ENV.fetch("DIGITALOCEAN_S3_ACCESS_KEY_SECRET") %>
  bucket: <%= ENV.fetch("DIGITALOCEAN_S3_BUCKET_NAME") %>
  endpoint: <%= ENV.fetch("DIGITALOCEAN_S3_ENDPOINT") %>

enabled with

# config/environments/development.rb
config.active_storage.service = :digitalocean

having a User model with

has_one_attached :avatar do |attachable|
  attachable.variant(:thumb, resize_to_fill: [350, 350])
end

After uploading a file from the edit form the variant gets created when accessing the show view.

Logs look like this

ActiveStorage::VariantRecord Create (1.8ms)  INSERT INTO "active_storage_variant_records" ("blob_id", "variation_digest") VALUES ($1, $2) RETURNING "id"  [["blob_id", 116], ["variation_digest", "cAfAf5iAxm52Ff0Mw7wXY2F9d4A="]]
ActiveStorage::Blob Load (1.2ms)  SELECT "active_storage_blobs".* FROM "active_storage_blobs" INNER JOIN "active_storage_attachments" ON "active_storage_blobs"."id" = "active_storage_attachments"."blob_id" WHERE "active_storage_attachments"."record_id" = $1 AND "active_storage_attachments"."record_type" = $2 AND "active_storage_attachments"."name" = $3 LIMIT $4  [["record_id", 51], ["record_type", "ActiveStorage::VariantRecord"], ["name", "image"], ["LIMIT", 1]]
ActiveStorage::Attachment Load (0.2ms)  SELECT "active_storage_attachments".* FROM "active_storage_attachments" WHERE "active_storage_attachments"."record_id" = $1 AND "active_storage_attachments"."record_type" = $2 AND "active_storage_attachments"."name" = $3 LIMIT $4  [["record_id", 51], ["record_type", "ActiveStorage::VariantRecord"], ["name", "image"], ["LIMIT", 1]]
ActiveStorage::Blob Create (0.3ms)  INSERT INTO "active_storage_blobs" ("key", "filename", "content_type", "metadata", "service_name", "byte_size", "checksum", "created_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id"  [["key", "1dousqzwcx79wlpkfn0o0f1sjfci"], ["filename", "sigma-16mm.jpg"], ["content_type", "image/jpeg"], ["metadata", "{\"identified\":true}"], ["service_name", "digitalocean"], ["byte_size", 52716], ["checksum", "d9vmvbLEGDR2ZLzl7XNgUg=="], ["created_at", "2024-01-04 17:49:27.381653"]]
ActiveStorage::Attachment Create (0.4ms)  INSERT INTO "active_storage_attachments" ("name", "record_type", "record_id", "blob_id", "created_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["name", "image"], ["record_type", "ActiveStorage::VariantRecord"], ["record_id", 51], ["blob_id", 117], ["created_at", "2024-01-04 17:49:27.382925"]]
TRANSACTION (0.4ms)  COMMIT
S3 Storage (120.4ms) Uploaded file to key: 1dousqzwcx79wlpkfn0o0f1sjfci (checksum: d9vmvbLEGDR2ZLzl7XNgUg==)

Attachments are being displayed, everything is fine, until deleting an attachment where I get the following exception

DEBUG -- :   TRANSACTION (0.2ms)  BEGIN
DEBUG -- :   ActiveStorage::Attachment Exists? (1.7ms)  SELECT 1 AS one FROM "active_storage_attachments" WHERE "active_storage_attachments"."blob_id" = $1 LIMIT $2  [["blob_id", 119], ["LIMIT", 1]]
DEBUG -- :   ActiveStorage::VariantRecord Load (0.5ms)  SELECT "active_storage_variant_records".* FROM "active_storage_variant_records" WHERE "active_storage_variant_records"."blob_id" = $1  [["blob_id", 119]]
DEBUG -- :   ActiveStorage::Attachment Load (0.8ms)  SELECT "active_storage_attachments".* FROM "active_storage_attachments" WHERE "active_storage_attachments"."record_id" = $1 AND "active_storage_attachments"."record_type" = $2 AND "active_storage_attachments"."name" = $3 LIMIT $4  [["record_id", 119], ["record_type", "ActiveStorage::Blob"], ["name", "preview_image"], ["LIMIT", 1]]
DEBUG -- :   ActiveStorage::Blob Destroy (1.4ms)  DELETE FROM "active_storage_blobs" WHERE "active_storage_blobs"."id" = $1  [["id", 119]]
DEBUG -- :   TRANSACTION (6.7ms)  COMMIT
 INFO -- :   S3 Storage (100.6ms) Deleted file from key: 5qag8y7y3qqol4wrgo9iapy2z8oy
 INFO -- :   S3 Storage (38.1ms) Deleted files by key prefix: variants/5qag8y7y3qqol4wrgo9iapy2z8oy/
ERROR -- : Error performing ActiveStorage::PurgeJob (Job ID: 508059d0-a343-4377-ae75-c89195417e61) from Sidekiq(default) in 161.91ms: Aws::S3::Errors::NoSuchKey (Aws::S3::Errors::NoSuchKey):

I see this error twice in my logs. One time for the original and one time for the variant.

Active Storage uploads the blobs without any prefix. See logs on top.

Am I doing something wrong? Is my application not configured correctly? What am I missing? Thank you 🙏

Expected behavior

Deleting an attachment should not throw errors

Variants are being uploaded to a variants subfolder

Actual behavior

Deleting an attachment throws an Aws::S3::Errors::NoSuchKey (Aws::S3::Errors::NoSuchKey) error as with the used key variants/5qag8y7y3qqol4wrgo9iapy2z8oy/ no objects can be found.

This is where Active Storage tries to delete the prefixed variants which have never been created before: https://github.com/rails/rails/blob/main/activestorage/app/models/active_storage/blob.rb#L325

System configuration

Rails version: 7.1.2

Ruby version: 3.2.2