[Feature Request] Support to S3 file storage

[pablomorato1]

On my application is necessary to store the media sent on a message on S3 right when my Waha client receives the message to because I use SQS to handle the requests and ensure the delivery of all messages on a server downtime.

I'd like to have a plug and play feature that make possible to deliver the media data directly to my S3 bucket, and have the file url for it returned instead of the local_storage file url.

Is there already any way to do it?

[allburov]

Not yet, but we'll add it, shouldn't be a problem!

[arielpts]

+1

[HuangDaHui]

+1

[mohammadkurdi]

+1

[shivkodam]

+1

[puntorigen]

+1

[meovary150]

+1

[melquisedeque-magalhaes]

+1

[devlikepro]

S3 support will be added in 2024.9.1 version :crossed_fingers:

[marcelomanchester]

• 1

[chardea]

+1

[devlikepro]

https://waha.devlike.pro/docs/how-to/storages/#media---s3

Media - S3

You can use the S3 storage to store the media files.

Any S3 Compatible storage can be used, such as AWS S3, MinIO, DigitalOcean Spaces, etc. For in-house solutions, you can use MinIO.

Configuration

• WAHA_MEDIA_STORAGE=S3 - enable the S3 storage
• WAHA_S3_REGION=eu-west-1 - the region of the S3 bucket
• WAHA_S3_BUCKET=waha - the name of the S3 bucket
• WAHA_S3_ACCESS_KEY_ID=minioadmin - the access key of the S3 bucket
• WAHA_S3_SECRET_ACCESS_KEY=minioadmin - the secret access key of the S3 bucket
• WAHA_S3_ENDPOINT=http://127.0.0.1:9000 - the endpoint of the S3 bucket (not required for AWS S3)
• WAHA_S3_FORCE_PATH_STYLE=True - force path style for the S3 bucket (not required for AWS S3)
• WAHA_S3_PROXY_FILES - proxy media files through WAHA (False by default)
  • WAHA_S3_PROXY_FILES=False - generate pre-signed URLs for media files and send them to the client in media.url
  • WAHA_S3_PROXY_FILES=True - WAHA will proxy media files through itself in media.url

After you enabled S3 here's example for message webhook payload:

{
  "event": "message",
  "session": "default",
  "engine": "WEBJS",
  "payload": {
    "id": "[email protected]_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
    "hasMedia": true,
    "media": {
        "url": "http://localhost:3000/api/files/[email protected]_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.jpg",
        "mimetype": "image/jpeg",
        "filename": null,
        "s3": {
            "Bucket": "bucket-name",
            "Key": "/default/[email protected]_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.jpg"
        },
        "error": null // if there was an error during file download
    }
    ...
  }
}

in addition to media.* field it will have media.s3.* field with the S3 bucket information.

S3 Metadata

Each file saved in S3 has additional metadata:

• X-Amz-Meta-Waha-Session=default - the session name
• X-Amz-Meta-Waha-Message-Id=true_111... - the message ID
• X-Amz-Meta-Waha-Media-File-Name=media.jpg - the media file name (if available)

[devlikepro]

In 2024.9.1 https://waha.devlike.pro/docs/overview/changelog/#20249

[HuangDaHui]

https://waha.devlike.pro/docs/how-to/storages/#media---s3

Media - S3

You can use the S3 storage to store the media files.

Any S3 Compatible storage can be used, such as AWS S3, MinIO, DigitalOcean Spaces, etc. For in-house solutions, you can use MinIO.

Configuration

• WAHA_MEDIA_STORAGE=S3 - enable the S3 storage

• WAHA_S3_REGION=eu-west-1 - the region of the S3 bucket

• WAHA_S3_BUCKET=waha - the name of the S3 bucket

• WAHA_S3_ACCESS_KEY_ID=minioadmin - the access key of the S3 bucket

• WAHA_S3_SECRET_ACCESS_KEY=minioadmin - the secret access key of the S3 bucket

• WAHA_S3_ENDPOINT=http://127.0.0.1:9000 - the endpoint of the S3 bucket (not required for AWS S3)

• WAHA_S3_FORCE_PATH_STYLE=True - force path style for the S3 bucket (not required for AWS S3)

• WAHA_S3_PROXY_FILES - proxy media files through WAHA (False by default)

  • WAHA_S3_PROXY_FILES=False - generate pre-signed URLs for media files and send them to the client in media.url
  • WAHA_S3_PROXY_FILES=True - WAHA will proxy media files through itself in media.url

After you enabled S3 here's example for message webhook payload:

{
  "event": "message",
  "session": "default",
  "engine": "WEBJS",
  "payload": {
    "id": "[email protected]_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
    "hasMedia": true,
    "media": {
        "url": "http://localhost:3000/api/files/[email protected]_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.jpg",
        "mimetype": "image/jpeg",
        "filename": null,
        "s3": {
            "Bucket": "bucket-name",
            "Key": "/default/[email protected]_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA.jpg"
        },
        "error": null // if there was an error during file download
    }
    ...
  }
}

in addition to media.* field it will have media.s3.* field with the S3 bucket information.

S3 Metadata

Each file saved in S3 has additional metadata:

• X-Amz-Meta-Waha-Session=default - the session name
• X-Amz-Meta-Waha-Message-Id=true_111... - the message ID
• X-Amz-Meta-Waha-Media-File-Name=media.jpg - the media file name (if available)

Can I use Azure Blobs?

[devlikepro]

Can I use Azure Blobs?

If there's S3 compatible interface - yes! but I don't think so https://learn.microsoft.com/en-us/answers/questions/1183760/s3-api-support-over-azure-blob-storage

However, you can use an S3-to-Azure gateway https://portal.azure.com/#create/flexify.azure-s3-apice

[HuangDaHui]

我可以使用 Azure Blob 吗？

如果有 S3 兼容接口 - 是的！ 但我不这么认为https://learn.microsoft.com/en-us/answers/questions/1183760/s3-api-support-over-azure-blob-storage

但是，您可以使用 S3 到 Azure 网关 https://portal.azure.com/#create/flexify.azure-s3-apice

ok let me look into it thank you very much

[HuangDaHui]

hello, i have a question If I use S3 to store media attachments, will the attachment be re-downloaded when I request this interface? downloadMedia=true /api/messages?chatId=11111111111%40c.us&downloadMedia=true&limit=1&session=default

[devlikepro]

will the attachment be re-downloaded

Hi! With all media storages it'll first check if the file is already in the storage and after that download or just return url from the existing file, so you can safely request it over and over, it will not re-download files multiple time if you're asking about that!

[HuangDaHui]

will the attachment be re-downloaded

Hi! With all media storages it'll first check if the file is already in the storage and after that download or just return url from the existing file, so you can safely request it over and over, it will not re-download files multiple time if you're asking about that!

Okay, but I encountered a problem recently. When I requested this interface, I got 20 messages and still couldn't get the result after more than 20 minutes.

/api/messages?chatId=11111111111%40c.us&downloadMedia=true&limit=20&session=default Error: response status is 524

[HuangDaHui]

will the attachment be re-downloaded

Hi! With all media storages it'll first check if the file is already in the storage and after that download or just return url from the existing file, so you can safely request it over and over, it will not re-download files multiple time if you're asking about that!

Okay, but I encountered a problem recently. When I requested this interface, I got 20 messages and still couldn't get the result after more than 20 minutes.

/api/messages?chatId=11111111111%40c.us&downloadMedia=true&limit=20&session=default Error: response status is 524

I don’t know why, but it keeps getting 524, so I want to try S3 to see if it can solve this situation.

[devlikepro]

@HuangDaHui hi! Could you send us your discord please so we can find out the problem with 20 messages? It should work like in a minute, even if all 20 messages would have simple image [email protected]

[HuangDaHui]

@HuangDaHui hi! Could you send us your discord please so we can find out the problem with 20 messages? It should work like in a minute, even if all 20 messages would have simple image [email protected]

Hi, I sent you an email but I don't know how to contact you using Discord.