[bug]upload file error

[chenzikun]

Self Checks

• [X] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.14.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

.env

storage_type=s3

opendal.UnexpectedError: Unexpected (temporary) at Writer::close, context: { url: https://s3.cn-northwest-1.amazonaws.com/llm-dify/upload_files/c463a29f-f080-42ab-adda-ec632932d9bc/28eb80f4-4c7f-4a1d-8691-81eafc94856c.png, called: http_util::Client::send, service: s3, path: upload_files/c463a29f-f080-42ab-adda-ec632932d9bc/28eb80f4-4c7f-4a1d-8691-81eafc94856c.png, written: 535646 } => send http request, source: error sending request for url (https://s3.cn-northwest-1.amazonaws.com/llm-dify/upload_files/c463a29f-f080-42ab-adda-ec632932d9bc/28eb80f4-4c7f-4a1d-8691-81eafc94856c.png)

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[chenzikun]

add context

opendal.UnexpectedError: Unexpected (temporary) at Writer::close, context: { url: https://s3.cn-northwest-1.amazonaws.com/llm-dify/upload_files/334df837-1844-4405-a44e-64c528e819b6/78c43d9b-9edb-4679-b882-c7f776fd382f.png, called: http_util::Client::send, service: s3, path: upload_files/334df837-1844-4405-a44e-64c528e819b6/78c43d9b-9edb-4679-b882-c7f776fd382f.png, written: 318298 } => send http request, source: error sending request for url (https://s3.cn-northwest-1.amazonaws.com/llm-dify/upload_files/334df837-1844-4405-a44e-64c528e819b6/78c43d9b-9edb-4679-b882-c7f776fd382f.png)
2024-12-16 13:37:00,241.241 INFO [Dummy-34] [_client.py:1038] - HTTP Request: POST http://ai-llms-proxy-enetestus.autel.com/v1/chat/completions "HTTP/1.1 200 OK"
2024-12-16 13:37:11,390.390 ERROR [Dummy-35] [ext_storage.py:84] - Failed to save file upload_files/c463a29f-f080-42ab-adda-ec632932d9bc/28eb80f4-4c7f-4a1d-8691-81eafc94856c.png
Traceback (most recent call last):
  File "/app/api/extensions/ext_storage.py", line 82, in save
    self.storage_runner.save(filename, data)
  File "/app/api/extensions/storage/opendal_storage.py", line 37, in save
    self.op.write(path=filename, bs=data)

[dosubot[bot]]

Hey @chenzikun! I'm here to help you with bugs, questions, and becoming a contributor. Let's get to the bottom of this together!

I found a similar unsolved discussion regarding file URL mismatches in a self-hosted Dify instance using S3, which might be related to your issue. You can view the discussion here ^[1].

Regarding the error you're encountering, the opendal.UnexpectedError typically indicates a temporary issue when closing the writer during a file upload to S3. This could be due to network instability, temporary S3 service issues, or incorrect configuration settings.

Here are a few steps you can take to troubleshoot and potentially resolve the issue:

1. Check Network Stability: Ensure that your network connection is stable and there are no interruptions during the file upload process.

2. Verify S3 Configuration: Double-check your S3 configuration in the .env file to ensure all details like S3_ENDPOINT, S3_BUCKET_NAME, S3_ACCESS_KEY, S3_SECRET_KEY, and S3_REGION are correct and match your S3 setup ^[2].

3. Retry Mechanism: If the error is indeed temporary, implementing a retry mechanism in your code might help overcome transient network or service issues.

4. Logs and Monitoring: Enable detailed logging to capture more information about the error. This can help identify if there are specific patterns or conditions under which the error occurs.

If the issue persists, further investigation into the specific error messages and logs might be necessary to pinpoint the exact cause.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[chenzikun]

api/extensions/ext_storage.py

 def get_storage_factory(storage_type: str) -> Callable[[], BaseStorage]:
     match storage_type:
         case StorageType.S3:
             from extensions.storage.opendal_storage import OpenDALStorage

             kwargs = _load_s3_storage_kwargs()
             return lambda: OpenDALStorage(scheme=OpenDALScheme.S3, **kwargs)
         case StorageType.OPENDAL:
             from extensions.storage.opendal_storage import OpenDALStorage

             scheme = OpenDALScheme(dify_config.STORAGE_OPENDAL_SCHEME)
             kwargs = _load_opendal_storage_kwargs(scheme)
             return lambda: OpenDALStorage(scheme=scheme, **kwargs)
         case StorageType.LOCAL:
             from extensions.storage.opendal_storage import OpenDALStorage
             

why case StorageType.S3, still use OpenDALStorage?

[dosubot[bot]]

The get_storage_factory function uses OpenDALStorage for case StorageType.S3 because it is designed to handle S3 storage using the OpenDAL framework. This setup allows OpenDALStorage to manage S3 storage operations by configuring it with the necessary S3 parameters such as bucket, endpoint, access_key_id, secret_access_key, and region, which are loaded using the _load_s3_storage_kwargs function. This approach ensures that OpenDALStorage can be used flexibly with different S3-compatible services, including handling special cases like AWS managed IAM and Cloudflare R2 ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[chenzikun]

second bug cause by the change code

Traceback (most recent call last):
  File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 489, in wrapper
    resp = resource(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask/views.py", line 110, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/.venv/lib/python3.12/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
    resp = meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/controllers/console/wraps.py", line 142, in decorated
    return view(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/libs/login.py", line 92, in decorated_view
    return current_app.ensure_sync(func)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/controllers/console/wraps.py", line 26, in decorated
    return view(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/controllers/console/workspace/models.py", line 360, in get
    models = model_provider_service.get_models_by_model_type(tenant_id=tenant_id, model_type=model_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/services/model_provider_service.py", line 284, in get_models_by_model_type
    provider_configurations = self.provider_manager.get_configurations(tenant_id)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/provider_manager.py", line 131, in get_configurations
    custom_configuration = self._to_custom_configuration(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/provider_manager.py", line 586, in _to_custom_configuration
    self.decoding_rsa_key, self.decoding_cipher_rsa = encrypter.get_decrypt_decoding(tenant_id)
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/core/helper/encrypter.py", line 35, in get_decrypt_decoding
    return rsa.get_decrypt_decoding(tenant_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/api/libs/rsa.py", line 57, in get_decrypt_decoding
    raise PrivkeyNotFoundError("Private key not found, tenant_id: {tenant_id}".format(tenant_id=tenant_id))
libs.rsa.PrivkeyNotFoundError: Private key not found, tenant_id: 98eca97f-0d92-4ddd-9b9f-4174eb947f22

[chenzikun]

to fix the bug, I have change the code to case StorageType.S3: from extensions.storage.aws_s3_storage import AwsS3Storage return AwsS3Storage

[taverngoal]

me too

[laipz8200]

OpenDAL can also connect to S3, and we plan to use it to support more storage types in the future.

Would you be willing to share your S3 configuration after masking sensitive information? This could help us with testing and troubleshooting issues.

[chenzikun]

@laipz8200

# S3 Configuration
# Whether to use AWS managed IAM roles for authenticating with the S3 service.
# If set to false, the access key and secret key must be provided.
S3_USE_AWS_MANAGED_IAM=true
# The endpoint of the S3 service.
S3_ENDPOINT=
# The region of the S3 service.
S3_REGION=cn-northwest-1
# The name of the S3 bucket to use for storing files.
S3_BUCKET_NAME=llm-dify
# The access key to use for authenticating with the S3 service.
S3_ACCESS_KEY=
# The secret key to use for authenticating with the S3 service.
S3_SECRET_KEY=
~~~~~~~~~~~~~~~~~~~~

[laipz8200]

Thanks, will fix it in the next version.

[majian159]

Thanks, will fix it in the next version.

Is there a temporary solution?

[chenzikun]

@majian159 使用旧代码，docker挂载进去，给你个参考 volumes: - ./ext_storage.py:/app/api/extensions/ext_storage.py

[laipz8200]

Thanks, will fix it in the next version.

Is there a temporary solution?

After our tests, connect to S3 with AK/SK is work well. Do you use AWS managed IAM to access S3?

[majian159]

Thanks, will fix it in the next version.

Is there a temporary solution?

After our tests, connect to S3 with AK/SK is work well. Do you use AWS managed IAM to access S3?

My exception message is:

opendal.ConfigInvalidError: ConfigInvalid (permanent) at Builder::build, context: { service: s3 } => region is missing. Please find it by S3::detect_region() or set them in env.

I use minio

It seems that the region is not configured. Is there any recommended configuration?

[laipz8200]

Thanks, will fix it in the next version.

Is there a temporary solution?

After our tests, connect to S3 with AK/SK is work well. Do you use AWS managed IAM to access S3?

My exception message is:

opendal.ConfigInvalidError: ConfigInvalid (permanent) at Builder::build, context: { service: s3 } => region is missing. Please find it by S3::detect_region() or set them in env.

I use minio

It seems that the region is not configured. Is there any recommended configuration?

Try S3_REGION=us-east-1

[laipz8200]

Hi @chenzikun, I’ve brought back the boto3-based S3 Storage in #11754, and it should now work seamlessly without any configuration changes. Feel free to give it a try!

Also, the updated OpenDAL now supports IAM and offers better performance and retry capabilities. You can update the configuration to start using it.