dify icon indicating copy to clipboard operation
dify copied to clipboard

Error: failed to deploy dify with object storage(obs)

Open btdan opened this issue 2 days ago • 26 comments

Self Checks

  • [x] This is only for bug report, if you would like to ask a question, please head to Discussions.
  • [x] I have searched for existing issues search for existing issues, including closed ones.
  • [x] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
  • [x] [FOR CHINESE USERS] 请务必使用英文提交 Issue,否则会被关闭。谢谢!:)
  • [x] Please do not modify this template :) and fill in all the required fields.

Dify version

1.0.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Hello, I want to create knowledge base with help of dify . and want the knowledge base stored on object storage(OBS). I follow this article: https://bbs.huaweicloud.com/blogs/447986
In the article above. The method to modified .env is as follows:

# The type of storage to use for storing user files.
# Supported values are `local` and `s3` and `azure-blob` and `google-storage` and `tencent-cos`,
# Default: `local`
STORAGE_TYPE=s3
# S3 Configuration
# Whether to use AWS managed IAM roles for authenticating with the S3 service.
# If set to false, the access key and secret key must be provided.
S3_USE_AWS_MANAGED_IAM=false
# The endpoint of the S3 service.
S3_ENDPOINT=XXX
# The region of the S3 service.
S3_REGION=XXX
# The name of the S3 bucket to use for storing files.
S3_BUCKET_NAME=XXX
# The access key to use for authenticating with the S3 service.
S3_ACCESS_KEY=XXX
# The secret key to use for authenticating with the S3 service.
S3_SECRET_KEY=XXX
#--------- .env content end --------------
And then I install minio on my ubuntu system, and can access from http://127.0.0.1:9090
And then I modify the .env as follows:

------------ .env content start --------------
# The type of storage to use for storing user files.
STORAGE_TYPE=s3

# Apache OpenDAL Configuration
# The configuration for OpenDAL consists of the following format: OPENDAL_<SCHEME_NAME>_<CONFIG_NAME>.
# You can find all the service configurations (CONFIG_NAME) in the repository at: https://github.com/apache/opendal/tree/main/core/src/services.
# Dify will scan configurations starting with OPENDAL_<SCHEME_NAME> and automatically apply them.
# The scheme name for the OpenDAL storage.
OPENDAL_SCHEME=fs
# Configurations for OpenDAL Local File System.
OPENDAL_FS_ROOT=storage

# S3 Configuration
#
S3_ENDPOINT=http://127.0.0.1:9090
S3_REGION=
S3_BUCKET_NAME=difydb
S3_ACCESS_KEY=XJCG7rKb4UFy1CZDBfV
S3_SECRET_KEY=foJKaecE3iOHQtN70cbLfZhn9xMSyPsEGg1V1qm
# Whether to use AWS managed IAM roles for authenticating with the S3 service.
# If set to false, the access key and secret key must be provided.
S3_USE_AWS_MANAGED_IAM=false

✔️ Expected Behavior

I want the comand : docker-compose up -d can run sucessfully and can store knowledge base on the objective storage(OBS).

❌ Actual Behavior

however, When I run the comand : docker-compose up -d the messages displayed are as follows:

Starting docker_db_1 ... 
Starting docker_redis_1 ... 
Starting docker_db_1              ... done
Starting docker_redis_1           ... done
Starting docker_web_1             ... done
Recreating docker_plugin_daemon_1 ... 
Starting docker_sandbox_1         ... done

ERROR: for docker_plugin_daemon_1  'ContainerConfig'
Starting docker_ssrf_proxy_1      ... done
Recreating docker_worker_1        ... 

Starting docker_weaviate_1        ... done

ERROR: for docker_worker_1  'ContainerConfig'

ERROR: for plugin_daemon  'ContainerConfig'

ERROR: for api  'ContainerConfig'

ERROR: for worker  'ContainerConfig'
Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 33, in <module>
    sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 81, in main
    command_func()
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 203, in perform_command
    handler(command, command_options)
  File "/usr/lib/python3/dist-packages/compose/metrics/decorator.py", line 18, in wrapper
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1184, in up
    to_attach = up(False)
                ^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1164, in up
    return self.project.up(
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/project.py", line 697, in up
    results, errors = parallel.parallel_execute(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 108, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 206, in producer
    result = func(obj)
             ^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/project.py", line 679, in do
    return service.execute_convergence_plan(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/service.py", line 579, in execute_convergence_plan
    return self._execute_convergence_recreate(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/service.py", line 499, in _execute_convergence_recreate
    containers, errors = parallel_execute(
                         ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 108, in parallel_execute
    raise error_to_reraise
  File "/usr/lib/python3/dist-packages/compose/parallel.py", line 206, in producer
    result = func(obj)
             ^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/service.py", line 494, in recreate
    return self.recreate_container(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/service.py", line 612, in recreate_container
    new_container = self.create_container(
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/service.py", line 330, in create_container
    container_options = self._get_container_create_options(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/service.py", line 921, in _get_container_create_options
    container_options, override_options = self._build_container_volume_options(
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/service.py", line 960, in _build_container_volume_options
    binds, affinity = merge_volume_bindings(
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/service.py", line 1548, in merge_volume_bindings
    old_volumes, old_mounts = get_container_data_volumes(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/compose/service.py", line 1579, in get_container_data_volumes
    container.image_config['ContainerConfig'].get('Volumes') or {}
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 'ContainerConfig'

By the way, when I change the .env to the default one, that is to say ,the copy of .env.exampe the comand docker-compose up -d can run sucessfully. Can anyone help me to solve this problem? Thanks very much.

btdan avatar Mar 03 '25 08:03 btdan