[FEATURE] minio and s3, as a file search alternative

[yanpassaro]

Describe the feature you'd like Something more robust would be of great value to the system, both for saving and retrieving documents. An alternative to this can be minio (an open source based on amazon's s3) and s3 itself

[davincios]

I agree @yanpassaro

Though S3 I can understand but about Minio, why would you use that?

[yanpassaro]

No specific reason, just because it's an open source platform :D, maybe azure storage or google storage could be added

[HenryHengZJ]

welcome any contribution to more doc loaders like S3 - https://js.langchain.com/docs/modules/data_connection/document_loaders/integrations/web_loaders/s3 !

[mikkelkrogsholm]

I can't connect to minio. I spin it up in the same compose file. It should work, but I get this all the time:

flowise | 2024-09-06 12:02:45 [INFO]: ⬆️ POST /api/v1/document-store/store flowise | 2024-09-06 12:02:54 [INFO]: ⬆️ POST /api/v1/document-store/loader/process flowise | 2024-09-06 12:02:55 [ERROR]: unhandledRejection: getaddrinfo ENOTFOUND flowise.minio flowise | Error: getaddrinfo ENOTFOUND flowise.minio flowise | at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26) flowise | Error: getaddrinfo ENOTFOUND flowise.minio flowise | at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26) flowise | 2024-09-06 12:02:55 [ERROR]: unhandledRejection: getaddrinfo ENOTFOUND flowise.minio flowise | Error: getaddrinfo ENOTFOUND flowise.minio flowise | at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)

[thiagolealassis]

In the past, I submitted a pull request to address this issue. In my environment, Flowise was successfully connecting to MinIO, and everything was working perfectly on version 2.2.2. However, after updating to the latest version, I started encountering this error.

2025-01-15 20:29:00 [ERROR]: unhandledRejection: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. at throwDefaultError (/usr/local/lib/node_modules/flowise/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20) at /usr/local/lib/node_modules/flowise/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5 at de_CommandError (/usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4935:14) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async /usr/local/lib/node_modules/flowise/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20 at async /usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18 at async /usr/local/lib/node_modules/flowise/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38 at async /usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:286:18 at async /usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22 at async /usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14 at async /usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22

My config was: STORAGE_TYPE: s3 S3_ENDPOINT_URL: https://minio.sample.com S3_STORAGE_ACCESS_KEY_ID: ####### S3_STORAGE_SECRET_ACCESS_KEY: ######## S3_STORAGE_REGION: us-east-1 S3_STORAGE_BUCKET_NAME: flowise S3_FORCE_PATH_STYLE: true

[ghost]

Ran to the same bug as reported by @thiagolealassis with version 2.2.4 (onPrem-setup with minio) - 2.2.3 works:

`

flowise 2025-01-20 12:38:08 [ERROR]: unhandledRejection: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. │ │ flowise PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. │ │ flowise at throwDefaultError (/usr/local/lib/node_modules/flowise/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20) │ │ flowise at /usr/local/lib/node_modules/flowise/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5 │ │ flowise at de_CommandError (/usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4935:14) │ │ flowise at process.processTicksAndRejections (node:internal/process/task_queues:95:5) │ │ flowise at async /usr/local/lib/node_modules/flowise/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20 │ │ flowise at async /usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18 │ │ flowise at async /usr/local/lib/node_modules/flowise/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38 │ │ flowise at async /usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:286:18 │ │ flowise at async /usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22 │ │ flowise at async /usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14 │ │ flowise at async /usr/local/lib/node_modules/flowise/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22

`

Config: STORAGE_TYPE: s3 S3_ENDPOINT_URL: http://minio:9000 S3_STORAGE_ACCESS_KEY_ID: ####### S3_STORAGE_SECRET_ACCESS_KEY: ######## S3_STORAGE_REGION: us-east-1 S3_STORAGE_BUCKET_NAME: flowise S3_FORCE_PATH_STYLE: true

@thiagolealassis could you share your PR?thx!

[ghost]

Issue fixed with [email protected]