Bug: throw er; // Unhandled 'error' event when using s3 as datasource

[TheAnachronism]

What happened?

I've tried to deploy zipline with a minio s3 bucket as data source. When trying to start the container, the error below was thrown in the logs.

Following S3 ENV variables:

  DATASOURCE_S3_ACCESS_KEY_ID: [my-id]
  DATASOURCE_S3_SECRET_ACCESS_KEY: [my-key]
  DATASOURCE_S3_ENDPOINT: [my-minio-endpoint]
  DATASOURCE_S3_BUCKET: zipline
  DATASOURCE_S3_FORCE_S3_PATH: "false"
  DATASOURCE_S3_REGION: eu-finland

It can't be minio, as other applications can easily use other buckts in there.

Version

latest (ghcr.io/diced/zipline:latest)

What browser(s) are you seeing the problem on?

No response

Zipline Logs

[90m2023-05-08 08:08:34,588 PM[0m [36minfo [0m [[34mdatasource[0m] using S3(zipline) datasource
[90m2023-05-08 08:08:34,617 PM[0m [36minfo [0m [[34mdatabase::migrations[0m] establishing database connection
[90m2023-05-08 08:08:34,618 PM[0m [36minfo [0m [[34mdatabase::migrations[0m] ensuring database exists, if not creating database - may error if no permissions
[90m2023-05-08 08:08:34,741 PM[0m [36minfo [0m [[34mdatabase::migrations[0m] exiting migrations engine - database is up to date
[90m2023-05-08 08:08:35,817 PM[0m [36minfo [0m [[34mserver[0m] listening on 0.0.0.0:3000
[90m2023-05-08 08:08:35,817 PM[0m [36minfo [0m [[34mserver[0m] started production [email protected] server

node:events:491
      throw er; // Unhandled 'error' event
      ^
S3Error
    at Object.parseError (/zipline/node_modules/minio/dist/main/xml-parsers.js:75:11)
    at /zipline/node_modules/minio/dist/main/transformers.js:167:22
    at DestroyableTransform._flush (/zipline/node_modules/minio/dist/main/transformers.js:91:10)
    at DestroyableTransform.prefinish (/zipline/node_modules/readable-stream/lib/_stream_transform.js:129:10)
    at DestroyableTransform.emit (node:events:513:28)
    at prefinish (/zipline/node_modules/readable-stream/lib/_stream_writable.js:611:14)
    at finishMaybe (/zipline/node_modules/readable-stream/lib/_stream_writable.js:620:5)
    at endWritable (/zipline/node_modules/readable-stream/lib/_stream_writable.js:643:3)
    at Writable.end (/zipline/node_modules/readable-stream/lib/_stream_writable.js:571:22)
    at IncomingMessage.onend (node:internal/streams/readable:705:10)
Emitted 'error' event on Readable instance at:
    at DestroyableTransform.<anonymous> (/zipline/node_modules/minio/dist/main/minio.js:1845:127)
    at DestroyableTransform.emit (node:events:513:28)
    at /zipline/node_modules/minio/dist/main/minio.js:1780:33
    at DestroyableTransform.<anonymous> (/zipline/node_modules/minio/dist/main/minio.js:573:13)
    at DestroyableTransform.emit (node:events:525:35)
    at done (/zipline/node_modules/readable-stream/lib/_stream_transform.js:192:25)
    at /zipline/node_modules/readable-stream/lib/_stream_transform.js:130:7
    at DestroyableTransform._flush (/zipline/node_modules/minio/dist/main/transformers.js:91:7)
    at DestroyableTransform.prefinish (/zipline/node_modules/readable-stream/lib/_stream_transform.js:129:10)
    at DestroyableTransform.emit (node:events:513:28) {
  amzRequestid: null,
  amzId2: null,
  amzBucketRegion: null
}

Node.js v18.15.0

Browser Logs

No response

Additional Info

I've tried with both DATASOURCE_S3_FORCE_S3_PATH true and false. There seems to go something wrong when parsing the S3 parameters or something.

[TacticalTechJay]

Do the env variables you set correctly reflect in the read-config script? You can run it with yarn scripts:read-config or npm run scripts:read-config.

[TheAnachronism]

I'm running in Kubernetes, which makes this execution a bit difficult, when the application crashes the entire time, but I managed it:

"s3": {
      "access_key_id": "my-key",
      "secret_access_key": "my-access-key",
      "endpoint": "my-endpoint",
      "bucket": "my-bucket-name",
      "force_s3_path": false,
      "region": "my-region",
      "use_ssl": false
    }

Basically all the info it needs is there.

[diced]

@TheAnachronism Try the DATASOURCE_S3_USE_SSL=true variable.

[dorogoy]

I had almost the same problem, but with IDrive e2 and DATASOURCE_S3_USE_SSL=true solved it. Thanks!

[TheAnachronism]

Using DATASOURCE_S3_USE_SSL=true, the instance crashes on startup with the following logs:

2023-06-01 12:50:26,648 PM info  [server] listening on 0.0.0.0:3000
2023-06-01 12:50:26,648 PM info  [server] started production [email protected] server
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPROTO 88E98D254F7F0000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1605:SSL alert number 40

    at __node_internal_captureLargerStackTrace (node:internal/errors:490:5)
    at __node_internal_errnoException (node:internal/errors:620:12)
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)
Emitted 'error' event on Readable instance at:
    at DestroyableTransform.<anonymous> (/zipline/node_modules/minio/dist/main/minio.js:1845:127)
    at DestroyableTransform.emit (node:events:513:28)
    at /zipline/node_modules/minio/dist/main/minio.js:1780:33
    at ClientRequest.<anonymous> (/zipline/node_modules/minio/dist/main/minio.js:588:9)
    at ClientRequest.emit (node:events:513:28)
    at TLSSocket.socketErrorListener (node:_http_client:502:9)
    at TLSSocket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -71,
  code: 'EPROTO',
  syscall: 'write'
}

Node.js v18.15.

[TheAnachronism]

Alright I've managed to get it running with force_s3_path to true. Uploaded files actually land in the S3 bucket, but I can't seem to download them anymore with Zipline. Images don't get displayed and PDFs aren't displayed either. I cannot download the files either, as they Zipline just shows a blank screen:

@diced is there any problem right now with S3?

[TheAnachronism]

Just tried the entire thing with DEBUG enabled and nothing shows in the log either. What is going on here?

[diced]

@TheAnachronism I'm honestly not sure what the issue here is.. if you are sure that the logs don't contain anything it would be pretty difficult to figure out what happened. you can try to make the bucket read/write for everyone (this can be dangerous, but it's worth a shot lol) and see if it still works, if it doesn't we know what the problem might be.

[TheAnachronism]

Mhh ok,

Currently, this is running with a minio bucket in the background. I'll try and see if something changes, if I use, i.e. Backblaze or something, to figure out if it's with S3 or Zipline.

[TheAnachronism]

OK so I fixed the upload issue. I get no errors uploading something. But now I still have the weird behavior, that after uploading something, the file lands properly in the target bucket, but I cannot access it through zipline. Calling the URL zipline provides just returns an empty response with the following DEBUG logs:

2023-10-08 09:25:14,605 PM info  [server::response] GET /u/8jExJN.png -> 200
2023-10-08 09:25:14,605 PM debug [server::response] {"method":"GET","url":"/u/8jExJN.png","headers":
{"host":"zipline.MY.DOMAIN","x-request-id":"061565da5e7b15fdb6a93bcd81d04d32","x-real-ip":"MYIP","x-forwarded-
for":"MYIP","x-forwarded-host":"zipline.MY.DOMAIN","x-forwarded-port":"443","x-forwarded-proto":"https","x-
forwarded-scheme":"https","x-scheme":"https","x-original-forwarded-for":"MYIP","cf-connecting-ip":"MYIP","cf-
ipcountry":"CH","accept-encoding":"gzip, br","cf-ray":"813174846e6f08cc-LAX","cf-visitor":"{\"scheme\":\"https\"}","sec-ch-
ua":"\"Microsoft Edge\";v=\"117\", \"Not;A=Brand\";v=\"8\", \"Chromium\";v=\"117\"","sec-ch-ua-mobile":"?0","sec-ch-ua-
platform":"\"Windows\"","dnt":"1","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 
Edg/117.0.2045.60","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,applicat
ion/signed-exchange;v=b3;q=0.7","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-
dest":"document","accept-language":"en-GB,en;q=0.9","cdn-loop":"cloudflare"}}

I very much doubt, that it has something to do with Cloudflare that is in front of it, so what is the problem?

[TheAnachronism]

Just changed removed the Cloudflare proxy and that didn't change anything. Also checked the anonymous access for the minio bucket again and that seems ok too. Any idea what is going on here? @diced

[TheAnachronism]

I've now given it another try and I don't get any error messages any more in the logs. But instead I got a

{"statusCode":500,"code":"AccessDenied","error":"Internal Server Error","message":"Valid and authorized credentials required"}

response when trying to open the uploaded image / file. The uploaded file DOES land in the minio bucket, so that works. But something is going wrong with reading this again.

How does zipline do that internally? Does it just redirect to the public URL from the s3 bucket, or does it get the file from there and then serve it by itself? @diced

[CraftJunkies]

Did you see this error on your way? Im stuck with that one....

[diced]

How does zipline do that internally? Does it just redirect to the public URL from the s3 bucket, or does it get the file from there and then serve it by itself?

It just requests the file from the bucket and returns whatever it got.

I'm not sure what service you are using for s3 but the only thing that comes to mind at this point is just messing around with the configuration options (which you have already done by the looks of it), and then messing around with rules and such that limit a certain api key from working. Zipline basically lets you configure everything about how it authenticates with s3 so I am like 50% confident it can't be a Zipline issue (but, it may always be that case who knows lol). This is kind of sad but this issue has been open for almost a year so lets try to fix whatever is going on here soon enough 😂

[TheAnachronism]

Alright, so:

I finally got some version of this working. I have no idea what is the issue with a normal minio server in as the S3 source, but well -.- I had to set force s3 path to false, and set up minio to allow for subdomain access so bucket.mydomain.com/file Only this way, zipline could properly upload and display things without running into access errors.

I'm really not sure, but I even set up a local dev environment and tested it with that, and it seems that this rather is an issue with the minio client, rather than your usage of it.

No idea what is going wrong, but I'm too fed up with this to further investigate, now that I got some version working.

Anyway, thanks for the support!