python-slack-sdk

python-slack-sdk copied to clipboard

Reproducible in:

# test.py
import sys
# Enable debug logging
import logging

from slack_sdk import WebClient

def uploadToSlackChannel():
    client = WebClient(
        token="you-slack-token", timeout=300)
    try:
        filepath = "file-to-upload"
        response = client.files_upload(
            channels="your-slack-channel-name", file=filepath)
        assert response["file"]  # the uploaded file
    except Exception as e:
        print(e)

uploadToSlackChannel()        

The Slack SDK version

slack-sdk==3.13.0

Python runtime version

Python 3.9.10

OS info

ProductName: Mac OS X ProductVersion: 10.15.7 BuildVersion: 19H1615 Darwin Kernel Version 19.6.0: Sun Nov 14 19:58:51 PST 2021; root:xnu-6153.141.50~1/RELEASE_X86_64

Steps to reproduce:

(Share the commands to run, source code, and project settings (e.g., setup.py))

1. copy the above snippet to any file (e.g. upload_to_slack.py)
2. run python3 upload_to_slack.py

Expected result:

Should succeed without any error

Actual result:

The file gets uploaded but also throws this error

Received a response in a non-JSON format: stream timeout
The server responded with: {'status': 408, 'headers': {'x-edge-backend': 'envoy-www', 'content-length': '14', 'content-type': 'text/plain', 'x-slack-edge-shared-secret-outcome': 'no-match', 'date': 'Wed, 26 Jan 2022 07:12:45 GMT', 'server': 'envoy', 'via': 'envoy-edge-bom-jbcm', 'connection': 'close'}, 'body': 'stream timeout'}

Requirements

For general questions/issues about Slack API platform or its server-side, could you submit questions at https://my.slack.com/help/requests/new instead. :bow:

Please read the Contributing guidelines and Code of Conduct before creating this issue or pull request. By submitting, you are agreeing to those rules.

Hi @iamarjun, thanks for writing in!

We recently received a similar feedback and that was due to some server-side issue: https://github.com/slackapi/python-slack-sdk/issues/1162#issuecomment-1016209037 In the case where HTTP status 408 is returned from the server-side, the timeout setting in WebClient does not affect at all.

I'm sorry for bothering you again but would you mind contacting our customer support agents from either https://my.slack.com/help/requests/new or /feedback in your Slack workspace? It'd be appreciated if you could understand this.

Hi @seratch, when I use curl to upload the file I get stream timeout within 9-10s, even for 5MB files I get this error

 sh "curl -s https://slack.com/api/files.upload -F initial_comment=\"" + initial_message + "\" -F token=\"" + access_token + "\" -F channels=\"" + channel_name + "\" -F title=\"" + file_name + "\" -F filename=\"" + file_name + "\" -F file=@" + file_path

Hence shifted to SDK, although the file gets uploaded I still get stream timeout, not sure what the problem is.

That may be due to some connectivity issue between your host and Slack but I'm unable to help you out for it.

Trying the same script in a different host may help for figuring out whether it's caused by Slack's workspace specific issue or network issues in your host. I hope this helps.

@seratch initially I thought it was the CI Server that is causing the issue, but I could reproduce it locally as well.

Hi @seratch here's the full log using curl

curl -v https://slack.com/api/files.upload -F token=access-token -F channels= "#ios-ajay-test1" -F file=@file_path

*   Trying 13.126.138.201...
* TCP_NODELAY set
* Connected to slack.com (13.126.138.201) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Slack Technologies, Inc.; CN=slack.com
*  start date: Apr 13 00:00:00 2021 GMT
*  expire date: Apr 18 23:59:59 2022 GMT
*  subjectAltName: host "slack.com" matched cert's "slack.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fcc26010800)
> POST /api/files.upload HTTP/2
> Host: slack.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 195338476
> Content-Type: multipart/form-data; boundary=------------------------ce8e5e2beebe823d
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
* We are completely uploaded and fine
< HTTP/2 408 
< x-edge-backend: envoy-www
< content-length: 14
< content-type: text/plain
< x-slack-edge-shared-secret-outcome: no-match
< date: Wed, 26 Jan 2022 08:48:33 GMT
< server: envoy
< via: envoy-edge-bom-3mz6
< 
* Connection #0 to host slack.com left intact
stream timeout* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
* Closing connection 0

@iamarjun Thanks for sharing more details!

I've checked internal reports and found that other people are also facing this server-side issue with relatively large size files (I know that your file is not so large but it seems that the situation can arise even with mega-byte-level size).

The server-side engineering teams are working hard to eliminate (at least mitigate) the issue now but I cannot tell when the situation can be improved.

The only workaround that I can suggest is that, if you receive the 408 error, your app can check files.list API results with the same token to know whether the upload was actually successful. If there is no new item in the response, your app needs to retry the same files.upload API call.

We're sorry for the disruption.

Even we are facing the same issue @seratch. We are not able to upload build artefacts to slack 😢

Keeping you in the loop here @jaypoojara. Let's follow this thread.

Facing the same issue, trying to upload 60MB file using curl request, failing with stream timeout error.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 61.0M    0     0  100 61.0M      0  50.2M  0:00:01  0:00:01 --:--:-- 50.2M
100 61.0M    0     0  100 61.0M      0  27.5M  0:00:02  0:00:02 --:--:-- 27.5M
100 61.0M    0     0  100 61.0M      0  18.9M  0:00:03  0:00:03 --:--:-- 18.9M
100 61.0M    0     0  100 61.0M      0  14.4M  0:00:04  0:00:04 --:--:-- 14.4M
100 61.0M    0     0  100 61.0M      0  11.6M  0:00:05  0:00:05 --:--:-- 11.6M
100 61.0M    0     0  100 61.0M      0   9.8M  0:00:06  0:00:06 --:--:--     0
100 61.0M    0     0  100 61.0M      0  8655k  0:00:07  0:00:07 --:--:--     0
100 61.0M    0     0  100 61.0M      0  7602k  0:00:08  0:00:08 --:--:--     0
100 61.0M    0     0  100 61.0M      0  6777k  0:00:09  0:00:09 --:--:--     0
100 61.0M  100    14  100 61.0M      1  6117k  0:00:14  0:00:10  0:00:04     2
100 61.0M  100    14  100 61.0M      1  6116k  0:00:14  0:00:10  0:00:04     3
stream timeout

Any other workaround? I'm trying the retry mechanism @seratch suggested but it is failing every time for me (408 and no actual message being posted).

@locomike I'm sorry to say this but I don't have anything further that I can suggest at this moment. We will continue communicating with the server-side teams.

I am facing the same issue, let me know once it's resolved.

I am facing the same issue here. Subscribing for the updates

I am facing the same issue here. Subscribing for the updates

Any other workaround? I'm trying the retry mechanism @seratch suggested but it is failing every time for me (408 and no actual message being posted).

Same here, failing all the time even after multiple retry.

I see these intermittently. Re-trying even a minute later and the same files upload successfully without issue.

The error was common two weeks ago, disappeared for a few weeks, and now it is back again today.

see also: https://stackoverflow.com/questions/70752546/receiving-an-error-from-server-when-trying-to-upload-a-file

Is there any correlation with file type? This started happening to us with ~33MB zip file.
Larger text (csv) files never had this issue...but our CSVs hit a ridiculous size and we started zipping them when they were over some threshold and then we started getting this issue. Makes me suspicious that this might relate to file-type rather than just raw size.

Can anyone else commend on the file types you are experiencing the problem with (binary vs text and also mime-type)? Maybe there is a correlation that will make this easier for the devs to repro.

@mattpr we are seeing this issue with files that are 2kb-3kb in size, plain text, and have .log file extension. The issue appears in ~5% of uploads. A file that triggers the error will upload without issue on a re-try seconds later.

Thus, file size/type does not seem to be a factor here.

Interesting. We get this error consistently on every try (100%) with the ~33MB zip that is generated each day. We have retry logic on failure (3x before hard fail) but the upload succeeds each time. So the upload happens successfully 3 times and the 408 error happens each of the three times.

Maybe a temporary workaround for those who are experiencing this issue intermittently. It retries three times each with a 3s pause.

for file_name in files_list:
    # Try up to three times because Slack file uploads are recently kinda flaky.
    # @see https://github.com/slackapi/python-slack-sdk/issues/1165
    tries = 3
    for i in range(tries):
        if i > 0:
            print(f"Trying upload again for {file_name} (attempt {i+1} of {tries}) ...")
        try:
            # Limit to 20 calls per minute so we don't get rate-limited by Slack.
            # @see https://api.slack.com/docs/rate-limits
            if len(files_list) > 20:
                time.sleep(3)
            upload_result = client.files_upload(file=file_name)
            logger.info(upload_result)
        except SlackApiError as e:
            logger.error(f"Error uploading file {file_name}. Attempt {i+1} of {tries}: {e}")
            if i < tries - 1:
                time.sleep(3)
                continue
            else:
                raise
        break

Edit: We have been using the above for a week now. Normally it does not need more than one try. But occasionally it goes up to three tries, with the third try succeeding.

I'm facing this same issue as well, on ~90% of tries, starting on 2022/02/22. In my case the files are all .xlsx files in the 30-60MB range.

Facing the same issue on 100% of all large file uploads. The files are uploaded successfully but the response is always returning as 408.

Received a response in a non-JSON format: stream timeout
The server responded with: {'status': 408, 'headers': {'x-edge-backend': 'envoy-www', 'content-length': '14', 'content-type': 'text/plain', 'x-slack-edge-shared-secret-outcome': 'no-match', 'date': 'Thu, 03 Mar 2022 22:00:39 GMT', 'server': 'envoy', 'via': 'envoy-edge-iad-zd04', 'connection': 'close'}, 'body': 'stream timeout'}

👋 It looks like this issue has been open for 30 days with no activity. We'll mark this as stale for now, and wait 10 days for an update or for further comment before closing this issue out. If you think this issue needs to be prioritized, please comment to get the thread going again! Maintainers also review issues marked as stale on a regular basis and comment or adjust status if the issue needs to be reprioritized.

Please, priorotize

Yup. Issue still there and happening every day. 30MB Zip file, get error/message that from API. But upload did not fail: file is posted to slack channel.
So we retry and post the same file multiple times until we give up (we keep getting the failure message).
Slack users get the same 30MB file 3 times every day (we retry twice).

It would save Slack some money in disk space to fix this. :)

I'd think given the specific error message that they would be able to track this case down server-side fairly easily. If someone at slack needs someone on the client side to generate the repro "while they are watching", just let us know. I'm sure someone here can help (me included).

I just excluded this issue from the auto-triage process.

We are sorry again for the disruption that many apps have been experiencing with files.upload API.

In the last few weeks, the server-side engineering team has applied the improvements to mitigate the issue for small-size files. The team is aware of this issue and they are working on the further steps to resolve the issue. We are unable to share the details at this moment. Once we publish the announcement, we will update you all here.

Hi all, just want to share some insights, we tried to throttle the upload speed to 500KB/sec and it seems to be working now.

It was a big problem for us because we generate the data to be uploaded concurrently as we upload the data to slack api, so when the api returns timeout error the file is still uploaded, but truncated. It only happens on our cloud environment though, we tried reproducing locally and it never happened, turns out the upload speed in our cloud env is faster, and it suddenly works when we throttle the speed to be similar as in local env.

Interesting find @idzharbae , I think the same is happening for us. It usually works with my slower home internet but fails with the gigabit connection on CI.

Calls from AWS lambda also result in the same error.

Facing the same issue. Any workaround? files getting uploaded repeatedly with retries yet it shows failed.

Another +1