Failed to upload Slab

[daniel-lucio]

trafficstars

Current Behavior

So, uploading a file in multiparts sometimes fails. This seems to be intermittent as retrying for a second time the same command worked.

curl -v -i -u :MYPASSWORD -X PUT 'http://127.0.0.1:9880/api/worker/multipart/foo?bucket=default&uploadid=0477c079851130f86e7fe50196545dadb6fd5a219202d498f0c2e4e9beb17c05&partnumber=2' --data-binary "@/tmp/fileEvKa3N.1" -H "Content-Type: application/octet-stream"

• Trying 127.0.0.1:9880...
• Connected to 127.0.0.1 (127.0.0.1) port 9880
• Server auth using Basic with user ''

PUT /api/worker/multipart/foo?bucket=default&uploadid=0477c079851130f86e7fe50196545dadb6fd5a219202d498f0c2e4e9beb17c05&partnumber=2 HTTP/1.1 Host: 127.0.0.1:9880 Authorization: Basic OmYxbDNtMG4zbGdyNG5kMw== User-Agent: curl/8.7.1 Accept: / Content-Type: application/octet-stream Content-Length: 134217728 Expect: 100-continue

< HTTP/1.1 100 Continue HTTP/1.1 100 Continue <

• upload completely sent off: 134217728 bytes < HTTP/1.1 500 Internal Server Error HTTP/1.1 500 Internal Server Error < Content-Type: text/plain; charset=utf-8 Content-Type: text/plain; charset=utf-8 < X-Content-Type-Options: nosniff X-Content-Type-Options: nosniff < Date: Wed, 15 May 2024 18:28:41 GMT Date: Wed, 15 May 2024 18:28:41 GMT < Content-Length: 1561 Content-Length: 1561 <

couldn't upload object: failed to upload slab: launched=12 uploaded=4 remaining=2 inflight=0 pending=2 uploaders=12 errors=8 510b4467: failed to upload sector to contract fcid:7704449f0beb03195b3f05736e6765b047fb5617432e5280e7758817467487b4, err: AppendSector: WriteResponse: stream was gracefully closed 04e326a6: failed to upload sector to contract fcid:c26bfa9e44eda7ca6c194f2655c00855fda91bdf7392c5247f8b9d0fa2feab73, err: AppendSector: ReadResponse: stream was gracefully closed 96bf3099: failed to upload sector to contract fcid:d97168521e5c1d98d963c45d2f6a1cbf5e2a1528855a73e5ce2c6ef1140de82f, err: AppendSector: ReadResponse: stream was gracefully closed a3aef7db: failed to upload sector to contract fcid:a2d6b8aafbaf7904ac41281383ce5b046ec3fcf75f7668ac5177e4a2d9839206, err: AppendSector: ReadResponse: stream was gracefully closed 95770d3b: failed to upload sector to contract fcid:ede8e61c21c3e97ecc664ec419722a28e94dab2264e15b1f2b3cb3a63f283dce, err: AppendSector: ReadResponse: stream was gracefully closed a6711440: failed to upload sector to contract fcid:e01ce50140b1217d11f10c8eac50d324594c3cb1b73c0c261da5706613f0e162, err: AppendSector: ReadResponse: stream was gracefully closed eb51aa36: failed to upload sector to contract fcid:e6976f167fdb1e429381b23412cae19afad79a75aa1e9c5b4b69a6585e8867a7, err: AppendSector: WriteResponse: stream was gracefully closed 34d5b8b7: failed to upload sector to contract fcid:170dfa6b0df661c4f7fc77dcabe779bd6b6dbd11f8338cd748eccc528b6c0556, err: AppendSector: WriteResponse: stream was gracefully closed

• Connection #0 to host 127.0.0.1 left intact

Background: This used to happen a lot on a computer, so I was advised to try with another. I installed renterd it on another hardware and the error went away (for a while), until today. It seems to be random.

I did some debug with the browser and the error happened as well. The only difference is that the browser hides it and retries until it works.

Expected Behavior

Working

Steps to Reproduce

Upload a multipart using the API.

Version

1.0.5

What operating system did the problem occur on (e.g. Ubuntu 22.04, macOS 12.0, Windows 11)?

Mageia 9

Autopilot Config

{ "contracts": { "set": "autopilot", "amount": 12, "allowance": "2310000000000000000000000000", "period": 6048, "renewWindow": 2016, "download": 14000000000, "upload": 140000000000, "storage": 1000000000000, "prune": false }, "hosts": { "allowRedundantIPs": false, "maxDowntimeHours": 336, "minRecentScanFailures": 10, "scoreOverrides": null } }

Bus Config

{ "default": "autopilot" } { "hostBlockHeightLeeway": 6, "maxContractPrice": "15000000000000000000000000", "maxDownloadPrice": "6000000000000000000000000000", "maxRPCPrice": "1000000000000000000000", "maxStoragePrice": "115740740741", "maxUploadPrice": "300000000000000000000000000", "migrationSurchargeMultiplier": 10, "minAccountExpiry": 86400000000000, "minMaxCollateral": "10000000000000000000000000", "minMaxEphemeralAccountBalance": "1000000000000000000000000", "minPriceTableValidity": 300000000000 } { "minShards": 2, "totalShards": 6 } { "enabled": true, "slabBufferMaxSizeSoft": 4294967296 }

Contract Set Contracts

12

Anything else?

PJ told me to say I am a grantee.

[daniel-lucio]

Additional note: after trying several times (with failing results). I got this:

curl -v -i -u :f1l3m0n3lgr4nd3 -X PUT 'http://127.0.0.1:9880/api/worker/multipart/foo?bucket=default&uploadid=0477c079851130f86e7fe50196545dadb6fd5a219202d498f0c2e4e9beb17c05&partnumber=8' --data-binary "@/tmp/fileNUJUyt.7" -H "Content-Type: application/octet-stream"

• Trying 127.0.0.1:9880...
• Connected to 127.0.0.1 (127.0.0.1) port 9880
• Server auth using Basic with user ''

PUT /api/worker/multipart/foo?bucket=default&uploadid=0477c079851130f86e7fe50196545dadb6fd5a219202d498f0c2e4e9beb17c05&partnumber=8 HTTP/1.1 Host: 127.0.0.1:9880 Authorization: Basic OmYxbDNtMG4zbGdyNG5kMw== User-Agent: curl/8.7.1 Accept: / Content-Type: application/octet-stream Content-Length: 134217728 Expect: 100-continue

< HTTP/1.1 500 Internal Server Error HTTP/1.1 500 Internal Server Error < Content-Type: text/plain; charset=utf-8 Content-Type: text/plain; charset=utf-8 < X-Content-Type-Options: nosniff X-Content-Type-Options: nosniff < Date: Wed, 15 May 2024 20:35:25 GMT Date: Wed, 15 May 2024 20:35:25 GMT < Content-Length: 84 Content-Length: 84 < Connection: close Connection: close <

couldn't upload object: 3 < 6: not enough contracts to support requested redundancy

• Closing connection

[peterjan]

@daniel-lucio the first error indicates the caller cancelled the request, the second error indicates you don't have enough contracts. Even though your configuration is correct and from the first error I can see you had 12 contracts at one point. Do these error occur on the same node? Can you add your renterd.log. I don't really know how to help you out here, everything indicates it's a problem on your end rather than an issue with renterd. Can you also upgrade to 1.0.6 to ensure you're on the most recent version.

[daniel-lucio]

Hi

I had no Internet for almost a week. I will try to reproduce the issue and upload the renterd.log file. I will upgrade to 1.0.7 first.

[peterjan]

@daniel-lucio reported he can no longer reproduce the error and his renterd is running fine. I think this was likely related to a networking related issue on his side, maybe dropping connections or a DNS resolving issue perhaps. I'll close this out, I added a TODO here though to test the renter with throttled connection speed to see if I can reproduce it locally.