Could not find the desired sector

[jkawamoto]

We've been getting Read: host rejected Read request: could not find the desired sector error after contract renewal. We could download the sector before renewal. The renewal happened because the funds were almost empty. I'm thus wondering if hosts remove sectors when the contract doesn't have enough funds before the period ends. In other words, if a contract has 3 month period but the renter uses up the funds on the first day. Does the host still hold the data in 3 months?

[jkawamoto]

Storage, actually we could download the data after contract renewal but suddenly all hosts start returning could not find the desired sector.

[MeijeSibbel]

@lukechampine do you have any clues as to why this might happen and what do we need to troubleshoot this issue?

[lukechampine]

I'm thus wondering if hosts remove sectors when the contract doesn't have enough funds before the period ends. In other words, if a contract has 3 month period but the renter uses up the funds on the first day. Does the host still hold the data in 3 months?

"Using up the funds" necessarily means paying the host to store the data for the contract period, so no, the host shouldn't delete any contract data.

actually we could download the data after contract renewal but suddenly all hosts start returning could not find the desired sector.

Can you elaborate on this? So you had a set of contracts with some hosts, you uploaded some data and downloaded it, then you renewed the contracts with all of the hosts, and then soon after the renewal the hosts (apparently) deleted the data? Are any of the sectors retrievable?

It might help to look up the contract renewals on SiaStats to make sure they were confirmed and that they reflect the intended amount of data.

[jkawamoto]

For example, we have a contract set with 40 hosts and upload a file, then we could download it multiple times but suddenly get this error:

too many hosts did not supply their shard (needed 5, got 2): 
0a0c32a6: Read: host rejected Read request: could not find the desired sector
1648ef6f: Read: host rejected Read request: could not find the desired sector
93338e1e: Read: host rejected Read request: could not find the desired sector
b287081a: Read: host rejected Read request: could not find the desired sector
1e871091: Read: WriteRequestID: write tcp 192.168.1.172:55478->84.6.49.164:9982: write: broken pipe
dfaca709: no record of that host
d40ade4f: Read: host rejected Read request: could not find the desired sector
51032222: Read: host rejected Read request: could not find the desired sector
0b78d138: Read: host rejected Read request: could not find the desired sector
fcafc04c: Read: host rejected Read request: could not find the desired sector
5f089782: Read: host rejected Read request: could not find the desired sector
33e5532d: Read: host rejected Read request: could not find the desired sector
b858f931: Read: host rejected Read request: could not find the desired sector
eedb600b: Read: host rejected Read request: could not find the desired sector
de517628: no record of that host
06f07d46: Read: host rejected Read request: could not find the desired sector
d304d705: Read: host rejected Read request: could not find the desired sector
fdaa28bc: Read: host rejected Read request: could not find the desired sector

It happens regardless of whether we renew contracts. ("no record of that host" is fine. That is because we disabled that hosts)

[lukechampine]

Hmm. This sounds like a host-side bug. I wonder if it's related to any recent changes in the host code. Are there any failing hosts that aren't on v1.5.5?

It might also be helpful to know if the failure always occurs after (e.g.) 5 downloads, or after 3 blocks, and if all of the hosts fail together (that is, as soon as one hosts fails, the rest will fail too) or if they fail separately.

[jkawamoto]

That happened a couple of days ago, but I remember most hosts run v1.5.5.

I'll dig into the log to find when it happens. But, suddenly we cannot download files anymore. That means, at least 35 hosts return "could not find the desired sector" together.

[jkawamoto]

too many hosts did not supply their shard (needed 5, got 2): 
c32db729: Read: host rejected Read request: could not find the desired sector
e93a0d98: Read: host rejected Read request: could not find the desired sector
eedb600b: Read: host rejected Read request: could not find the desired sector
ca57de76: Read: host rejected Read request: could not find the desired sector
33e5532d: no record of that host
b287081a: Read: host rejected Read request: could not find the desired sector
a049f625: no record of that host
5ddb2e3d: Read: host rejected Read request: could not find the desired sector
324ea0a0: no record of that host
21761371: Read: host rejected Read request: could not find the desired sector
5f089782: Read: host rejected Read request: could not find the desired sector
abdda1b3: Read: host rejected Read request: could not find the desired sector
886728ba: Read: host rejected Read request: could not find the desired sector
0a0c32a6: Read: host rejected Read request: could not find the desired sector
06f07d46: Read: host rejected Read request: could not find the desired sector
d304d705: Read: host rejected Read request: could not find the desired sector
f93161b3: Read: host rejected Read request: could not find the desired sector
b797bd4d: Read: host rejected Read request: could not find the desired sector

Related contract renewal:

• e93a0d98: 2h ago,
• 88b3c832: 2h ago, (sector still exists)
• d304d705: 3h ago,
• 3eeb53a6: 7h ago, (sector still exists)
• 0a0c32a6: 8h ago,
• d5c6b3aa: 22h ago, (sector still exists)
• c32db729: 32h ago.

I could get the shard around 48h ago.

[jkawamoto]

@lukechampine I found strange things. We got this error on 2021-04-01T23:22:16.754Z:

too many hosts did not supply their shard (needed 5, got 2):
06f07d46: Read: WriteRequestID: write tcp 143.110.217.205:51016->95.19.148.5:9982: write: broken pipe
0a0c32a6: Read: host rejected Read request: could not find the desired sector
21761371: Read: host rejected Read request: could not find the desired sector
324ea0a0: no record of that host
33e5532d: no record of that host
5ddb2e3d: Read: host rejected Read request: could not find the desired sector
5f089782: Read: host rejected Read request: could not find the desired sector
886728ba: Read: host rejected Read request: could not find the desired sector
a049f625: NewUnlockedSession: lookup werd.ddns.net on 127.0.0.53:53: no such host
abdda1b3: Read: host rejected Read request: could not find the desired sector
b287081a: Read: host rejected Read request: could not find the desired sector
b797bd4d: Read: host rejected Read request: could not find the desired sector
c32db729: Read: host rejected Read request: could not find the desired sector
ca57de76: Read: host rejected Read request: could not find the desired sector
d304d705: Read: host rejected Read request: could not find the desired sector
e93a0d98: Read: host rejected Read request: could not find the desired sector
eedb600b: Read: host rejected Read request: could not find the desired sector
f93161b3: Read: host rejected Read request: could not find the desired sector

and then we disabled 33e5532d and a049f625, and migrated the file with new 2 hosts. The migration succeeded. How can the migration recover the shard from 2 hosts?

The other strange thing is even the migration succeeded, we still cannot download the shard on 2021-04-02T22:58:02.850Z:

too many hosts did not supply their shard (needed 5, got 4):
06f07d46: Read: WriteRequestID: write tcp 143.110.217.205:48614->95.19.148.5:9982: write: broken pipe
0a0c32a6: Read: host rejected Read request: could not find the desired sector
21761371: Read: host rejected Read request: could not find the desired sector
324ea0a0: Read: host rejected Read request: could not find the desired sector
5ddb2e3d: Read: host rejected Read request: could not find the desired sector
5f089782: Read: host rejected Read request: could not find the desired sector
886728ba: Read: host rejected Read request: could not find the desired sector
abdda1b3: Lock: contract cannot be revised further
b287081a: Read: host rejected Read request: could not find the desired sector
b797bd4d: Read: host rejected Read request: could not find the desired sector
c32db729: Read: host rejected Read request: could not find the desired sector
ca57de76: Read: host rejected Read request: could not find the desired sector
d304d705: Read: WriteRequestID: write tcp 143.110.217.205:57992->75.64.241.254:9982: write: broken pipe
e93a0d98: Read: host rejected Read request: could not find the desired sector
eedb600b: Read: host rejected Read request: could not find the desired sector
f93161b3: Read: host rejected Read request: could not find the desired sector

(There were 4 shards existing but it's not enough)

[lukechampine]

Hmm. I wonder if the problem is actually that the wrong sectors are being requested from hosts?

You could check this (very inefficiently) by calling SectorRoots on all your hosts. Then, when a download fails, check to see if the root is present in the list of roots stored on that host. Alternatively, you could log the sector root (and host key) every time you upload and download, then compare the logs later when a download fails.

[jkawamoto]

I can check sectors manually. But, the strange thing is we could download the shard at least one time. The migration process doesn't change existing shard info in the meta DB, right?

Another possibility is sharing meta DB has a problem. But, I get 5 files broken out of 10000.

[jkawamoto]

Actually, sector roots are stored in the meta DB. I can dump them after uploads and before downloads.

[lukechampine]

Actually, sector roots are stored in the meta DB. I can dump them after uploads and before downloads

I recommend dumping them as close to the actual Upload/Download call as possible, just to rule out any issues with the DB (especially concurrency issues)