estuary icon indicating copy to clipboard operation
estuary copied to clipboard
verified publisher icon application-research

SPs continue to see commP errors on larger Estuary deals

Open stuberman opened this issue 3 years ago • 11 comments

Describe the bug SPs receiving large deals continue to see commP failures such as the one below: Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqadv2acolvsatmo6fqbmv6owgxd65uj5oatklm6dqq5xsun74s6ii, actual=baga6ea4seaqmjjmizj5fywuonasf2kozxfumae54nthan42u2ha3idrtfl35epq

To Reproduce Accept Estuary deals of all sizes I am running Lotus v1.16.0 and separate Boost v1.1.0 I do not see this behavior with other large deals from tools like bidbot.

Expected behavior I do not expect to see any commP failures after downloading any files.

Actual behavior Deal failures after download on larger deals (see table below)

Additional context Here is a list of recent Estuary deals as shown in Boost GUI from client ID f3vnq2cmwig3qjisnx5hobxvsd4drn4f54xfxnv4tciw6vnjdsf5xipgafreprh5riwmgtcirpcdmi3urbg36a Notice the deal sizes.

Start Deal ID Size Client State
4h a7cfdf89… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqadv2acolvsatmo6fqbmv6owgxd65uj5oatklm6dqq5xsun74s6ii, actual=baga6ea4seaqmjjmizj5fywuonasf2kozxfumae54nthan42u2ha3idrtfl35epq
9h c61790b5… 14.7 GiB f3vnq2cm… Ready to Publish
20h 7edfa8f7… 3.7 GiB f3vnq2cm… Sealer: Proving
20h 0399de10… 3.7 GiB f3vnq2cm… Sealer: Proving
20h 80f6acfa… 3.7 GiB f3vnq2cm… Sealer: Proving
20h 014d9b5b… 3.6 GiB f3vnq2cm… Sealer: Proving
20h 11482ba6… 3.7 GiB f3vnq2cm… Sealer: Proving
2 days dc2c2703… 422.5 MiB f3vnq2cm… Sealer: Proving
2 days b06156d9… 608.3 MiB f3vnq2cm… Sealer: Proving
2 days 366e1735… 315.5 MiB f3vnq2cm… Sealer: Proving
2 days c0eb8a98… 14.3 GiB f3vnq2cm… Sealer: Proving
2 days 7c67d49d… 14.3 GiB f3vnq2cm… Sealer: Proving
2 days 6c96c35e… 251.6 MiB f3vnq2cm… Sealer: Proving
2 days b938cc94… 14.3 GiB f3vnq2cm… Sealer: Proving
2 days ed3a9596… 311.7 MiB f3vnq2cm… Sealer: Proving
2 days 7360b74f… 841.6 MiB f3vnq2cm… Sealer: Proving
2 days bf62f2d9… 2.6 GiB f3vnq2cm… Sealer: Proving
2 days fc8ff993… 2.2 GiB f3vnq2cm… Sealer: Proving
2 days d6dcdbe7… 631.6 MiB f3vnq2cm… Sealer: Proving
2 days fc06934c… 620.2 MiB f3vnq2cm… Sealer: Proving
2 days 0dae69d3… 12.8 GiB f3vnq2cm… Sealer: Proving
2 days c3b2e876… 214.1 MiB f3vnq2cm… Sealer: Proving
2 days 09d521b3… 3.7 GiB f3vnq2cm… Sealer: Proving
2 days 14b6e1a7… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqpw63bkeitj4vduzu7uofarrue774ycvxblpxizelf2zgqffrgegy, actual=baga6ea4seaqdelfglbd77euwnmdrbeh2e5uikcrlhirkocj4tgcsipwr2apyyoa
2 days 59402905… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqkav3pbu75mojw74pfuu6fbub5nhdlhrlqqgtq5ym3laapgnmmmci, actual=baga6ea4seaqehgrejcvbc2m6qf2vntlzbjs5d65ftg7f3k6bhvpdzij7gziuigq
2 days 8cc2ecf8… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqacrhdn4isltc2vrek3hnicbbcnyh7ctvp4lmtjcmv3oz7ndbeqma, actual=baga6ea4seaqduq6b3sz22ngui7ort724jijpuuqy332s4vfawsssyx3osgis6aa
2 days 309a178d… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqe4mz5z6eeexs7hryf4gy34eeffsnmxfugqmuxogh7d4744lzl6li, actual=baga6ea4seaqf65yr3cfxyd4egbvcco7nhwv34elskfcurlun52vksg224xzdqhq
2 days 614bfd0d… 386.3 MiB f3vnq2cm… Sealer: Proving
2 days c31b7945… 14.2 GiB f3vnq2cm… Sealer: Proving
2 days a833852f… 506.2 MiB f3vnq2cm… Sealer: Proving
2 days 1d371b1c… 41.6 MiB f3vnq2cm… Sealer: Proving
2 days 9b15488c… 141 MiB f3vnq2cm… Sealer: Proving
2 days f3267ac0… 5.1 GiB f3vnq2cm… Sealer: Proving
2 days ac98582f… 14 GiB f3vnq2cm… Sealer: Proving
2 days f8c0fbb3… 162.9 MiB f3vnq2cm… Sealer: Proving
2 days 4d8f7b91… 5.7 GiB f3vnq2cm… Sealer: Proving
2 days fe3be4fe… 13.8 GiB f3vnq2cm… Sealer: Proving
2 days 2bb71026… 218.5 MiB f3vnq2cm… Sealer: Proving
2 days a442210c… 314.6 MiB f3vnq2cm… Sealer: Proving
3 days 73ef7a15… 8.5 GiB f3vnq2cm… Sealer: Proving
4 days 4785e707… 349.3 MiB f3vnq2cm… Sealer: Proving
4 days aa651bc3… 422.3 MiB f3vnq2cm… Sealer: Proving
4 days 813ed988… 13.9 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqmicpd6izlq3k2qfpu3npyapp2rvf7ar4xic5a22wqu3xvajvvioi, actual=baga6ea4seaqhb7ni666u2ojrpjlbngvo5mpvf6jpbqu7lqomqijphyv5chr4qni
4 days ab5cba7a… 438.2 MiB f3vnq2cm… Sealer: Proving
4 days 95605188… 288.4 MiB f3vnq2cm… Sealer: Proving
5 days 549273e1… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqm5ldqz3z2635jl5yi3422wixqtb5cd3b43wt6j62fp2ogrxwocfi, actual=baga6ea4seaqakoxqacilicux6chpp35gj3dofuzifxjb7nkrv42d6a5kghkq2ga
6 days 7bf59200… 539.8 MiB f3vnq2cm… Sealer: Proving
14 days d1c1d3a9… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqc27gdfi6yh7axui6ggtaf4alo7sadogcclpfzabbq7wnpj4njiha, actual=baga6ea4seaqi7wuxveewpeo736nwixna53akpb7hlpowzx4g7wksmljbzix24bi
14 days eddbf39b… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqjl3vljqnwvsv4x3bduhmuhweumgnplf7i7hl5dneqyjimektaubq, actual=baga6ea4seaqkll64dwo4fprfdlot5r6enctvep4rm7xlgr6ggibzgwhuspdlepq
14 days bfc2754b… 31.3 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqa46axaoh2xaqmoa62i5gw6lb4jnq4sxrn755rxaq63mcufzojqcq, actual=baga6ea4seaqfyth7qmyd4bwtpym7nize7fpgpcm4ctopyanxinkuto7qfavuici

stuberman avatar Jun 29 '22 15:06 stuberman

Hi @stuberman - thanks for your report. Quick clarifying question:

To Reproduce Accept Estuary deals of all sizes I am running Lotus v1.16.0 and separate Boost v1.1.0 I do not see this behavior with other large deals from tools like bidbot.

So if you use bidbot instead of boost, you are able to accept Estuary deals of all sizes, is that correct?

10d9e avatar Jul 29 '22 15:07 10d9e

That is not correct.
I use boostd v1.2.0 and deals from bidbot work. Large deals from Estuary error out with commP error.
Currently on lotus v1.16.1

stuberman avatar Jul 29 '22 15:07 stuberman

Bidbot requires either Boost or Marketv1 as does Estuary.

stuberman avatar Jul 29 '22 15:07 stuberman

I see exact same problem. All larger deals from Estuary contains incorrect data. This seems to be an estuary only problem, so it looks like the data preparation is broken.

Daemon:  1.17.0-rc3+mainnet+git.9794652e0+api1.5.0
Local: lotus-miner version 1.17.0-rc3+mainnet+git.9794652e0
boostd version 1.3.0-rc1+git.b85d1b9
Start Deal ID Size Client State
2022-07-29 08:48:53.992 d51c4b5b… 28.5 GiB f3vnq2cm… Transferring 3%
2022-07-29 06:52:48.520 3a1515c3… 25.3 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqcjpm6ix22557mcwju627uqmdx7isvjatqqxcatcjhnovky7xrufi, actual=baga6ea4seaqh5xs56r3uuksb56bsalcjd6jkeivzno7icbsd5pwgiaxxedrjwmq
2022-07-29 06:38:06.658 de749314… 31.2 GiB f3vnq2cm… Transferring 75%
2022-07-29 02:19:17.662 651bfbd1… 26.2 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqbcseenju34x57zfdjhs3qvrgvl5qltd5w7yt4srur73s5jduikpy, actual=baga6ea4seaqashs3x4fqfddnok4gcri23jfhkjn45ox5uer2z4ddwhuruknmigq
2022-07-29 01:06:34.288 adb310cd… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqbdxprl3kjamommynqdll3ttciktntvj5omiswmkknhfdrjkloqfa, actual=baga6ea4seaqc3jv5dmqwjggwdutqbpkcsm5vmpyb5fquwh7ob4ybzrpw6hpocay
2022-07-29 00:56:04.560 e9d42913… 18.7 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqb42ha3qwu2d4z5tujtiu2zezus2z2f5oav5vhsycaxi3si52mcli, actual=baga6ea4seaqgfx7po3epse66662eiefysjkqlr7w6xmeyms4as7546gg44wwadi
2022-07-28 22:44:18.216 06b5a90f… 12.8 GiB f3vnq2cm… Ready to Publish
2022-07-28 20:59:17.301 d79f9078… 7.3 GiB f3vnq2cm… Ready to Publish
2022-07-28 20:59:13.170 18d5cc6e… 23.6 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqi2nbxna6a2w5pvtpyvcosl2jy75ubhyg35w6v5admlqdhtgraelq, actual=baga6ea4seaqhzit32ccns7rxgtebawzaicghrselbs7dg62ealowmo7mdorwmji
2022-07-28 19:59:49.621 dc3b15ea… 6.6 GiB f3vnq2cm… Ready to Publish
2022-07-28 19:59:12.372 c4a82708… 4.3 GiB f3vnq2cm… Ready to Publish
2022-07-28 16:55:30.362 6e358181… 14.3 GiB f3vnq2cm… Ready to Publish
2022-07-28 15:05:23.496 f78eaa3b… 24.3 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqac7ad5ff7tk33fdahbdwpdp7kbpkrhuaycfqj3mzmkm5n4qydscq, actual=baga6ea4seaqpxugugpbpanhz7as3sjvaha3ep53rnp3sgbouf2lsqfbdckt2opy
2022-07-28 01:21:31.862 5fafd26d… 11.1 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqkhroq3n7worouhqv6ti64q4hlpuyl6xqi2htvtkmpymme7y7oyni, actual=baga6ea4seaql45n5kcwcqdwo5pqeeauwhg7lyprrxj3nq33dc3zwvo3zhgefkpi
2022-07-28 00:54:54.237 d7e289e6… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqbs6g6byoesmwajsmxwlouwrmmutdrm7l2s4o5ok46jg5yjvjloia, actual=baga6ea4seaqixc6dm7yejktvpil2m7uooikt5job2amleolleip4z2zsczl7aiq
2022-07-27 23:01:24.950 d15cd316… 6.1 GiB f3vnq2cm… Sealer: Proving
2022-07-27 21:38:40.142 c066c032… 13.8 GiB f3vnq2cm… Sealer: Proving
2022-07-27 20:44:27.242 8b11decb… 8.4 GiB f3vnq2cm… Sealer: Proving
2022-07-27 18:27:51.018 bf472fcc… 24 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqhffayb5dbrlzcyfktc75edsojdig2v4xc3ryq5ipacjc762nlsfy, actual=baga6ea4seaqeu4hk27hvahyur3kzz4zyhtpxxftjwvbwwemnviprcpepjr4tcia
2022-07-27 17:29:37.429 8d69a1d9… 9.6 GiB f3vnq2cm… Sealer: Proving

benjaminh83 avatar Jul 29 '22 16:07 benjaminh83

Another storage provider seeing the CommP error:

I get CommP errors daily...
f01611097
lotus version 1.16.0+mainnet+git.01254ab32
boostd version 1.2.0+git.c355880
3 CommPs today
Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqb42ha3qwu2d4z5tujtiu2zezus2z2f5oav5vhsycaxi3si52mcli, actual=baga6ea4seaqgfx7po3epse66662eiefysjkqlr7w6xmeyms4as7546gg44wwadi

Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqe4mz5z6eeexs7hryf4gy34eeffsnmxfugqmuxogh7d4744lzl6li, actual=baga6ea4seaqf65yr3cfxyd4egbvcco7nhwv34elskfcurlun52vksg224xzdqhq

Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqh2z7gvx2akjhhc5uaaxdmzmufztontyfv3itf7lzzfkwmpiyfkky, actual=baga6ea4seaqifceptkbhcbegmaul2cfpixnfcmwkm6pjyvzzanaxk54txtdhghq

Confirmed in logs:

2022-07-29 16:34:57 | 2022-07-29T19:34:57.853Z	ERROR	estuary	estuary/replication.go:1606	deal state for deal 85d4ae75-9fb5-43e0-8481-282646f38ba9 from miner f01611097 is error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqe4mz5z6eeexs7hryf4gy34eeffsnmxfugqmuxogh7d4744lzl6li, actual=baga6ea4seaqf65yr3cfxyd4egbvcco7nhwv34elskfcurlun52vksg224xzdqhq	{"app_version": "v0.1.7-dirty"}
2022-07-29 16:34:47 | 2022-07-29T19:34:47.199Z	ERROR	estuary	estuary/replication.go:1606	deal state for deal 29c6c819-efe5-45cb-abce-060e62419709 from miner f01611097 is error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqb42ha3qwu2d4z5tujtiu2zezus2z2f5oav5vhsycaxi3si52mcli, actual=baga6ea4seaqgfx7po3epse66662eiefysjkqlr7w6xmeyms4as7546gg44wwadi	{"app_version": "v0.1.7-dirty"}
2022-07-29 15:50:53 | 2022-07-29T18:50:53.658Z	ERROR	estuary	estuary/replication.go:1606	deal state for deal 6f12643a-b206-41be-8f7d-0bcaf8569170 from miner f01611097 is error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqh2z7gvx2akjhhc5uaaxdmzmufztontyfv3itf7lzzfkwmpiyfkky, actual=baga6ea4seaqifceptkbhcbegmaul2cfpixnfcmwkm6pjyvzzanaxk54txtdhghq	{"app_version": "v0.1.7-dirty"}
2022-07-29 15:50:04 | 2022-07-29T18:50:03.856Z	ERROR	estuary	estuary/replication.go:1606	deal state for deal 59f294d7-a09d-424d-9608-c135f0722a8e from miner f01611097 is error: Cancelled	{"app_version": "v0.1.7-dirty"}

10d9e avatar Jul 29 '22 19:07 10d9e

I think I just got couple of larger deals here for troubleshooting. 0% success rate :(

Start Deal ID Size Client State
2022-07-29 09:02:56.107 4a0650a7… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqgvpbar337irzejjaxelbrpr3ksi6jgsmokdlskehy2a4v3435ima, actual=baga6ea4seaqjtqdvfxdw6isigyvhrv6xp3sbpcps5q3bx5s5l3nnux3dubzc4ja
2022-07-29 08:48:53.992 d51c4b5b… 28.5 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqgpshwczzwrhagvf6qnq4kyzps4uby65ftv6xe7yebtuiz743u6cq, actual=baga6ea4seaqhxinl4itmfpdxoe35llitsthp5bnpreht6e2gdsbmp5nkrgrtgay
2022-07-29 06:52:48.520 3a1515c3… 25.3 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqcjpm6ix22557mcwju627uqmdx7isvjatqqxcatcjhnovky7xrufi, actual=baga6ea4seaqh5xs56r3uuksb56bsalcjd6jkeivzno7icbsd5pwgiaxxedrjwmq
2022-07-29 06:38:06.658 de749314… 31.2 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqew4sucbllnmstfgz3mzcrbukh4urvsz6j7lb7ghdsaz3uixx4enq, actual=baga6ea4seaqjt224npvtney5yhncevtfjjqatpqmmuehfxiudkff64apmwfu6mi
2022-07-29 02:19:17.662 651bfbd1… 26.2 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqbcseenju34x57zfdjhs3qvrgvl5qltd5w7yt4srur73s5jduikpy, actual=baga6ea4seaqashs3x4fqfddnok4gcri23jfhkjn45ox5uer2z4ddwhuruknmigq
2022-07-29 01:06:34.288 adb310cd… 31.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqbdxprl3kjamommynqdll3ttciktntvj5omiswmkknhfdrjkloqfa, actual=baga6ea4seaqc3jv5dmqwjggwdutqbpkcsm5vmpyb5fquwh7ob4ybzrpw6hpocay
2022-07-29 00:56:04.560 e9d42913… 18.7 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqb42ha3qwu2d4z5tujtiu2zezus2z2f5oav5vhsycaxi3si52mcli, actual=baga6ea4seaqgfx7po3epse66662eiefysjkqlr7w6xmeyms4as7546gg44wwadi

benjaminh83 avatar Jul 29 '22 22:07 benjaminh83

Steps to replicate:

  1. Pull down one of the failing car files with ipfs dag export, ie.ipfs dag export bafybeietzjzc4vudlevv6k6sxdixhp5nnmfblyjhqheyjyd4d3uluvqdgm > output.car
  2. Add car file to estuary via content/add-car
  3. Once the content is loaded, explicitly make a deal with miner where the problem is occurring.
  4. Wait for the deal to fail.

cc @stuberman

10d9e avatar Jul 31 '22 01:07 10d9e

Upon receiving the staging file (1e347bf0-ce28-47a2-bf15-59472f4f87cf.download) from @stuberman I was able to recreate the issue by composing a small program that emulates what boost does when it generates a commP. ( within the filecoin-project/boost project space to preserve module dependencies )

package main

import (
	"fmt"

	m "github.com/filecoin-project/boost/storagemarket"
	"github.com/filecoin-project/go-state-types/abi"
	"github.com/ipfs/go-cid"
)

func main() {

	InboundFilePath := "1e347bf0-ce28-47a2-bf15-59472f4f87cf.download"
	var PieceSize abi.PaddedPieceSize = 34359738368

	// expected from Estuary
	clientPieceCid, err := cid.Decode("baga6ea4seaqadv2acolvsatmo6fqbmv6owgxd65uj5oatklm6dqq5xsun74s6ii")
	if err != nil {
		panic(err)
	}

	pieceCid, err := m.GeneratePieceCommitment(InboundFilePath, PieceSize)
	if err != nil {
		fmt.Printf("failed to generate CommP: %s\n", err)
	}

	fmt.Println(pieceCid.String())
	if pieceCid != clientPieceCid {
		fmt.Printf("commP mismatch, expected=%s, actual=%s\n", clientPieceCid, pieceCid)
	}
}

giving us the expected output, that also came out of boostd:

commP mismatch, expected=baga6ea4seaqadv2acolvsatmo6fqbmv6owgxd65uj5oatklm6dqq5xsun74s6ii, actual=baga6ea4seaqmjjmizj5fywuonasf2kozxfumae54nthan42u2ha3idrtfl35epq

However, when I ran the same code against the original CAR file, there was no error, the piece commitment was generated correctly (baga6ea4seaqadv2acolvsatmo6fqbmv6owgxd65uj5oatklm6dqq5xsun74s6ii)

Upon further investigation, I noticed that the .download file received by boost and the original CAR file were identical in size, however their contents differed, confirmed with a simple sum call:

boost's download file:

% ls -slap
33756598209 30 Jul 22:51 1e347bf0-ce28-47a2-bf15-59472f4f87cf.download

% shasum -a 256 1e347bf0-ce28-47a2-bf15-59472f4f87cf.download
6465fcb4b9f4641a8572635a6e14d4d9ec5751078f7e2b9325a619cd63a6fc1f

original CAR file:

% ls -salp ~/test-cars/output.car
33756598209 Jul 30 19:23 /root/test-cars/output.car

% shasum -a 256 ~/test-cars/output.car
7d1896d0b758cfc3f4951d5adf66cc1114517f753276c55d2b903aa16e3aa2e7

This implies that somehow the CAR's contents are mutated in transit/process from Estuary to the filesystem through Boost, explaining the sum / commP discrepancies.

10d9e avatar Jul 31 '22 02:07 10d9e

@jlogelin can you use the go-car cli util to inspect the blocks for both car files and do a diff

en0ma avatar Jul 31 '22 09:07 en0ma

@en0ma Confirmed that the block CIDS are identical, they just aren't in the same order:

list1 = []
list2 = []
with open('output.car.out') as car_out:
    for line in car_out:
        list1.append(line.rstrip('\n').split(','))
with open('1e347bf0-ce28-47a2-bf15-59472f4f87cf.out') as car_out:
    for line in car_out:
        list2.append(line.rstrip('\n').split(','))

print("Unsorted equality: ", list1 == list2)       
print("Sorted equality: ",sorted(list1) == sorted(list2))

Output

% python3 check_contents.py
Unsorted equality:  False
Sorted equality:  True

10d9e avatar Aug 02 '22 01:08 10d9e

https://github.com/filecoin-project/boost/issues/673

10d9e avatar Aug 02 '22 12:08 10d9e

@stuberman are you still experiencing this, so I know if we can close this ticket

en0ma avatar Oct 14 '22 09:10 en0ma

I am unable to tell, most Estuary deals fail before they download for the last month.

stuberman avatar Oct 18 '22 13:10 stuberman

I agree with @stuberman. Download errors from Estuary is now so dominant that it's quite hard to test for this issue. I don't know how it can run so broken, but apparently it keeps going like this for weeks. I think most SPs are probably not complaining, as this is like a never ending story :(

See my resent boost logs here (Its somewhat impressive):

Start Deal ID Size Client State
5h 5611ecd9… 8.8 GiB f3vnq2cm… Transferring 5%115.6 kbps
6h 98854656… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
7h 03a1de59… 5.4 GiB f3vnq2cm… Error: failed to verify CommP: commP mismatch, expected=baga6ea4seaqg4orda5lc66nhby6x2ykowrrnegip6c75wtomxi3nm64nfw6cola, actual=baga6ea4seaqkew4s6j3q55hwsbzchzok7cjiaalvv73jj7gdbuyvv6pulnb6kky
8h 9b1a9e10… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
8h 5d71711a… 8.8 GiB f3vnq2cm… Transferring 9%54.5 kbps
12h c19cc24d… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
12h 9f6ea8d9… 8.8 GiB f3vnq2cm… Transferring 14%201.0 kbps
13h c7314ff3… 14.3 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
13h e24c611f… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
13h e779c91b… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
14h c711e794… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
14h 8ce6bd40… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
14h d8c71f6b… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
14h 50e2d38a… 13.9 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
15h a66f4af6… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
15h ee291f42… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
16h 01f31232… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
16h c21f64da… 14.3 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
17h a183a936… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
18h ba8e7cc2… 14.2 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
18h e6a54097… 14.2 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
19h dfce83a9… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
19h 9096ad6d… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
20h aff09170… 8.8 GiB f3vnq2cm… Sealer: PreCommit1
20h 903eabb1… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 16e77b34… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 02b2ba00… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day df4947c7… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 2dde267f… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 6e79a8d3… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 68a7b013… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day a6a08f3d… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day cb8d15df… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day d3348f3f… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 7b2d6d11… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day df291ee3… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 1edbd716… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day ff9a4b6f… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 38093cd5… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 19ec980f… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day f17c4b1a… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day e763c9ed… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day a1bc7910… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 820651cb… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day ea60f4a5… 4.1 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 61d75905… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 20c2ac21… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF
a day 8d003e9a… 8.8 GiB f3vnq2cm… Error: data-transfer failed: could not finish transfer even after 15 attempts, lastErr: error reading from http response stream: unexpected EOF

benjaminh83 avatar Oct 18 '22 13:10 benjaminh83