lotus icon indicating copy to clipboard operation
lotus copied to clipboard

Published 20 hours ago • verified publisher icon filecoin-project

[Sealing Issue] Failed to acquire sector: Conection refused when fetching from PC1 -> PC2

Open aaron3481 opened this issue 3 years ago • 3 comments

When sealing task go from PC1 to PC2, the miner's log repeat showing: seal pre commit(1) failed: storage call error 0: failed to acquire sector and then PC1 restart. I am pretty sure connection between miner and worker is fine. Here is part of miner's log:

  1. 2021-07-18 22:06:25 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  2. 2021-07-18 22:06:25 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 232} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-232] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-232 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-232: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-232": dial tcp 127.0.0.1:3456: connect: connection refused

  3. 2021-07-18 22:07:25 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  4. 2021-07-18 22:07:25 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 232} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-232] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-232 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-232: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-232": dial tcp 127.0.0.1:3456: connect: connection refused

  5. 2021-07-18 22:08:25 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  6. 2021-07-18 22:08:25 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 232} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-232] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-232 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-232: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-232": dial tcp 127.0.0.1:3456: connect: connection refused

  7. 2021-07-18 22:09:25 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  8. 2021-07-18 22:09:25 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 232} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-232] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-232 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-232: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-232": dial tcp 127.0.0.1:3456: connect: connection refused

  9. 2021-07-18 22:10:25 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  10. 2021-07-19 00:23:11 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: task aborted

  11. 2021-07-19 00:24:11 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  12. 2021-07-19 00:24:11 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 232} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-232] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-232 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-232: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-232": dial tcp 127.0.0.1:3456: connect: connection refused

  13. 2021-07-19 00:25:11 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  14. 2021-07-19 00:25:11 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 232} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-232] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-232 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-232: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-232": dial tcp 127.0.0.1:3456: connect: connection refused

  15. 2021-07-19 00:26:11 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  16. 2021-07-19 00:26:11 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 232} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-232] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-232 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-232: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-232": dial tcp 127.0.0.1:3456: connect: connection refused

  17. 2021-07-19 00:30:40 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  18. 2021-07-19 00:34:31 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 232} from remote(1): sector not found

  19. 2021-07-19 00:38:01 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

Version

The output of lotus --version. Daemon: 1.10.1+mainnet+git.d920b47e7+api1.3.0 Local: lotus version 1.10.1+mainnet+git.d920b47e7

Setup

Miner and worker at the same machine. miner do not accept any tasks CPU: AMD EPYC 7502 32-Core RAM: 512GB GPU: RTX 3080

Sectors status

  1. 2021-07-18 22:06:19 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  2. 2021-07-18 22:06:19 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 230} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-230] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-230 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-230: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-230": dial tcp 127.0.0.1:3456: connect: connection refused

  3. 2021-07-18 22:07:19 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  4. 2021-07-18 22:07:19 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 230} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-230] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-230 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-230: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-230": dial tcp 127.0.0.1:3456: connect: connection refused

  5. 2021-07-18 22:08:19 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  6. 2021-07-18 22:08:19 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 230} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-230] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-230 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-230: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-230": dial tcp 127.0.0.1:3456: connect: connection refused

  7. 2021-07-18 22:09:19 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  8. 2021-07-19 00:23:03 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: task aborted

  9. 2021-07-19 00:24:03 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  10. 2021-07-19 00:24:03 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 230} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-230] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-230 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-230: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-230": dial tcp 127.0.0.1:3456: connect: connection refused

  11. 2021-07-19 00:25:03 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  12. 2021-07-19 00:25:03 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 230} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-230] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-230 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-230: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-230": dial tcp 127.0.0.1:3456: connect: connection refused

  13. 2021-07-19 00:26:03 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  14. 2021-07-19 00:26:03 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 230} from remote (tried [{914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4 [http://127.0.0.1:3456/remote/unsealed/s-t01081349-230] 10 true false false}]): 1 error occurred:

    • fetch error http://127.0.0.1:3456/remote/unsealed/s-t01081349-230 (storage 914a85e0-f66f-4bd7-88ec-d1cc32c9eaa4) -> /home/filworker1/seal/unsealed/fetching/s-t01081349-230: do request: Get "http://127.0.0.1:3456/remote/unsealed/s-t01081349-230": dial tcp 127.0.0.1:3456: connect: connection refused

  15. 2021-07-19 00:27:03 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

  16. 2021-07-19 00:34:31 +0000 UTC: [event;sealing.SectorSealPreCommit1Failed] {"User":{}} seal pre commit(1) failed: storage call error 0: failed to acquire sector {1081349 230} from remote(1): sector not found

  17. 2021-07-19 00:38:01 +0000 UTC: [event;sealing.SectorRetrySealPreCommit1] {"User":{}}

aaron3481 avatar Jul 19 '21 01:07 aaron3481

Absolutely the same issue. Can it be because of different usernames on workers?

Shekelme avatar Sep 26 '21 20:09 Shekelme

But this worker can process C2 tasks successfully. Lotus v1.11.2

Shekelme avatar Sep 29 '21 09:09 Shekelme

There may be a similar problem on the newest release candidate: https://github.com/filecoin-project/lotus/discussions/8578

Shekelme avatar May 03 '22 04:05 Shekelme