polkadot-sdk error[E0583]: file not found for module `target

Is there an existing issue?

[X] I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

[X] This is not a support question.

Description of bug

#10 1464.6    Compiling bp-messages v0.7.0 (/substrate/bridges/primitives/messages)
#10 1464.7 error[E0583]: file not found for module `target_chain`
#10 1464.7   --> bridges/primitives/messages/src/lib.rs:39:1
#10 1464.7    |
#10 1464.7 39 | pub mod target_chain;
#10 1464.7    | ^^^^^^^^^^^^^^^^^^^^^
#10 1464.7    |
#10 1464.7    = help: to create the module `target_chain`, create file "bridges/primitives/messages/src/target_chain.rs" or "bridges/primitives/messages/src/target_chain/mod.rs"
#10 1464.7    = note: if there is a `mod target_chain` elsewhere in the crate already, import it with `use crate::...` instead
#10 1464.7
#10 1466.2    Compiling pest_derive v2.7.2
#10 1466.3 For more information about this error, try `rustc --explain E0583`.
#10 1466.3 error: could not compile `bp-messages` (lib) due to 1 previous error

Steps to reproduce

build with the following using RUST_NIGHTLY=2024-04-29 or even the latest Rust version and it generates the error

cargo build --locked --release \
	-p polkadot-parachain-bin

May 13 '24 14:05 ltfschoen

The file exists, ensure that your git checkout is valid.

May 13 '24 14:05 bkchr

The file exists, ensure that your git checkout is valid.

so the way i checked that my git checkout was valid was to run the following:

check latest local commit

git rev-parse --short HEAD

which output fb7362f67e

check latest upstream commit

git ls-remote https://github.com/paritytech/polkadot-sdk master

which output fb7362f67e3ac345073b203e029bcb561822f09c refs/heads/master

i also tried removing the target folder rm -rf target/

i also changed to time docker build --no-cache ... instead of time docker build ... in substrate/docker/build.sh

i also updated it to use the latest Rust nightly RUST_NIGHTLY=2024-05-12

but despite all that i still got the error.

I've provided all the code that I'm using here https://github.com/paritytech/polkadot-sdk/pull/4450

I'm using Docker Desktop for Mac 4.22.0 (117440)

May 14 '24 01:05 ltfschoen

i tried doing a fresh clone of the polkadot-sdk of commit 594c3ed5750bc7ab97f82fb8387f82661eca1cc4

git clone https://github.com/paritytech/polkadot-sdk

then only copied the changes that i made in this PR https://github.com/paritytech/polkadot-sdk/pull/4450 to the following files:

substrate/docker/build.sh
substrate/docker/substrate_builder.Dockerfile

and then lastly ran the script

./substrate/docker/build.sh

but it gave the same error after 45 mins

#10 2072.7    Compiling bp-messages v0.7.0 (/substrate/bridges/primitives/messages)
#10 2073.2 error[E0583]: file not found for module `target_chain`
#10 2073.2   --> bridges/primitives/messages/src/lib.rs:39:1
#10 2073.2    |
#10 2073.2 39 | pub mod target_chain;
#10 2073.2    | ^^^^^^^^^^^^^^^^^^^^^
#10 2073.2    |
#10 2073.2    = help: to create the module `target_chain`, create file "bridges/primitives/messages/src/target_chain.rs" or "bridges/primitives/messages/src/target_chain/mod.rs"
#10 2073.2    = note: if there is a `mod target_chain` elsewhere in the crate already, import it with `use crate::...` instead

May 14 '24 11:05 ltfschoen

update: i get that error with the latest polkadot-sdk commit even when i use latest Rust stable (same as with latest Nightly) in this commit https://github.com/paritytech/polkadot-sdk/pull/4450/commits/bc8c9ebaa0bb14332950f02c56b6d3794e8c20de

May 14 '24 22:05 ltfschoen

i was able to bypass the error by only building the polkadot package by commenting out building the other ones too as shown in this commit so it only builds CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build --locked --release -p polkadot https://github.com/paritytech/polkadot-sdk/pull/4450/commits/fe6761dfd11da47a920bc8d3d74a8ec992508b57#diff-23f5aad22ff35defc4c554ae69e5aaf27a959db7a4a94123cb4b28feb0cc6c3eR62

so i need to find out which of these other packages is the one that is causing that error error[E0583]: file not found for module target_chain

polkadot-parachain-bin
staging-node-cli
staging-chain-spec-builder
subkey
solochain-template-node
parachain-template-node
minimal-template-node
test-parachain-adder-collator

May 14 '24 23:05 ltfschoen

@bkchr so i went through each package, and the only package that triggers the error error[E0583]: file not found for module target_chain is polkadot-parachain-bin. but only when you run ./substrate/docker/build.sh using the Dockerfile here https://github.com/paritytech/polkadot-sdk/pull/4450 when it configured so it runs the command CARGO_NET_GIT_FETCH_WITH_CLI=true cargo build --locked --release -p polkadot-parachain-bin here in that substrate/docker/substrate_builder.Dockerfile.

whereas if i just clone https://github.com/paritytech/polkadot-sdk, and install relevant dependencies and rustup update to use stable-aarch64-apple-darwin with rustc 1.78.0 (9b00956e5 2024-04-29) and run the same thing cargo build --locked --release -p polkadot-parachain-bin (without Docker) then it compiles successfully.

May 16 '24 07:05 ltfschoen

but only when you run ./substrate/docker/build.sh

Did you ensure that it copied the files correctly?

May 16 '24 12:05 bkchr

but only when you run ./substrate/docker/build.sh

Did you ensure that it copied the files correctly?

i think i'll try doing something like this https://stackoverflow.com/questions/545387/linux-compute-a-single-hash-for-a-given-folder-contents for the folder mentioned.

May 16 '24 16:05 ltfschoen

so since the issue is in the ./bridges folder, i tried running tar -C ./bridges2 -cf - --sort=name ./bridges2 | sha256sum on the host machine, which output

tar: Option --sort=name is not supported
Usage:
  List:    tar -tf <archive-filename>
  Extract: tar -xf <archive-filename>
  Create:  tar -cf <archive-filename> [filenames...]
  Help:    tar --help
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

and then i modified the ./substrate/docker/substrate_builder.Dockerfile so its content was

...
COPY . /substrate

RUN tar -C /substrate/bridges -cf - --sort=name /substrate/bridges | sha256sum
...

which output

#10 [builder 4/4] RUN tar -C /substrate/bridges -cf - --sort=name /substrate/bridges | sha256sum
#10 0.196 /bin/tar: Removing leading `/' from member names
#10 0.218 6ee34446bacdbf721940a1a68d176519b184b705d910540338b079309ea57b9d  -

so although it shows different hashes, the original tar executable on mac returned error tar: Option --sort=name is not supported, but that's important as highlighted here https://stackoverflow.com/a/64774578/3208553 that says "make sure you sort the path during tar, otherwise it is always different."

so i installed brew reinstall gnu-tar on macOS, which installed

gtar --version
tar (GNU tar) 1.35

and then ran

gtar -C $PWD/bridges2 -cf - --sort=name $PWD/bridges2 | sha256sum
gtar: Removing leading `/' from member names
97975462d46ed37284c3a0c7a7482bbacd8bbbc26ea8ecc1688db9978c86ba65  -

and then i modified the ./substrate/docker/substrate_builder.Dockerfile so its content was

...
COPY . /substrate

RUN apt-get update -y && \
	apt-get upgrade -y && \
	apt-get install tar -y && \
	tar --version && \
	tar -C /substrate/bridges -cf - --sort=name /substrate/bridges | sha256sum
...

which output the following when i ran ./substrate/docker/build.sh

#10 41.99 tar is already the newest version (1.34+dfsg-1+deb11u1).
#10 41.99 tar set to manually installed.
#10 41.99 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
#10 42.01 tar (GNU tar) 1.34
#10 42.01 Copyright (C) 2021 Free Software Foundation, Inc.
#10 42.01 License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
#10 42.01 This is free software: you are free to change and redistribute it.
#10 42.01 There is NO WARRANTY, to the extent permitted by law.
#10 42.01
#10 42.01 Written by John Gilmore and Jay Fenlason.
#10 42.03 /bin/tar: Removing leading `/' from member names
#10 42.05 6ee34446bacdbf721940a1a68d176519b184b705d910540338b079309ea57b9d  -

so i can't see any matching hash

then i tried doing the same for another random folder ./cumulus on host machine macOS 14.1.1

gtar -C $PWD/cumulus -cf - --sort=name $PWD/cumulus | sha256sum
gtar: Removing leading `/' from member names
8b972ba4965eb74dfa8b9b5ac113aeb1537aaaaaf62442ae0f85880432cfd6ed

and then i modified the ./substrate/docker/substrate_builder.Dockerfile so its content was

...
COPY . /substrate

RUN apt-get update -y && \
	apt-get upgrade -y && \
	apt-get install tar -y && \
	tar --version && \
	tar -C /substrate/cumulus -cf - --sort=name /substrate/cumulus | sha256sum
...

which output the following when i ran ./substrate/docker/build.sh

#10 41.51 tar is already the newest version (1.34+dfsg-1+deb11u1).
#10 41.51 tar set to manually installed.
#10 41.51 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
#10 41.53 tar (GNU tar) 1.34
#10 41.53 Copyright (C) 2021 Free Software Foundation, Inc.
#10 41.53 License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
#10 41.53 This is free software: you are free to change and redistribute it.
#10 41.53 There is NO WARRANTY, to the extent permitted by law.
#10 41.53
#10 41.53 Written by John Gilmore and Jay Fenlason.
#10 41.55 /bin/tar: Removing leading `/' from member names
#10 41.74 9f5a31b09f705275a6f274db983a2a0a3c74f361607c6caa6bc5fd3c5d81a0a7  -

so even the hash of that folder doesn't match

so i just created a random folder and file mkdir temp && touch ./temp/test.md then gtar -C $PWD/temp -cf - --sort=name $PWD/temp | sha256sum output ca2a581c71f2f44350749c78a99af825ebaa6561ee83e178c0cc7d27be020069

and then i modified the ./substrate/docker/substrate_builder.Dockerfile so its content was

...
COPY . /substrate

RUN apt-get update -y && \
	apt-get upgrade -y && \
	apt-get install tar -y && \
	tar --version && \
	tar -C /substrate/temp -cf - --sort=name /substrate/temp | sha256sum
...

and that output

1cab2f2809330a9205175c42d682d6fd54c65ab4f122cc9b37e61e7b8c39a5f9  -

and even that doesn't match.

so i'm not sure if i'm not sure if it's because i'm using tar 1.35 on the host machine macOS vs tar 1.34 in the Docker container linux, or due to some discrepency like mentioned here https://crypto.stackexchange.com/a/72300, or some other reason @bkchr

May 16 '24 22:05 ltfschoen

i just noticed this other open issue https://github.com/paritytech/polkadot-sdk/issues/582, i'm going to see if that encounters the same issue when running docker build -f ./docker/dockerfiles/polkadot-parachain/polkadot-parachain_builder.Dockerfile --tag polkadot:v1.0.0 ./, since that Dockerfile is similar.

May 17 '24 01:05 ltfschoen