feat: rebuild only necessary canisters

[vporton]

Description

DFX doesn't compile canisters for which all dependencies are elder than the .wasm file. This results in big compilation speedups.

The PR also adds tracing and comments intended for future improvement of compilation speed.

This PR is a WIP. There is potential for further speedup and there is missing logging. More testing is necessary.

How Has This Been Tested?

I run it before the .wasm file has been created and after it with a significant speedup. Also dfx build -vv correctly does not output any moc compilation logs, when invoking dfx build for the second time.

Later I did bats make_like.bash, add_dependency.bash, broken_canister_dep.bash, and dotenv.bash automated testing.

Checklist:

• [x] The title of this PR complies with Conventional Commits.
• [x] I have edited the CHANGELOG accordingly.
• [x] I have made corresponding changes to the documentation.

[vporton]

dfx deploy of my current DFX version running in https://github.com/vporton/zondirectory2 repo causes an infinite loop (but dfx build --all doesn't).

[vporton]

The last commit (79faa772dd3b8a7683ed9670a1447ecd0b2893ba) seems working!

Also note that unlike the original (master) code, my code does recompile dependent canisters (canister:* dependencies) when a dependency changed.

The code needs refactoring and more serious testing.

[vporton]

Testing shows a bug: When specifying a dependency to build: dfx build dependency, the dependent canister may recompile, too.

[vporton]

The latest version commit 030f9af902515edd70d60acde5d2cb547c200b40 works as expected on simple (manual) tests (with more tests than the past time).

You may try to use it for your home projects, but don't try it for mission-critical tasks or if you fear to be lost in wrong recompilations, because errors are still possible.

[vporton]

Fixed crash on dfx build --all (creates a new bug, about compiling non-Motoko canister: dependencies).

[vporton]

Automated tests in make_like.bash passed. There are yet several FIXME and TODO in the source.

[vporton]

Sorry for the confusion: Some versions above were non-functional. The last version passed my test make_like.bash. Yet need to test for the case when zero arguments to dfx build or dfx deploy provided. There has been no testing at all for dfx generate.

[vporton]

The apparently working version is created. Please, do code review.

It can be tested by bats tests-dfx/make_like.bash.

[vporton]

When trying to compile https://github.com/vporton/zondirectory2 by my updated DFX, errors show up. Additional research is needed.

[vporton]

New e2e test add_dependency.bash fails. Need to correct it before merging to master.

[vporton]

New e2e test add_dependency.bash fails. Need to correct it before merging to master.

Apparently I just forgot to recompile dfx before testing. It works now. Probably, you should now merge to master.

[vporton]

When I tried to compile a real software, it still bugs.

[vporton]

I've found an error: My function traverse2 in src/dfx/src/lib/graph/traverse_filtered.rs is totally wrong.

[vporton]

The bug is fixed. It has been tried to compile real software with updated DFX and it seems to work (except that the "real software" is now not fully compilable because of frontend TypeScript errors).

Probably, appropriate to merge this to master.

[vporton]

After updating DFX to my version in a real project MS Code produces error message like file "/home/porton/Projects/zondirectory2/.dfx/local/lsp/be2us-64aaa-aaaaa-qaabq-cai.did" does not exist for import order "canister:order"; (despite I did deploy the order canister). I poorly understand what LSP is and how it should be used.

Please, help to debug.

[vporton]

After updating DFX to my version in a real project MS Code produces error message like file "/home/porton/Projects/zondirectory2/.dfx/local/lsp/be2us-64aaa-aaaaa-qaabq-cai.did" does not exist for import order "canister:order"; (despite I did deploy the order canister). I poorly understand what LSP is and how it should be used.

Please, help to debug.

https://github.com/dfinity/sdk/discussions/3732 shows a related hard to reproduce (it seems not to appear most of the time) and hard to understand why bug. Any ideas?

[vporton]

After updating DFX to my version in a real project MS Code produces error message like file "/home/porton/Projects/zondirectory2/.dfx/local/lsp/be2us-64aaa-aaaaa-qaabq-cai.did" does not exist for import order "canister:order"; (despite I did deploy the order canister). I poorly understand what LSP is and how it should be used. Please, help to debug.

#3732 shows a related hard to reproduce (it seems not to appear most of the time) and hard to understand why bug. Any ideas?

I reproduced this bug not only with my development version but also with stable 0.19.0: https://github.com/dfinity/sdk/issues/3733

[vporton]

Attempt to import undefined canister: causes a backtrace.

[vporton]

After updating DFX to my version in a real project MS Code produces error message like file "/home/porton/Projects/zondirectory2/.dfx/local/lsp/be2us-64aaa-aaaaa-qaabq-cai.did" does not exist for import order "canister:order"

Bug fixed.

[vporton]

Attempt to import undefined canister: causes a backtrace.

Bug fixed.

So, all known bugs are fixed. Please, review code and merge to master.

Meanwhile, I will try to use new DFX in my other project.

[vporton]

Sorry, testing on a real use case:

thread 'main' panicked at src/dfx/src/lib/canister_info.rs:227:17: It is only valid to call get_canister_id after setting the canister id.

[vporton]

thread 'main' panicked at src/dfx/src/lib/canister_info.rs:227:17: It is only valid to call get_canister_id after setting the canister id.

This was an incompatibility with #3719 - fixed.

Also fixed several bugs with the above commits. After this I succeeded to use new DFX to compile https://github.com/vporton/zondirectory2

[vporton]

"Field testing" trying to compile https://github.com/vporton/zondirectory2 triggers a bug:

dfx deploy -vv
...
src/backend/main.mo:9.1-9.30: import error [M0009], file ".dfx/local/lsp/be2us-64aaa-aaaaa-qaabq-cai.did" does not exist
src/backend/main.mo:14.1-14.40: import error [M0009], file ".dfx/local/lsp/bd3sg-teaaa-aaaaa-qaaba-cai.did" does not exist

[vporton]

After fixing a bug, https://github.com/vporton/zondirectory2 successfully compiled. There are no known bugs introduced in this PR ATM. However, it needs refactoring.

[vporton]

Ugh, compiling zondirectory2 still bugs.

[vporton]

Ugh, compiling zondirectory2 still bugs.

The bug was fixed.

Now this PR has no remaining known bugs except that it seems to read Motoko dependencies from this two or more times (on deploy). Also the function add_imports needs refactoring.

[vporton]

I've fixed the issue of calling moc --print-deps more than once for a given file. However a small issue remains: it outputs Building dependencies graph. twice on dfx deploy.

No other remaining issues. What's about code review?

[ericswanson-dfinity]

I've fixed the issue of calling moc --print-deps more than once for a given file. However a small issue remains: it outputs Building dependencies graph. twice on dfx deploy.

No other remaining issues. What's about code review?

These are the kinds of issues I see that indicate the work is not yet ready for code review:

• numerous // TODO
• commented-out code
• panic!("programming error");
• error checking with assert! rather than returning an error

It also adds a new feature, deploy: bool for canisters, that we haven't discussed and should be first submitted as a separate PR anyway.

[vporton]

This MR has been "battle-tested" a little. It worked as expected in developing https://github.com/vporton/zondirectory2 and in testing of https://github.com/vporton/joining-proxy-rust without any visible errors.

[vporton]

There has been found yet a bug:

Compiling https://github.com/vporton/zondirectory2/commit/b145a4c57de8c28b34dd64ffbd4bace9bc607f2b it was output:

...

The command '"moc-wrapper" "src/backend/personhood.mo" "-o" "/home/porton/Projects/zondirectory/.dfx/local/canisters/personhood/personhood.wasm" "-c" "--debug" "--idl" "--stable-types" "--public-metadata" "candid:service" "--public-metadata" "candid:args" "--actor-idl" "/home/porton/Projects/zondirectory/.dfx/local/canisters/idl/" "--actor-alias" "CanDBIndex" "bkyz2-fmaaa-aaaaa-qaaaq-cai" "--actor-alias" "ic_eth" "avqkn-guaaa-aaaaa-qaaea-cai" "--package" "base" ".mops/[email protected]/src" "--package" "candb" ".mops/_github/candb#no-blob/src" "--package" "nacdb" ".mops/[email protected]/src" "--package" "stable-rbtree" ".mops/[email protected]/src" "--package" "stable-buffer" ".mops/_github/stable-buffer#v0.2.0/src" "--package" "prng" ".mops/[email protected]/src" "--package" "stableheapbtreemap" ".mops/[email protected]/src" "--package" "sha2" ".mops/[email protected]/src" "--package" "stablebuffer" ".mops/_github/stablebuffer#v0.2.0/src" "--package" "StableBuffer" ".mops/_github/StableBuffer#v0.2.0/src" "--package" "icrc1" ".mops/[email protected]/src" "--package" "array" ".mops/_github/array#main/src" "--package" "StableTrieMap" ".mops/_github/StableTrieMap#main/src" "--package" "itertools" ".mops/[email protected]/src" "--package" "base-0.7.3" ".mops/_github/base-0.7.3#aafcdee0c8328087aeed506e64aa2ff4ed329b47/src" "--package" "xtendedNumbers" ".mops/_github/xtendedNumbers#v1.1.0/src" "--package" "motoko-lib" ".mops/_github/motoko-lib#0.7/src" "--package" "map7" ".mops/_github/map7#v7.0.0/src" "--package" "sha" ".mops/_github/sha#master/src" "--package" "matchers" ".mops/_github/matchers#v1.3.0/src" "--package" "map" ".mops/[email protected]/src" "--package" "btree" ".mops/_github/btree#v0.3.3/src" "--package" "CanDBMulti" ".mops/_github/CanDBMulti#v0.10.0@86d00711216349ba4879370d6313d44a26d16f61/src" "--package" "xtended-numbers" ".mops/[email protected]/src" "--package" "json.mo" ".mops/[email protected]/src" "--package" "parser-combinators" ".mops/_github/parser-combinators#v0.1.2/src" "--package" "nacdb-reorder" ".mops/[email protected]/src" "--package" "passport-client" ".mops/[email protected]/src" "--package" "sequence" ".mops/_github/sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "--package" "stable-hash-map" ".mops/_github/stable-hash-map#v0.2.1@eb7edf4127233f2b1f3732ede7e2c55827ac6886/src" "--package" "encoding" ".mops/_github/encoding#v0.3.2@8e0fe1d8f5c2d284e77d719703c42e0e271839b1/src" "--package" "candy" ".mops/_github/candy#v0.3.0@907a4e7363aac6c6a4e114ebc73e3d3f21e138af/src" "--package" "stablebuffer_1_3_0" ".mops/_github/stablebuffer_1_3_0#v1.3.0@acdde6bb5b939227997cebdbb8919d2e6da8691c/src" "--package" "map9" ".mops/_github/map9#v9.0.1@10b68f6ea8df5e72dfa4c07a50c8bb60a916c233/src" "--package" "candb-multi" ".mops/[email protected]/src" "--package" "test" ".mops/[email protected]/src" "--package" "date.mo" ".mops/_github/date.mo#master@cfe51ad7d8a576d53e701136d4234233b94970eb/src" "--package" "testing" ".mops/_github/testing#main/src" "--package" "fmt" ".mops/_github/fmt#v0.1.0/src" "--package" "motoko-sequence" ".mops/_github/motoko-sequence#366c419@366c4191d856ed4842267f5ab89d7222ed2d71d0/src" "-fshared-code" "--actor-idl" ".dfx/local/lsp"' failed with exit status 'exit status: 1'. Stdout:

Stderr: src/backend/personhood.mo:5.1-5.40: import error [M0009], file ".dfx/local/lsp/bkyz2-fmaaa-aaaaa-qaaaq-cai.did" does not exist

src/backend/personhood.mo:5.1-5.40 is about CanDBIndex (bkyz2-fmaaa-aaaaa-qaaaq-cai).