starknet-devnet-rs icon indicating copy to clipboard operation
starknet-devnet-rs copied to clipboard

CI workflow sometimes overuses memory

Open FabijanC opened this issue 1 year ago • 2 comments

Sometimes Devnet's CircleCI workflow fails because of memory overuse. Maybe every month or so I see a failure with a message like this:

  = note: collect2: fatal error: ld terminated with signal 9 [Killed]
          compilation terminated.

I'm pretty sure it's due to parallelization (related Stack Overflow post). Currently we use 7 parallel cargo jobs to test the code: https://github.com/0xSpaceShard/starknet-devnet-rs/blob/37dc6e698297cd5173234397e77a72a756bfa270/.circleci/config.yml#L70

So far, it has always been sufficient to simply restart the failed workflow (example of a successful restart)

This is the memory usage of the passing restarted workflow (peaking at 98% memory used): image

Perhaps we could set the number of jobs/cores to 6 instead of 7, but that would probably slow down the testing. Anyway, consider this a heads-up that you might need to restart until something else is implemented.

FabijanC avatar Mar 12 '24 09:03 FabijanC

Caching target/debug/.fingerprint, target/debug/deps, target/debug/incremental didn't help when splitting building and testing into 2 different jobs.

marioiordanov avatar Mar 21 '24 16:03 marioiordanov

Changing the number of jobs for running integration tests produced the following results: cargo test --test '*' --jobs 4 -> https://app.circleci.com/pipelines/github/0xSpaceShard/starknet-devnet-rs/1532/workflows/12a6ae28-e5d4-4170-abb0-d186991b1604 cargo test --test '*' --jobs 3 -> https://app.circleci.com/pipelines/github/0xSpaceShard/starknet-devnet-rs/1530/workflows/da5376e5-a9b1-44e6-beee-4037876a116b

marioiordanov avatar Mar 21 '24 16:03 marioiordanov

Closed via https://github.com/0xSpaceShard/starknet-devnet-rs/pull/554

FabijanC avatar Sep 23 '24 09:09 FabijanC