KomodoPlatform
split docker tests by protocol
Currently, there're some annoying issues with how our docker tests is being ran, both in local and CI environments. Issues like long execution times, not able to run a specific tests etc.
My suggested fix is refactoring unified Docker test entrypoint into protocol-specific test targets to improves test isolation, and allow targeted test to run both locally and in CI which also improves docker tests running time
- run-docker-tests-utxo
- run-docker-tests-eth
- run-docker-tests-ibc
- run-docker-tests-sia
- run-docker-tests-qtum
- run-docker-tests-zcoin
Additionally, we can have a run-docker-tests feature to be used for running all docker tests locally
cc. @shamardy
Thank you for bringing this to attention. I guess the problem, which this issue is to solve, is that when you are developing a particular docker test you need to start only needed containers for it, to speed up the environment. I see couple problems with having multiple features like run-docker-tests-utxo, run-docker-tests-eth etc for that:
- Swap tests need two containers, so not clear how to run them with features for a specific coin.
- There are many pieces of test code (in swap code) enabled with the existing "run-docker-tests" feature, so you would need to replace it with a more complex cfg like
#[cfg(any(test, feature = "run-docker-tests-utxo", feature = "run-docker-tests-eth",...))]
I think, when we are developing a test, to run only needed containers, another solution could be having a few env variables which would disable to create not needed containers.
Thank you for bringing this to attention. I guess the problem, which this issue is to solve, is that when you are developing a particular docker test you need to start only needed containers for it, to speed up the environment. I see couple problems with having multiple features like run-docker-tests-utxo, run-docker-tests-eth etc for that:
- Swap tests need two containers, so not clear how to run them with features for a specific coin.
- There are many pieces of test code (in swap code) enabled with the existing "run-docker-tests" feature, so you would need to replace it with a more complex cfg like
#[cfg(any(test, feature = "run-docker-tests-utxo", feature = "run-docker-tests-eth",...))]I think, when we are developing a test, to run only needed containers, another solution could be having a few env variables which would disable to create not needed containers.
aha this is def a big edge I overlooked. I will need some time to reason about your approach and others.
update: I agree with your approach of introducing few env variables. Thanks!
Swap tests need two containers, so not clear how to run them with features for a specific coin.
Well, we can use #[cfg(all(feature = "run-docker-tests-utxo", feature = "run-docker-tests-eth"))], maybe combining features under one flag aka utxo-eth-docker-tests.
The problem will be in CI resources as we will run more docker nodes than we currently do in exchange for speed of the jobs as they will run in parallel. The benefit this brings is better organization or separation of what kind of test coverage we have, for instance, do we cover only utxo-eth cross chain tests or all protocols?
I see this refactor as one of the prerequisites to refactoring each protocol to a different crate and then probably repo, a protocol (ETH for example) should have docker tests for wallet operations among other same chain operations but also cross chain swaps / tests with each of the other protocols, maybe coverage of cross chain swaps with utxo is enough if we want to lower the resources.
For CI resources we should consider service containers https://docs.github.com/en/actions/use-cases-and-examples/using-containerized-services/about-service-containers to share the same container across multiple jobs while keeping them isolated where needed. I am not sure if we can make this work with rust testrunners, so we need to take a look at that.
Swap tests need two containers, so not clear how to run them with features for a specific coin.
Well, we can use
#[cfg(all(feature = "run-docker-tests-utxo", feature = "run-docker-tests-eth"))], maybe combining features under one flag akautxo-eth-docker-tests.The problem will be in CI resources as we will run more docker nodes than we currently do in exchange for speed of the jobs as they will run in parallel. The benefit this brings is better organization or separation of what kind of test coverage we have, for instance, do we cover only utxo-eth cross chain tests or all protocols?
I see this refactor as one of the prerequisites to refactoring each protocol to a different crate and then probably repo, a protocol (ETH for example) should have docker tests for wallet operations among other same chain operations but also cross chain swaps / tests with each of the other protocols, maybe coverage of cross chain swaps with utxo is enough if we want to lower the resources.
For CI resources we should consider service containers https://docs.github.com/en/actions/use-cases-and-examples/using-containerized-services/about-service-containers to share the same container across multiple jobs while keeping them isolated where needed. I am not sure if we can make this work with rust testrunners, so we need to take a look at that.
looks like service containers could work after reading the docs(without testrunners) but I couldn't find anything relating to hooking up volume like we currently do here. However, I assume there's going to be some sort of label like volume to use e.g
services:
# Label used to access the service container
redis:
# Docker Hub image
image: redis
#
ports:
# Opens tcp port 6379 on the host and service container
- 6379:6379
volume: "/root/.zcash-params"
This could work for z_coin docker test since
With service containers, to run docker tests locally w/o CI, users would have to run needed containers manually (or with some script), right? (Wondering though would it be possible to keep containers up and re-run tests w/o restarting containers)
With service containers, to run docker tests locally w/o CI, users would have to run needed containers manually (or with some script), right? (Wondering though would it be possible to keep containers up and re-run tests w/o restarting containers)
You just need to define the service script(the script above) in the same CI script..
Wondering though would it be possible to keep containers up and re-run tests w/o restarting containers
I doubt, from what Ive seen, every job will have it's own service instance definition