Proposal: End to End Tests for Deal Flow in Lotus

[alanshaw]

Rendered

[alanshaw]

1. Long term ownership - who is responsible for the infrastructure to run these tests, keeping them updated, and fixing them if they get flaky (not due to Lotus itself being flaky).

Good question. I'd like to know the story for current testground infra to really be able to comment on that. I assume this is more infra for the infra team but it would be nice if there were an alternative to that.

IMHO if you change the CLI commands for the deal flow then you are responsible for updating the tests. Like with any tests.

Changes to the code that cause a test to periodically fail need to be investigated. That's a PITA because it's not always possible to determine which developer is responsible. It could be the dev who wrote the test, could be the dev who made the change, or could be a combination of the two (the change has caused an assumption in the test to not hold anymore).

I would suggest that it's the responsibility of the sub-team involved in markets/deals to own any disputes but any developer what causes the tests to fail where the reason is obviously because of a change they made should be responsible for fixing them.

This is a good point because it needs buy in from the teams involved. If the tests are not seen as valuable then I'd rather not spend the time. However it is the responsibility of the implementer to make them as reliable and useful as possible.

2. What network(s) will these run against? It would be ideal to have these also run against the production network so we detect network issues before clients report them but understood that is likely out of scope right now and is debatable even if it's worth doing.

I'd anticipated for a network to be spun up just for running the tests, but a testnet that is already running might speed up the round trip time.

I assume we'd create actors as needed for running the tests, so I'd expect to receive similar results running the same tests on mainnet...and running on mainnet would be way slower as we'd probably speed up the epoch period for testing. I also don't think we should pollute mainnet with test actors and deals.

[BigLep]

Currently waiting on resourcing and after @raulk's Lotus testing sprint.

[willscott]

noting that some of this may overlap with the dealbot now doing regular testing of the deal flow