Implement "AntaTest" and "AntaAsyncTest" as protocols

[gusmb]

Sharing some ideas that could improve the framework... Need a way to enforce the format of the test functions to ensure all functions are written correctly (return boolean, etc) and the user can introduce more functions on runtime. Both normal/sequential and async modes could be supported, running the async tests in parallel on a "AntaAsyncTestRunner" and the other functions in a "AntaTestRunner". A test manager could handle the runners and dispatch the tests to each of them delegating the execution. So having the tests modelled in a flexible way could be the first step towards a more generic execution framework that can be customized/extended by the user on runtime.

[mtache]

Thanks for raising this :) Implementing tests as coroutines is definitely needed for this framework to scale. Do you have any example of a test that would require to be run sequentially ?

[gusmb]

Thanks for raising this :) Implementing tests as coroutines is definitely needed for this framework to scale. Do you have any example of a test that would require to be run sequentially ?

I think it could be useful for large deployments, or where there are a vast number of tests with high correlation. For example, some BGP overlay session tests wouldn't make sense if the loopback is not defined, therefore you could cluster tests together, running the tests inside the cluster as co-routines and separate cluster tests in sequence. For instance:

• Test group 1:
  • test_function_1
  • test_function_2
  • test_function_3
• Test group 2:
  • test_function_4
  • test_function_5
  • test_function_6

Where usually the functions in test_group1 need to pass in order to make sense out of the tests in group 2, etc.

async def run_sequence(*functions: Awaitable[Any]) -> None:
  for function in functions:
    await function

async def run_parallel(*functions: Awaitable[Any]) -> None:
  await asyncio.gather(*functions)

await run_sequence(
  run_parallel(
    test_function_1(),
    test_function_2(),
    test_function_3()
  ),
  run_parallel(
    test_function_4(),
    test_function_5(),
    test_function_6()
  )
)

Other use cases could be managing how tests run across the inventory of devices (e.g., first spines, then the leafs, etc). This kind of flexibility may allow to easily optimize performance based on the inventory and test catalog...

[mtache]

PR #122 implements asyncio in ANTA. However, we still cannot define a test hierarchy/dependency if some of them needs to be run sequentially: everything runs in parallel today. However, we have written the anta_test decorator to factorise code that is identical across all tests.