Question(dev): Benchmarking script

[Uzaaft]

Hi,

Is it possible to add some kind of benchmarking tests in the test suit, so that one can test for performance regression while implementing changes to the code? I'm not a biologist/gene guy(Just a random HPC enthusiast), so I don't have the brain power to do so myself.

[rob-p]

Hi @Uzaaft,

That's a nice idea! While we have some experience with integration and performance regression testing, we've always done that outside of the context of the code tests themselves (mostly because we were working in C++ and there wasn't an obvious testing framework to use for this purpose). I imagine that some performance tests could be put together using criterion.rs — though we'd have to look into how large test data has to be before meaningful performance numbers can be inferred. We'll certainly look into this.

Best, Rob

[Uzaaft]

I think some "real world" workload, from your guys' own work might perhaps be most applicable. To see what is the slowest parts of the code, etc etc. Again, I'm not a biologist, so I have absolutely no insight whatsoever regarding how you guys work.

[rob-p]

Got it — so I think this would be possible, but would probably require the download of moderately sized external files (stuff that can't easily be put into VC). Presumably there is some standard way to only invoke the tests if the data is present, or to ask to grab the data automatically if the user wants to run these tests. I'm not familiar if there is an idiomatic way to do this in rust or not.

[Uzaaft]

I think a way could be to cache the data in the /tmp dir, and then make the script look after it there, before it downloads it and caches it again. This way, the data is not saved across reboots(which avoid eating up space), but is there across invocations during a "session".

[Uzaaft]

For example, the script could be within the \tests folder, e.g called: \tests\performance_regression.rs