philterd
Have performance tests run during the build
Have performance tests run during the build and fail the build if a decrease in critical metrics.
@jzonthemtn wonder if adding phileas-benchmark as a Phileas submodule would make sense?
In the last week or so I've cleaned up phileas-benchmark -- before it had some test cases for issues that we were trying to fix, and some domain-specific filtering logic that I was trying out. (phileas-benchmark is now pretty generic and not as specific to the "resurfaceio" usage of Phileas as it was early on)
Anyway, I was gonna ask you about perhaps moving phileas-benchmark over to the philterd org in GitHub -- but if you're looking to make benchmarking part of the regular build, might be better as a submodule?
ps. Because we have a few other integrations working now, we'll likely be focusing more on end-to-end benchmarking -- so I won't be bothered if phileas-benchmark morphs into something completely different that produces different results. I don't need backwards compatibility here so wouldn't want that to hold us back.
I think a submodule would be good. If it was Philter with an API I might say a separate repository to build as a docker image and run against the API but I think that's overly complicating it for what there is now. I'd love to include it and set up gates around it!
The mentioned repository is at https://github.com/philterd/phileas-benchmark.