convoy icon indicating copy to clipboard operation
convoy copied to clipboard

add Mayhem for API testing as a github workflow

Open dlowe opened this issue 2 years ago • 1 comments

Add a github action to run Mayhem for API (which is a free automated testing tool for http APIs) against the api: given the swagger spec and api server, it generates and runs random test payloads, looking for exceptional behavior.

Disclosure: working on this tool is my day job!

As with any github workflow, test results can be consumed through github. Alternatively, Mayhem for API also has a dashboard, over here: https://mayhem4api.forallsecure.com/mayhemheroes/convoy.

Currently this finds a bunch of Internal Server Errors and Server Crashes that can be pretty easily triggered, as well as lots of mismatches between the openapi spec and the empirical responses (these warnings can be disabled if they're not interesting, but might be useful to keep your spec and code aligned, depending on how you maintain the specs.)

To make it work in the upstream repo, there's a bit of out-of-PR effort to integrate:

  • sign up for a free mapi account, creating a frain-dev organization
  • create a mapi service account within the frain-dev organization
  • modify the mapi.yaml action in this PR to point at your mapi organization
  • add the service account's API token to github as a repository secret named MAPI_TOKEN

dlowe avatar Jun 09 '22 23:06 dlowe

Deploy Preview for getconvoy canceled.

Name Link
Latest commit 3b46491405e29324b77640753e9f4e9370cdac1d
Latest deploy log https://app.netlify.com/sites/getconvoy/deploys/62a27cea9c344500072dd847

netlify[bot] avatar Jun 09 '22 23:06 netlify[bot]

Hi @dlowe,

Thanks for raising this PR. This looks interesting. But I don't think we want to merge and maintain this test suite right now. Hopefully, you can submit this PR again sometime soon, early next year, and we can re-assess!

Thanks!

subomi avatar Sep 01 '22 18:09 subomi