deep-dive: IPFS Integration testing and continuous delivery

[olizilla]

Problems with IPFS test & release process today

• Slow tests
• Duplicated test cases
• Time consuming manual releases
• Test silos per implementation language.
• Testing source code rather than deployment artefact

Proposal

Continuous Deployment! - Release often and automatically, to massively reduce the burden on maintainers

What needs to happen to make us comfortable with that?

• Run tests on packaged / bundled artefacts (ensure no packaging errors)
• Create system tests: mininet tests for virtual, multi-node tests in varied network topologies
• Reduce test-case duplication (interop, interface, api, tests)
• Speed up tests: run different test classes in parallel
• Run shaness tests against js-ipfs cli
• Define shared http api tests / spec
• Automate "Canary in the coal mine tests" where we check new release works with selected existing apps
• Define automatic release criteria? Merge to master? Nightly?

[achingbrain]

[olizilla]

This deep-dive was inspired by https://github.com/ipfs/notes/issues/294