Make clarinet release process more reliable

[hugoclrd]

Context

The release process currently 1) is very manual, 2) is lacking in automated testing, and 3) can be broken by stacks-core dependencies.

Clarinet needs to be released more frequently (once every 1 or 2 weeks) in order to stay up to date with the latest changes happening in stacks-core and clarity-wasm.

Clarinet is a monorepo that includes multiple components that need to be released on multiple platforms:

• clarinet (the cli - built and published with the CI)
• stacks-devnet-js (published on NPM via CI)
• clarity-sdk (manually published on NPM)
• vscode extension (manually published on open vsx and vscode marketplace)
• a few packages manually published on crates.io (clarinet-files, clarinet-deployments, stacks-rpc-client)

Tasks

The release process needs to be easier and more reliable. This in an epic issues listing the different actions that can be taken to make it happen

• [x] Discuss the release workflow
  • [x] Do we need the develop and the main branches or could we only keep main?
  • [x] Simplify the version update process (which is error prone today)
• [ ] CI rework
  • [x] Run stacks-devnet-js build on every target for every PR https://github.com/hirosystems/clarinet/pull/1340
  • [ ] Isolate CI for each components for better maintanibility
    • [x] clarinet-cli (disabled for now but avaialble) https://github.com/hirosystems/clarinet/pull/1104
    • [x] stacks-devnet-js
    • [x] clarity-sdk
      • [x] We should at least check that it does build on every PR
      • [x] And run unit tests
      • [x] Automated release
    • [ ] vscode extension
• [x] Fix vscode extension tests
• [ ] improve external dependencies management
  • [ ] on the stacks-core CI, add a check for the clarity build targeting wasm
  • [x] better dependencies management between stacks-core, clarity-wasm, chainhook, clarinet
• [ ] Code quality
  • [ ] Enforce clippy check on CI
  • [x] Fix recent clippy warning (,o warning were introduced with rust 1.75.0 and above)
• [ ] More tests
  • [ ] unit tests
  • [ ] e2e tests, some part of the release playbook can be automated

[aaronb-stacks]

Just surfacing a slight pain-point about the stacks-core:develop dependencies: those definitely change enough that they seem to break clarinet CLI's build pretty often.

It would be easy to change the dependencies to use specific git tags, but I guess that would end up requiring more active monitoring of changes in stacks-core, so if you feel like you prefer the current setup, feel free to ignore my suggestion.

[hugo-stacks]

@aaronb-stacks Historically, we used to develop so that we could easily satisfy the dependencies of both the chainhook repo and the clarinet repo, without needing to make synced PRs on both to upgrade clarity. That was a mess, but since we recently removed the chainhook dependency, we could switch back to a specific git rev indeed, that'd be more explicit.

Also, the exact commit is saved in the cargo.lock, I know I never run into this issue unless I intentionally run cargo upgrade clarity, so I wonder how you ran into this issue?

Ideally, we'd like clarity to be published and versioned on crates.io, but that's a bigger topic.