capa-rules release process and versioning guarantees

[williballenthin]

see issues #551 #552 #555

Here's a proposal for how we should handle breaking changes to capa and its rule syntax. In essence, I think we should maintain major version branches for capa-rules and point users to fetch the rules that match their major version of capa.

---

We're in a weird spot right now: we can't introduce breaking changes to the master branch until v4 because our documentation points v3 users to checkout rules from master.

Until v4 is released, we should make breaking changes to the v4-dev branch. This is a temporary branch for pending v4 changes. We'll delete it once we release v4. We can make non-breaking changes to master.

When we release v4, we'll create a branch v4 that contains all the changes for that line of releases. We'll sync it from master upon each minor or point release (or more often, doesn't hurt). Once v5 is released, master can no longer be merged into v4; however, we can backport non-breaking changes into the v4 branch after v5 is released.

Over time, we will accumulate branches corresponding to major releases.

release plans

In order to release v4:

• [x] create branch v3 from master. This is where any backported changes will go.
  • we'll point users to pull from this branch when using capa v3.
• [ ] merge branch v4-dev into master. Master now contains pending breaking changes.
• [ ] delete branch v4-dev to avoid any confusion. All breaking changes are now in master.
• [ ] create branch v4 from master. This will contain all changes up until v5 (and backported changes beyond then).
  • at each minor or patch release of v4, we will merge master into v4
  • once we hit v5, we will no longer merge master into v4
• [ ] cut release in github.
• [ ] create branch v5-dev from master. This will contain any pending breaking changes prepared for v5.

During the v4 line of development, non-breaking changes are made to master while breaking changes (for v5) are made to v5-dev.

At a minor or patch release:

• [ ] merge master into v4
• [ ] cut release in github

Upon reaching v5:

• [ ] merge master into v4. Now all pending non-breaking changes are in v4.
• [ ] merge v5-dev into master. Master can no longer be merged into v4.
• [ ] create branch v5 from master. This will contain all changes up until v6 (and backported changes beyond then).
• [ ] cut release in github.
• [ ] create branch v6-dev from master. This will contain any pending breaking changes prepared for v6.

considerations

1. when accepting PRs, double check the target branch:
   • non-breaking changes go into master. this is the common case for the community submitting rules.
   • breaking changes go into v4-dev. when capa devs make changes to capa, such as adding new features/logic/blocks or removing old ones, and then update the ruleset, these changes need to go into v4-dev.
   • its ok to sync master into v4-dev.
   • capa devs probably want to track v4-dev.
2. point users to check out rules from the branch that matches their major version, e.g. v4.
3. do we need to check the capabot logic and how it syncs the submodule in the capa repo?
4. i think we may want to update the capa-rules submodule away from master when making a new major release? or at least if we ever backport changes into a capa-rules branch.

[williballenthin]

@mr-tz @mike-hunhoff @Ana06 discussion and thoughts, please!

[Ana06]

I may be missing something, but I think this is not needed. We tag capa-rules when releasing capa: https://github.com/mandiant/capa-rules/tags

If we are doing the releases correctly any minor release should be compatible. To be sure, we could also tell users to use the exact same version as that is the one we have tested.

Shouldn't be enough to specify in the documentation to download the correct capa-rules version?

[williballenthin]

Using tags is probably fine and is less complex for us, so is probably the way to go. Thanks for reminding me to pick the simpler route :-)

So in summary:

• continue to develop on master
• upon releases, tag as we've been doing, like v3.1.0
• update documentation to tell users to pick the latest compatible release (major version)
  • this puts a bit more burden on users to pick the right version, but our users aren't dumb, and we should trust them
  • still, it might be nice to have in the documentation a clear table: "if you have capa v3, use tag 3.0.3"
  • git clone .... --branch <tag> will also work fine

... much simpler.

As @mr-tz mentioned, I think there's still an issue of breaking changes on master. We may still want to consider doing breaking changes on a feature branch like v4-dev. This way we can tag from master. Otherwise, we can't intersperse v3 rule updates with v4 rule development.

[mr-tz]

That sounds good to me!

[mike-hunhoff]

Please correct me if I'm wrong but based on the most recent post from @williballenthin our decision is to continue development, including breaking changes, on master. This doesn't play well with our current CI setup for capa-rules as we test using the latest release which won't support any breaking changes that we've added to master. Consider #601 containing rules that are not supported by our most recent release, v4.0.1.

I propose we use a feature branch as discussed above to develop breaking changes. This feature branch should use the latest master branch for testing, not the latest release as the breaking changes won't be supported. Non-breaking changes can be developed in master and master should use the latest release for testing.

This workflow may looking something like:

• got breaking changes? branch from v5-dev for development
  • v5-dev's CI uses capa master branch for testing
• got non-breaking changes? branch from master for development
  • master's CI uses latest capa release for testing, e.g. v4.0.1

In most cases, new rules or rule updates won't use breaking changes that haven't been released yet so users shouldn't see much of a change in their rule dev workflow.

Thoughts?

[Ana06]

I think we should keep the master branches of both projects in sync. We always release capa rules together with a capa version. So if there are breaking changes in master in capa, new rules should be adapted to the breaking changes. This way we can release capa knowing everything in capa-rules is alright.

The CI at the moment is using capa master for the linter and last release to run the rules. 😕 I think we should update the test to run the rules with master.

[mr-tz]

See my proposal in https://github.com/mandiant/capa-rules/pull/621 to change the CI check:

• check latest released version is compatible with rules master
• for branches into a dev-X branch, only run the linter (based on code master)

This requires Willi's and Mike's proposal to merge into a dev-X branch for rules that are in master but not in the latest release yet.

[mike-hunhoff]

@Ana06 totally agree it is preferable to keep both master branches in sync. I also think it's worth enabling users the ability to use point releases of capa rules between major releases. For example, if a user merges 10 new rules that work with the latest release but capa rules master contains breaking changes then these 10 rules can't be used until the next major release. This could be a long time depending on the release cycle. Is there a way around this limitation with our current setup?

An extension to the proposals above could be to create a feature branch of capa master containing all breaking changes contained in the next major release. This requires an additional branch but we could keep capa rules v*-dev in sync with capa v*-dev and capa rules master in sync with capa master.

[Ana06]

IMO users should use released rules, because they may not know if breaking changes have been introduced and the new rules are not compatible with old releases.

We could release a new version of capa (and capa-rules as they are released together) before merging breaking changes into master. The dev branch allows to merge the breaking changes later, but may generate merge conflicts that give us more work.

[mr-tz]

After a lot of back and forth, I think we just need to update the CI check at this point. I propose to change the workflow to:

• ensure code and rules master are compatible
• (optional: ensure the latest release is compatible with the latest respective branch)

Our current workflow:

• all the latest code and rules are in master (including unreleased and backwards incompatible features, one could also refer to it as development)

Our users:

• use a PyInstaller compiled binary with embedded rules (no issue)
• use code master and rules master (no issue)
• use pip install flare-capa/PyInstaller binary and rules master (potential issue)

If the new rules are not compatible (breaking change has been introduced) we already point users to the respective branch via the documentation and the command line (per the discussion above).

[mike-hunhoff]

After a lot of back and forth, I think we just need to update the CI check at this point. I propose to change the workflow to:

* ensure code and rules master are compatible

* (optional: ensure the latest release is compatible with the latest respective branch)

Our current workflow:

* all the latest code and rules are in `master` (including unreleased and backwards incompatible features, one could also refer to it as `development`)

Our users:

* use a PyInstaller compiled binary with embedded rules (no issue)

* use code `master` and rules `master` (no issue)

* use `pip install flare-capa`/PyInstaller binary and rules `master` (potential issue)

If the new rules are not compatible (breaking change has been introduced) we already point users to the respective branch via the documentation and the command line (per the discussion above).

works for me - let's update #621 to reflect this workflow which as I understand means changing CI to run master rules against master code?

[williballenthin]

i also agree with this proposal On Sep 16, 2022, at 4:15 PM, Mike Hunhoff @.***> wrote:

After a lot of back and forth, I think we just need to update the CI check at this point. I propose to change the workflow to:

• ensure code and rules master are compatible

• (optional: ensure the latest release is compatible with the latest respective branch)

Our current workflow:

• all the latest code and rules are in master (including unreleased and backwards incompatible features, one could also refer to it as development)

Our users:

• use a PyInstaller compiled binary with embedded rules (no issue)

• use code master and rules master (no issue)

• use pip install flare-capa/PyInstaller binary and rules master (potential issue)

If the new rules are not compatible (breaking change has been introduced) we already point users to the respective branch via the documentation and the command line (per the discussion above).

works for me - let's update #621 to reflect this workflow which as I understand means changing CI to run master rules against master code?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[mr-tz]

I've updated https://github.com/mandiant/capa-rules/pull/621 accordingly.

[mr-tz]

We hope this is addressed via rule releases.