BitBucket implementation + webhook events / integration with Studio

[duijf]

Hey CML team! As you've probably noticed, I'm working on the integration between Studio + CML PR comments on BitBucket.

I'm running into a limitation with BitBucket webhooks and wanted to check if you are open to changing the CML BitBucket implementation to make the integration work more reliably.

Problem: pullrequest:comment_updated is very unreliable

BitBucket does not send webhook events for pullrequest:comment_updated in a lot of scenario's.

From the BB docs:

If a user updates the same comment with not much time in between, Bitbucket only sends the event request the first time the comment is updated. However, if some time passes and that user updates the comment again, Bitbucket sends the event request a second time.

From my experimentation, "not much time in between" can mean "not even after 20 minutes" (!). I have as-of-yet not been able to make BitBucket send me a single comment updated event[^wrong]. (Not when editing a PR comment with CML, but also not when editing manually).

Update: I've edited a comment I've posted after 3 days, and BitBucket still did not send any update payload. Seems like this is just plain broken.

[^wrong]: I triple checked that I'm not doing anything weird. The webhook I have is configured to send updates when PR comments are edited. The created+deleted events work fine.

How CML performs edits

When cml send-comment is invoked with the --pr and --update flags, CML checks if a PR comment exists and updates it if this is the case. The update contains the new contents of the report.

This update happens by sending a PUT request to BitBucket:

https://github.com/iterative/cml/blob/0a674fef92b0ff210c95fb83bd710e972199d630/src/drivers/bitbucket_cloud.js#L154-L166

Why Studio needs to know about updates

When CML updates a PR comment, the new comment may contain a new SHA. In these cases, Studio needs to associate the CML report with a different commit and display it accordingly. We are therefore interested in knowing when things change.

Options to resolve this

1. In theory, Studio could periodically poll BitBucket to see if any CML reports have been updated. I'd really like to avoid this though, as providing these updates to users in a timely manner would take up quite a sizeable portion of the API limits.
2. Make CML delete old BitBucket reports when invoked with --pr and --update.
3. Provide a different flag (e.g. --delete / --delete-old) and tell users to use this if they want smooth integration with Studio.
4. Don't do anything special / tell users not to use --update if they want to use CML with Studio

Out of all these options, I'm inclined to lean towards --delete / --delete-old (name up for discussion ofc.):

• The flag could be useful to other users as well, if they want notifications every time a new CML report is available, but don't care about the old ones any longer.
• Some CML users may use BitBucket and not use Studio and appreciate the current behavior of --edit.

What do you think about the options I posted? Anything I missed?