Any release plans? 5.12.1 or 5.13.0 or 6.0.0 ?

[hnyman]

It is now over 3 years since the last release, 5.12.0.

Looking at the current branches, I am totally confused on how the development is going on and if there are any release plans?

From what I have followed collectd from "use for OpenWrt perspective" for over a decade, the development is currently somewhat stuck, and a "big change"(?) for 6.0 will likely take quite long ?

There are currently four branches that have seen commits this year:

• main branch. Last commit 6 December 2023.
• 6.0 branch: 341 commits ahead and 242 commits behind main. Last commit 6 December 2023.
• 5.13 branch: 2 commits ahead and 240 commits behind main. Last commit 24 March 2023.
• 5.12 branch: 6 commits ahead and 258 commits behind main. Last commit on 23 March 2023.

Four differing branches are a lot. Especially when only 5.12 has been released and both 5.13 and 6.0 are unreleased release branches.

Differences between 5.12 and 5.13 are minimal, so the role of 5.13 branch is a puzzle.

Also, main and 6.0 have differed quite a lot. Why? Is main going to be 5.14?

Could there be a 5.12.1 maintenance release or a 5.13 release while waiting for 6.0? Are both 5.12 and 5.13 really needed, if 5.13 is never going to be released?

[octo]

Hi @hnyman,

that's an excellent question and I think we should have a community discussion about next steps and where we focus our energy.

• collectd-5.12 branch

  Historically, we have committed bug fixes to "version branches" and merged them back into main to avoid cherry picking and duplicate commits. My suspicion is that recently bug fixes have simply been committed to main instead. My suggestion would be to package up a version 5.12.1 with the few fixes we have in there, since it's not much work. I wouldn't expect much more development to happen here.

• collectd-5.13 branch

  Sounds like a release branch that got abandoned. I think we should rebase the few commits on there onto main (unless they are there already) and delete the collectd-5.13 branch.

• main branch

  This is the current development branch for the collectd-5.* line. I think it contains numerous bug fixes and new features that we haven't released yet, and we should publish that as 5.13.0 at some point. I don't have a time line for that in mind at the moment. What do others think?

• collectd-6.0 branch

  This is the development branch for the collectd-6.* line. I think it is a bit of a never ending story at this point, which I would love to change. We should discuss what has to happen before we can publish a "release candidate" and later a proper release. For example, I think that we should have the data structures locked in (e.g. #4187) and vet the metric names of "converted" plugins (e.g. #4181). It would be fine for me to publish a version 6 MVP that does not ship all of version 5's plugins, e.g. the Python plugin is probably very hard to migrate.

[octo]

@mrunge What do you think?

[eero-t]

IMHO:

• collectd-5.12: at last CI fixes for latest distros should be included before .1 release, otherwise it's fairly useless
• collectd-5.13: agree
• main =>collectd-5.13: Earlier there were about 2 releases per year, but it's been >2 years since last release, so having new one rather sooner than later would be good to show that project is alive again... Here are few things that may be good to handle before that though:
  • https://github.com/collectd/collectd/issues/4050
  • https://github.com/collectd/collectd/issues/4061
  • https://github.com/collectd/collectd/issues/4127
  • https://github.com/collectd/collectd/issues/4119
  • https://github.com/collectd/collectd/pull/3973
  • https://github.com/collectd/collectd/pull/3952
  • https://github.com/collectd/collectd/pull/3936
  • https://github.com/collectd/collectd/pull/3781
  • https://github.com/collectd/collectd/pull/3777
• collectd-6.0: main thing before release candidate, after structure stuff is locked down, would be syncing all 5.x changes to plugins to v6, as it's a long time since v6 was forked

[octo]

Syncing all deviating changes is quite a tall order, given that collectd-6.0 is ~250 commits behind main. I could try to just cherry pick all non-merge commits and see how many of them apply cleanly. If that leaves <10 commits to manually merge, that should be feasible. But if it is more than that, I doubt we have the capacity to do this anytime soon and would prefer to just get out something, even if sub-optimal.

[mrunge]

I would suggest to merge the main branch into the 5.13 branch. It was created earlier and then abandoned. I also agree with @eero-t that we should sync the ci fixes into the 5.12 branch to allow distributions to pick up the changes.

Tbh. I have no good overview at this time on the 6.0 status.

[octo]

If collectd-5.13 has just two commits, I say we cherry-pick those into main and delete collectd-5.13. These merges over 200+ commits make the commit history incomprehensible.

What are your feelings about reinstating the collectd community call? It would give us a place where we can discuss these things and give status updates.

[eero-t]

Syncing all deviating changes is quite a tall order, given that collectd-6.0 is ~250 commits behind main. I could try to just cherry pick all non-merge commits and see how many of them apply cleanly. If that leaves <10 commits to manually merge, that should be feasible.

Looking at individual commits is relevant only for plugins that have already been ported to 6.0 API. I think majority still has not been?

For ones that still have not been migrated to 6.0 API, their latest code in main can be just copied again to 6.0 branch. Preferably before applying any (several years old) "migrate to 6.0" PRs to them... :-)

(I think hnez did that for the plugins that he ported to 6.0 API.)

[octo]

I looked into merging plugin changes from main to collectd-6.0. I looked at src/*.c as a proxy for plugins, though this list also contains unit tests, e.g. This is the summary of my results:

Status	Count
No op; unchanged in main and collectd-6.0	105
No op; changes only in collectd-6.0	53
Diverged; changes in both main and collectd-6.0	30
Easily updatable; changes only in main	14

The plugins that can easily be updated are:

• bind

• collectd-nagios

• cpufreq

• hugepages

• intel_pmu

  This plugin has seen significant work; it might be better to import commits rather than copying the file. // CC: @kwiatrox

• mysql

• nginx

• ovs_stats

• processes

• smart_test

• swap (may cause merge conflict with #4190)

• ubi

• virt

• vmem

I have created a PR for these: #4198

The diverged plugins are:

Plugin	Changes in main	Changes in collectd-6.0	Status
amqp1	1	2	diverged
amqp	3	2	diverged
apple_sensors	1	1	diverged
battery	2	2	diverged
capabilities	1	2	diverged
csv	1	2	diverged
disk	2	6	diverged
gmond	2	3	diverged
intel_rdt	8	2	diverged
memory	3	7	diverged
mmc	3	4	diverged
modbus	1	1	diverged
mqtt	1	3	diverged
netlink_test	1	1	diverged
network	3	1	diverged
nut	2	1	diverged
perl	1	4	diverged
procevent	6	5	diverged
python	2	1	diverged
redfish	1	1	diverged
smart	2	1	diverged
snmp	4	2	diverged
statsd	1	2	diverged
write_graphite	1	3	diverged
write_http	2	8	diverged
write_influxdb_udp	5	6	diverged
write_log	1	6	diverged
write_mongodb	1	2	diverged
write_prometheus	2	5	diverged
write_stackdriver	3	4	diverged

I haven't looked into this yet, but I suspect that these will run into a fair amount of merge conflicts.

[eero-t]

Ouch, I hadn't realized v6 was branched from main so long time ago, it has v6 API changes that are already 3.5 years old!

For the plugins with just 1 new commit in main, you could try scripting cherry-picking their HEAD commits to v6. If those apply, it would halve the list (I looked at couple of them, and they seemed ok).

write_prometheus is already migrated to v6 API, but the other commit is already in, and cherry-pick for HEAD one (https://github.com/collectd/collectd/commit/951fababf57087dbf77fa241e133f98ad73a7389) one should apply fine.

[eero-t]

Links to some of the patches distros are using on top of current 5.12 release:

• Debian (trixie/sid): https://sources.debian.org/patches/collectd/5.12.0-15/
• Fedora (rawhide): https://src.fedoraproject.org/rpms/collectd/tree/rawhide

@mrunge, maybe you can comment on whether any of the Fedora patches are still needed with CI fixes in main?

(I.e. will CI fixes be enough for 5.12.1 release, or is something else also missing.)

[eero-t]

On quick look at the Debian patches, they are fixed in main except for ones changes config & include file paths, and the CGI one: https://github.com/collectd/collectd/pull/3840

[octo]

I just merged the Perl CGI fix.

[eero-t]

@octo Early this year Leonard checked and updated several plugins in v6 branch to latest changes from main, before finishing migrating subset of them to v6 APIs, see https://github.com/collectd/collectd/pull/4026#issuecomment-1378869196.

[octo]

I created the 6 MVP Milestone to track the things I think we need to get done before packaging our first collectd 6 release. Please take a look.

[octo]

I started drafting a 6.0 release candidate at https://github.com/collectd/collectd/releases/edit/untagged-54315dd568937fe38d17

[eero-t]

I started drafting a 6.0 release candidate at https://github.com/collectd/collectd/releases/edit/untagged-54315dd568937fe38d17

@octo I assume this is list of changes compared to latest (v5) release before it.

Of the significant v5->v6 changes that above list is missing, I can right away name at least:

• Core: https://github.com/collectd/collectd/pull/4026
• New plugins: https://github.com/collectd/collectd/pull/3968
• v5 -> v6 plugin migration by @hnez:
  • https://github.com/collectd/collectd/pull/4077
  • https://github.com/collectd/collectd/pull/4075

(Last one is based on @manuelluis work, but @hnez did lot of checking before those old PRs were merged, so I think it's fair to attribute that to him: https://github.com/collectd/collectd/pull/4026#issuecomment-1378869196)

As to plugin changes, while there have been quite a few PRs to Sysman plugin after I added it, IMHO those latter ones do not need to be explicitly listed in the changes, as plugin was not in any release.

PS. Of the significant fixes in v5 that should be added to v6 before any -rc release, I would propose at least your fix to missing riemann_set_event() terminators in riemann plugin. :-)

[octo]

This is based on me grep'ing through the commit history to find PRs:

git log --pretty=oneline "main..collectd-6.0" | egrep 'Merge pull request #([1-9][0-9]*)'

The PRs you mentioned have been merged manually, probably because the automatic builders were broken.

Unfortunately I don't think we can get a list of PRs since the last release from the GitHub API. We can get a list of PRs using the collectd-6.0 base branch, so that should work for the initial collectd 6 release. Let me re-generate that.

[octo]

PS. Of the significant fixes in v5 that should be added to v6 before any -rc release, I would propose at least your fix to missing riemann_set_event() terminators in riemann plugin. :-)

Cherry-picked that into collectd-6.0 as f6036685694b23032f53be8a37d26c7b33e670b3

[octo]

Okay, I'm getting this list. Does that look better?

https://github.com/octo/collectd/releases/tag/untagged-e88ba30fd34c7c8e5dce

[eero-t]

Unfortunately I don't think we can get a list of PRs since the last release from the GitHub API.

Merged ones with "[collectd 6]" in title: https://github.com/collectd/collectd/pulls?q=is%3Apr+is%3Aclosed+is%3Amerged+%22%5Bcollectd+6%5D%22

https://github.com/octo/collectd/releases/tag/untagged-e88ba30fd34c7c8e5dce

Not found?

[octo]

Copied the changes to https://github.com/collectd/collectd/releases/edit/untagged-b0aa78c17d2b8ac9c9d5

[eero-t]

List content looks better to me now.

Please sort the list and split it at least to "Core" & "Plugin changes" sections, so that related changes are easier to find.

Note: if you add PR number after the component name (at the beginning) instead of at end, then the list will naturally sort both by the component, and the order in which changes to it were done.

Overview of the major user visible changes in the release would be nice at the start. Maybe something like this:

• Metric names are changed to conform to OpenTelemetry standard: (link to Wiki doc)
• Several plugins were refactored to use new "metric families" and labels, to support this. To build plugins that have not been ported over, new --enable-compatibility-mode option needs to be used
• Write plugins (Prometheus...?) are changed to convert new metric & label names for their own formats: (link to Wiki doc)
• Sysman (Intel GPU) input + OTEL output plugins were added, XXX plugins removed

=> It's better if Wiki metrics table lists in separate column(s) the new metrics names also for each write plugin which cannot output OTEL names as-is.

[octo]

Good news everyone! All PRs in the "6.0 MVP" milestone have been merged! That means my next goal is to get collectd 6 into some users' hands.

I created new draft release notes. Please take a look. (The URL appears to change with each update: if the link doesn't work anymore, navigate to https://github.com/collectd/collectd/releases and look for the release.)

[faxm0dem]

The latest listed release is 5.12.0 I guess you need to tag it

[octo]

Okay, I'll tag it once #4251 is in.

[eero-t]

I created **[new draft release notes](https://github.com/collectd/collectd/releases/tag/untagged-

Looks good!

IMHO following would still be good items to highlight in overviews for all releases up to final one:

• Metric names are changed to conform to OpenTelemetry standard: (link to OTEL Wiki doc)
• Plugins are refactored to use "metric families" and labels, to support this: (link to changes Wiki table)
• Use one write thread per write plugin, to prevent stalling plugins blocking other write plugins from working

I.e. have direct link to metric changes (updates one needs to do in Grafana dashboard etc), and reason for such breaking changes.

[octo]

I have created the 6.0.0.rc0 release, uploaded new tarballs, updated the website, and sent an announcement email.

[eero-t]

IMHO following would still be good items to highlight in overviews for all releases up to final one:

• Metric names are changed to conform to OpenTelemetry standard: (link to OTEL Wiki doc)
• Plugins are refactored to use "metric families" and labels, to support this: (link to changes Wiki table)

No need for overview section, "Read plugins" section has link to Wiki, which has link to OTEL info.

• Use one write thread per write plugin, to prevent stalling plugins blocking other write plugins from working

But this info could be added to the "Write plugins" section, as it's a major behavior change for all of them.

[eero-t]

Wiki is missing some of the changes: https://github.com/collectd/collectd/wiki/collectd-6

Config option names were changed for all plugins with changed metrics, not just memory plugin (I still disagree with the Usage/Utilization option names, and would much prefer Absolute/Ratio instead though).

[eero-t]

As to new 5.x release. What if 5.13 is released first, and then if that seems fine, pick few most relevant fixes from that to a 5.12.1 release (at least CI fixes)?

• main =>collectd-5.13: Earlier there were about 2 releases per year, but it's been >2 years since last release, so having new one rather sooner than later would be good to show that project is alive again... Here are few things that may be good to handle before that though:

Updated list of what IMHO would be good to handle before 5.13 release:

• https://github.com/collectd/collectd/issues/4254
• https://github.com/collectd/collectd/issues/4245
• https://github.com/collectd/collectd/issues/4240
  • https://github.com/collectd/collectd/pull/4247
• https://github.com/collectd/collectd/issues/4237
• https://github.com/collectd/collectd/issues/4225
• https://github.com/collectd/collectd/issues/4127
  • https://github.com/collectd/collectd/pull/3974
  • https://github.com/collectd/collectd/pull/4128
• https://github.com/collectd/collectd/issues/4061
• https://github.com/collectd/collectd/pull/4238
• https://github.com/collectd/collectd/pull/3973
• https://github.com/collectd/collectd/pull/3952
• https://github.com/collectd/collectd/pull/3936
• https://github.com/collectd/collectd/pull/3781

There are also couple of older issues that IMHO might be good to look at, if somebody has time:

• https://github.com/collectd/collectd/issues/3444
• https://github.com/collectd/collectd/issues/3440
• https://github.com/collectd/collectd/issues/3373
• https://github.com/collectd/collectd/issues/3081
• https://github.com/collectd/collectd/issues/2511
• https://github.com/collectd/collectd/issues/2465
• https://github.com/collectd/collectd/issues/2413
• https://github.com/collectd/collectd/issues/2411
• https://github.com/collectd/collectd/issues/2031
• https://github.com/collectd/collectd/issues/2180
• https://github.com/collectd/collectd/issues/1748

(Especially https://github.com/collectd/collectd/issues/2465 would IMHO be nice to untangle, as there are several related tickets and still-open old PRs, that could be closed along with it.)

I've pinged some of those and some other old bugs. IMHO near decade old tickets with "Pending feedback" and "Waiting for response" labels could be just closed with a comment asking for them to be re-opened if they're still relevant with collectd 5.12.

EDIT: added couple of other relevant looking (easy) old bugs.