copr icon indicating copy to clipboard operation
copr copied to clipboard

Allow for messages per build, instead of per chroot

Open penguinpee opened this issue 1 year ago • 14 comments

While reporting https://github.com/fedora-infra/fmn/issues/1138 I thought this particular case probably belongs to Copr. FMN notifications are finally working and I was excited at first. But that's no longer the case.

I've been flooded by useless messages today. Copr appears to be sending messages per chroot. If I remember correctly (I've cleaned up my mailbox meanwhile) it sends one for each of the following per chroot:

  1. SRPM rebuild starts
  2. SRPM rebuild finishes
  3. Build starts
  4. Build finishes

So, with a Copr configured for several branches and arches, this adds up quickly. I'm mostly not interested in state changes per chroot. I'd like to know when a build starts. That is when it starts running. And I'd like to know when it finishes. In those messages the information per chroot can be accumulated. E.g.:

fedora-rawhide-x86_64: succeeded
fedora-rawhide-aarch64: failed
fedora-rawhide-s390x: skipped

Compare it to looking at the list of builds vs. looking at a specific build. Before FMN started working, I would usually keep an eye on the build list, refreshing every so often until a build was finished. Only then would I look at the outcome per change root. And mostly only when a build failed. Because it might have failed only for a specific chroot.

More informative messages also make for better opportunities filtering thos messages as pointed out in https://github.com/fedora-infra/fmn/issues/1138.

Koji does it better. It sends a long messages when a build is completed successfully containing all the details of that build.

penguinpee avatar Apr 26 '24 20:04 penguinpee

I just realized that we already do this?

  • https://apps.fedoraproject.org/datagrepper/v2/search?topic=org.fedoraproject.prod.copr.build.start
  • https://apps.fedoraproject.org/datagrepper/v2/search?topic=org.fedoraproject.prod.copr.build.end

FrostyX avatar May 24 '24 13:05 FrostyX

Aren't those the per chroot messages I was "complaining" about?

Example:

"body": {
    "build": 7484770,
    "chroot": "fedora-40-x86_64",
    "copr": "ros2",
    "ip": "2620:52:3:1:dead:beef:cafe:c1cb",
    "owner": "tavie",
    "pid": 1288763,
    "pkg": "ros-jazzy-libcurl-vendor",
    "status": 3,
    "user": "tavie",
    "version": "3.4.2-1",
    "what": "build start: user:tavie copr:ros2 pkg:ros-jazzy-libcurl-vendor build:7484770 ip:2620:52:3:1:dead:beef:cafe:c1cb pid:1288763",
    "who": "backend.worker-rpm_build_worker:7484770-fedora-40-x86_64"
  }

What I'm looking for is something akin to:

https://apps.fedoraproject.org/datagrepper/raw?topic=org.fedoraproject.prod.buildsys.task.state.change

penguinpee avatar May 24 '24 15:05 penguinpee

Oh, you are right, sorry about that. They are really badly named :-/

FrostyX avatar May 24 '24 22:05 FrostyX

I also started getting Matrix notifications from COPR (first noticed them today), and they are ... bad:

Screenshot from 2024-06-15 21-08-57

It appears that there's one message per chroot, and the message contents are just the "topic" or whatever it's called, but no actual information (build number, URL, chroot name, etc.).

decathorpe avatar Jun 15 '24 22:06 decathorpe

I think we could set the chroot messages to have severity = DEBUG, no? There are always build messages associated with them. With this, people wouldn't get notified by default, as the default level is INFO (in fedora-messaging and in FMN).

abompard avatar Jun 24 '24 10:06 abompard

I missed this getting closed. However, looking at the discussion in the PR, it seems there are still some changes pending. Time to reopen?

penguinpee avatar Nov 09 '24 18:11 penguinpee

Not sure :shrug: what changes do you mean? I know there was some problem with non-stringified objects being part of e-mail subjects of something, probably related to https://github.com/fedora-copr/copr/pull/3344#discussion_r1770988337 but I can not find the issue now.

praiskup avatar Nov 11 '24 09:11 praiskup

I can not find the issue now.

It is #3493.

praiskup avatar Nov 11 '24 09:11 praiskup

Well, it seems I was partially mislead by Kevin's post in the Matrix room. I just re-enabled my Copr rule in FMN and it seems the issue described in #3493 is affecting me as well.

However, the issue described in this ticket originally (also see the linked ticket filed against FMN), appears to be still present as well. That is Copr sends one message per chroot instead of one message per build. This is also acknowledged in https://github.com/fedora-copr/copr/pull/3344#issuecomment-2273233282.

I'm unable to tell if #3344 brought any improvement with regards to the message content due to #3493. But it seems to me the original issue is not fully fixed.

penguinpee avatar Nov 11 '24 11:11 penguinpee

That is Copr sends one message per chroot instead of one message per build.

Ah, yes, that's right. We should reopen this issue - the RFE clearly asks to have per-build messages. Not trivial to implement, but a valid reqeust.

praiskup avatar Nov 11 '24 16:11 praiskup

uh ... something changed here, but not for the better 😆

[Copr] <bound method BuildChrootEnded.summary of BuildChrootEndedV1(id='e8c7c3df-f5b8-4a85-8e7f-ce6c2fdfb538', topic='[org.fedoraproject.prod.copr.build](http://org.fedoraproject.prod.copr.build/).end', body={'user': 'decathorpe', 'copr': 'rust2rpm-27', 'owner': 'decathorpe', 'pkg': 'rust2rpm', 'build': 8299670, 'chroot': 'fedora-rawhide-x86_64', 'version': '27.0.0~rc1-1', 'status': 1, 'ip': '2620:52:3:1:dead:beef:cafe:c18d', 'who': 'backend.worker-rpm_build_worker:8299670-fedora-rawhide-x86_64', 'pid': 2879276, 'what': 'build end: user:decathorpe copr:rust2rpm-27 build:8299670 pkg:rust2rpm version:27.0.0~rc1-1 ip:2620:52:3:1:dead:beef:cafe:c18d pid:2879276 status:1'})>

decathorpe avatar Nov 22 '24 00:11 decathorpe

Ah, this looks like the summary() method is missing its @property decorator.

abompard avatar Nov 22 '24 10:11 abompard

@nikromen you might be interested in this in regards to #3493

FrostyX avatar Nov 22 '24 10:11 FrostyX

WRT the summary() problem, please follow #3493 - this one is a Copr RFE :)

praiskup avatar Nov 23 '24 21:11 praiskup