osbuild-composer icon indicating copy to clipboard operation
osbuild-composer copied to clipboard

Improvements to dbjobqueue

Open lzap opened this issue 2 years ago • 11 comments

Couple of commits:

  • Because slogger was imported from internal package, it was not possible to import jobqueue effectively. Go was erroring out, the change moves the simple logger package to pkg/ and also moves the interface into a separate file where it belongs. Both noop and logrus implementations were moved into own packages so it's now nice and clean - these can be imported without extra dependencies and the interface itself can be also imported without logrus dependency.
  • Another small commit allows passing in an existing pgxpool so we can reuse the one we already have in our app. We migrated from database/sql to the native pgx last week.
  • Finally, when I was testing this, I needed to create a migration for the tests so I am dropping a Makefile target to do that so others don't need to figure it out

Edit: Initially the PR had some changes around context passing, but this is worth looking into a separate PR I think.

Cheers!

lzap avatar Oct 06 '22 08:10 lzap

I am gonna drop the context change, that's really weird, all functions in the dbjobqueue interface should really have a ctx parameter.

lzap avatar Oct 06 '22 08:10 lzap

What's really weird is the dbjobqueue tests passed the first time locally, now I am getting error inserting the job's heartbeat: ERROR: insert or update on table "heartbeats" violates foreign key constraint "heartbeats_id_fkey" (SQLSTATE 23503) even if I drop the db and recreate it from scratch.

Looks like this is passing on the CI. Weird stuff.

lzap avatar Oct 06 '22 09:10 lzap

Rebased, I've also implemented the suggested CI change.

lzap avatar Oct 07 '22 09:10 lzap

Looks like some checks are stuck, shall I rebase once again?

lzap avatar Oct 17 '22 08:10 lzap

Looks like some checks are stuck, shall I rebase once again?

image

Github is actually throwing up an error, there needs to be a - in front of the run. (or - name: ...\n)

croissanne avatar Oct 17 '22 08:10 croissanne

Sure, rebased. Crossing my fingers :-)

lzap avatar Oct 18 '22 13:10 lzap

And once again, Mr. Schutz had some issues, but the page was 500 so not sure.

lzap avatar Oct 21 '22 08:10 lzap

And once again, Mr. Schutz had some issues, but the page was 500 so not sure.

The CI takes quite long, and often has at least 1 or 2 infra flakes. Easier to just retry the individual jobs.

croissanne avatar Oct 21 '22 08:10 croissanne

I dont understand, Schutzbot job reports an error, when I click on the link it says it's running.

And the RPM, oh, I almost forgot how bad rpmbuild is with reporting errors. It fails but I am not following why.

lzap avatar Oct 21 '22 09:10 lzap

I dont understand, Schutzbot job reports an error, when I click on the link it says it's running.

And the RPM, oh, I almost forgot how bad rpmbuild is with reporting errors. It fails but I am not following why.

fedora37 ppc and centos tream 9 are also failing on other PRs; it's unrelated. I'll keep retrying schutzbot and merge once green

croissanne avatar Oct 21 '22 11:10 croissanne

Ok let me know if I need to rebase or something. Thanks.

lzap avatar Oct 21 '22 11:10 lzap

It looks like our tasking system will be more simple than I thought, we are aiming to use very simple "deffered function call" style of execution and we ended up not using dependant jobs at all, therefore, I am closing this PR. Thank you for the effort, apologies for that you worked on extracting the jobqueue to pkg/ it looks like it won't be used at the moment.

lzap avatar Jan 13 '23 09:01 lzap