scoring-load-balancer-plugin icon indicating copy to clipboard operation
scoring-load-balancer-plugin copied to clipboard

Add workaround to prevent simultaneous builds from starting on the wrong node (#30)

Open voruti opened this issue 1 year ago • 3 comments

see Discussion at https://github.com/jenkinsci/scoring-load-balancer-plugin/issues/30#issuecomment-2182665516

Testing done

With this workaround, all builds are scheduled on the correct node, but there also is an additional/artificial delay added to the later builds:

Jun 22, 2024 11:48:00 AM INFO jp.ikedam.jenkins.plugins.scoringloadbalancer.ScoringLoadBalancer reportScores
Scoring for hudson.model.FreeStyleProject@559cbb7c[TestJob1]:
          jenkinsagent: 1000
                      : -500

Jun 22, 2024 11:48:03 AM INFO jp.ikedam.jenkins.plugins.scoringloadbalancer.ScoringLoadBalancer reportScores
Scoring for hudson.model.FreeStyleProject@60ed73e0[TestJob2]:
          jenkinsagent: 1000
                      : -500

Submitter checklist

  • [x] Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • [x] Ensure that the pull request title represents the desired changelog entry
  • [x] Please describe what you did
  • [x] Link to relevant issues in GitHub or Jira
  • [x] ~Link to relevant pull requests, esp. upstream and downstream changes~
  • [x] Ensure you have provided tests - that demonstrates feature works or fixes the issue

voruti avatar Jun 22 '24 09:06 voruti

Can you please fix the conflict?

viceice avatar Jul 08 '24 08:07 viceice

  • [x] make it configurable
  • [x] Tests
  • [x] ~Japanese translation/strings (I can't do this myself)~

voruti avatar Jul 13 '24 12:07 voruti

@viceice Any suggestion on how we can proceed with this? Should we skip the missing translation?

voruti avatar Sep 26 '24 07:09 voruti

@daniel-beck @timja Sorry for pinging, but do you have any suggestions about this particular issue? This workaround looks wiered to me.

viceice avatar Dec 02 '24 09:12 viceice

I don't understand it enough, better to ask on the mailing list with a concise description of the problem, linking here

timja avatar Dec 02 '24 10:12 timja

it seems it needs somwe test fixes too

viceice avatar Jan 23 '25 10:01 viceice

it seems it needs somwe test fixes too

The failing test testTwoSimultaneousBuilds_runsOnWrongNodeWithoutWorkaround indicates that the workaround is no longer necessary. I'll dive into the Jenkins source code to find out what's going on...

voruti avatar Jan 23 '25 20:01 voruti

I'll dive into the Jenkins source code to find out what's going on...

Strange and unstable things are going on. The following log is without the workaround:

2025-01-23 21:35:00.547+0000 [id=142]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@17dddd48[TestJob1]:
                 Node3: 1000
                      : -500
2025-01-23 21:35:00.550+0000 [id=142]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@44b16915[TestJob2]:
                      : -500
2025-01-23 21:35:30.757+0000 [id=147]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@44b16915[TestJob2]:
                 Node3: 1000
                      : -500
2025-01-23 21:35:32.538+0000 [id=149]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@17dddd48[TestJob1]:
                 Node3: 1000
                      : -500
2025-01-23 21:36:02.463+0000 [id=152]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@44b16915[TestJob2]:
                 Node3: 1000
                      : -500
2025-01-23 21:36:04.246+0000 [id=154]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@17dddd48[TestJob1]:
                 Node3: 1000
                      : -500
2025-01-23 21:37:00.534+0000 [id=159]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@17dddd48[TestJob1]:
                 Node3: 1000
                      : -500
2025-01-23 21:37:00.535+0000 [id=159]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@44b16915[TestJob2]:
                      : -500
2025-01-23 21:38:00.534+0000 [id=166]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@17dddd48[TestJob1]:
                 Node3: 1000
                      : -500
2025-01-23 21:38:00.534+0000 [id=166]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@44b16915[TestJob2]:
                      : -500
2025-01-23 21:39:00.534+0000 [id=174]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@17dddd48[TestJob1]:
                 Node3: 1000
                      : -500
2025-01-23 21:39:00.534+0000 [id=174]   INFO    j.i.j.p.s.ScoringLoadBalancer#reportScores: Scoring for hudson.model.FreeStyleProject@44b16915[TestJob2]:
                      : -500

Sometimes the (new) default behaviour of Jenkins chooses the correct nodes/executors, sometimes it doesn't, just as before.

So the workaround is still required to get to choosing correct nodes all the time. Testing that the workaround is required is just not that easy, as the results are unstable.

voruti avatar Jan 23 '25 22:01 voruti

do you find the users who changed the Jenkins code recently? so we could ask them for help?

viceice avatar Jan 27 '25 22:01 viceice

do you find the users who changed the Jenkins code recently? so we could ask them for help?

No, unfortunately I didn't find a code location that would explain this behavior.

voruti avatar Jan 28 '25 07:01 voruti

this is now conflicted, please deconflict. I'll merge the workaround then.

viceice avatar Mar 26 '25 09:03 viceice

this is now conflicted, please deconflict.

I need to add the configuration as code stuff for that. Maybe I have time on the weekend.

voruti avatar Mar 26 '25 20:03 voruti

please fix merge conflicts 😇

viceice avatar Apr 02 '25 12:04 viceice