Add support for ephemeral services.

[veluca93]

Ephemeral services are services that are not fixed in the configuration file, but dynamically added as they connect. This is especially useful in a setup in which cmsWorker/cmsContestWebServer are scaled dynamically, as one might do when configuring CMS for running on cloud services.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 71.15385% with 30 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@b22e14c). Learn more about missing BASE report. Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
cms/conf.py	60.41%	19 Missing :warning:
cms/util.py	65.00%	7 Missing :warning:
cms/service/workerpool.py	76.92%	3 Missing :warning:
cms/server/contest/server.py	83.33%	1 Missing :warning:

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1302   +/-   ##
=======================================
  Coverage        ?   68.01%           
=======================================
  Files           ?      332           
  Lines           ?    26623           
  Branches        ?        0           
=======================================
  Hits            ?    18108           
  Misses          ?     8515           
  Partials        ?        0           

Flag	Coverage Δ
functionaltests	46.05% <58.58%> (?)
unittests	55.51% <48.07%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Virv12]

@wil93 I added a few lines of doc, it's not much but there is not much to say.

[wil93]

I guess the updates to the cms.conf.sample file can also be considered part of the documentation, in a way. However I am still worried it might be unclear how to this new feature works... Some questions:

Is "not providing a shard number" the one and only way to opt in to ephemeral services? If I have both the static and the ephemeral configuration in place in cms.conf, what happens if I launch cmsResourceService -a? Also, can any service can be ephemeral, or is it just Worker and ContestWebServer? If it's any service, couldn't we simplify the definition to something like the following?

"ephemeral_services":
    {
            "subnet": "127.0.0.0/8",
            "min_port": 26000,
            "max_port": 26999
    },

For the contest_listen_address we now say "When using ephemeral services only the first address and port are used", what does this mean?

[wil93]

Also, and this could be a bad idea, but would it be possible to introduce ephemeral services as more "clean cut" way of launching things, i.e. force our users to choose either "all ephemeral" or "old way", instead of letting our users run CMS with a mix of ephemeral and non-ephemeral? Would it make sense?

[Virv12]

Also, can any service can be ephemeral, or is it just Worker and ContestWebServer?

Any service can be ephemeral, but only cws and worker have been tested.

If it's any service, couldn't we simplify the definition to something like the following?

I guess we could, we will lose the possibility the have different services in different subnets but I guess it's not a big problem.

Also, and this could be a bad idea, but would it be possible to introduce ephemeral services as more "clean cut" way of launching things, i.e. force our users to choose either "all ephemeral" or "old way", instead of letting our users run CMS with a mix of ephemeral and non-ephemeral? Would it make sense?

Could we? Yes. Does it make sense? I don't think, sometimes it's useful to mix the two kind of services, mostly to debug or quickly try something.

[wil93]

The benefit of having a clean-cut "switch" between hardcoded and ephemeral services is that it's less logic to maintain (we don't need to even test the case where we run with a mix of hardcoded and ephemeral services, since that would not be possible) and it also allows us (if we so choose) to deprecate the hardcoded style later on, and only support a single way of declaring services.

Also, would something like this help us to implement ephemeral services with less code? https://github.com/python-zeroconf/python-zeroconf

[veluca93]

The benefit of having a clean-cut "switch" between hardcoded and ephemeral services is that it's less logic to maintain (we don't need to even test the case where we run with a mix of hardcoded and ephemeral services, since that would not be possible) and it also allows us (if we so choose) to deprecate the hardcoded style later on, and only support a single way of declaring services.

I don't think the increase in logic is that significant.

Also, would something like this help us to implement ephemeral services with less code? https://github.com/python-zeroconf/python-zeroconf

Multicast has a bad habit of not working in a surprising number of environments, so I would not rely on that.

[gollux]

I would appreciate better documentation. In particular, it should define what an ephemeral service is (instead of just mentioning that it can be used in some cases) and what is the meaning of shard numbers for such services. Also, it should specify which services can be ephemeral (I guess it's workers and the CWS now).

[veluca93]

@Virv12 what's the status here?