BAPCtools icon indicating copy to clipboard operation
BAPCtools copied to clipboard

Clean up timeouts

Open RagnarGrootKoerkamp opened this issue 2 years ago • 2 comments

  • There is the global program timeout for generators/visualizers/building (i.e.anything not submissions), which defaults to 30s and is overridden by --timeout for e.g. bt generate
  • There is a per problem submission timeout, which is derived from the timelimit and overridden using --timeout on bt run.

This makes for some inconsistency because now timeout is both a program and problem property.

We could 'fix' the code and always make timeout a program property, which is either derived from the global timeout or from the problem timeout, but I'm not sure that's actually worth fixing.

I think we should fix the duplicate --timeout flag name, since this makes bapctools.yaml configuration a bit annoying otherwise. I'm not exactly sure how though. One option is to give a different name to one of them, but have -T as shortcut for both anyway. (-t is currently the shortcut for timelimit already.)

I'm not sure whether --timelimit is more common/useful for the problem or generate case. For problems, I typically just change the timelimit, not the timeout, if I want to see how long things take. On the other hand, although rare, I could see generators taking more than 30 seconds.

CC @mpsijm

RagnarGrootKoerkamp avatar Nov 12 '21 21:11 RagnarGrootKoerkamp

For problems, I typically just change the timelimit, not the timeout, if I want to see how long things take.

I agree with that one, so in that sense I'd say we could rename/rework how the timeout works. As I have written in https://github.com/RagnarGrootKoerkamp/BAPCtools/pull/163#discussion_r748593805, we could change the submission timeout to work more like Kattis' safety_margin, so that the distinction with a Program timeout is clearer?

mpsijm avatar Nov 12 '21 22:11 mpsijm

Implementing the limits: options as problemtools uses them is a long outstanding issue anyway, so safety_margin is definitely something we should do. Still, on the command line you would want to pass an absolute time, not a factor, I think.

Thinking about it, timeout: in a bapctools.yaml really should be for the generators, not the problem. So maybe the cleanest is to just drop the run --timeout flag (or make it do the same as generate --timeout), and force the user to do this using safety_margin in the problem config or by just doing -t/--timelimit. Alternatively, we could make --timeout override both generator and submission timeouts when it is specified on the command line, but that's probably confusing.

RagnarGrootKoerkamp avatar Nov 12 '21 22:11 RagnarGrootKoerkamp