feldera icon indicating copy to clipboard operation
feldera copied to clipboard

Make it clear that Docker form factor does not respect resource limits

Open lalithsuresh opened this issue 1 year ago • 5 comments

@Karakatiza666 @snkas we should either:

  • remove the red line on the graph for now and only bring it up in the cloud form factor where we do respect those resource limits
  • or actually enforce the resource limits per pipeline even in docker.

lalithsuresh avatar Sep 03 '24 01:09 lalithsuresh

Is there a way to identify whether Feldera is running in cloud form factor?

Karakatiza666 avatar Sep 03 '24 12:09 Karakatiza666

Currently not directly, I could add one, though it'd be best to have it be related to runner info, as the runner is the one which configures limit enforcement.

snkas avatar Sep 03 '24 12:09 snkas

@snkas might be as simple as returning an "edition" field in the configuration, which we set when running the pipeline-manager via --edition (local, docker, enterprise).

lalithsuresh avatar Sep 03 '24 14:09 lalithsuresh

Differentiating by the "edition" is the nicest solution, but: Is it a common occurrence for Docker form-factor to have memory limits? We could specify in docs that this is not enforced in Docker, and there is no limit by default, so the user would have to specify the limit themselves, ignoring the docs?

Also, if we introduce distinctions of current edition there probably needs to be a back-end check to not allow non-null memory limit when current is "Docker" in the first place?

Karakatiza666 avatar Sep 03 '24 15:09 Karakatiza666

Is it a common occurrence for Docker form-factor to have memory limits? We could specify in docs that this is not enforced in Docker, and there is no limit by default, so the user would have to specify the limit themselves, ignoring the docs?

This was a user reported issue: they thought the red line in the graph meant the pipeline won't exceed that memory use but it did.

lalithsuresh avatar Sep 03 '24 15:09 lalithsuresh