arriven
Scale up automatically based on available file descriptors
Why don't we (I may try too) implement something like checking the (maximum) available number of file descriptors (possible ports / sockets / connections) on app's start and automatically scaling the number of jobs up to something close to this limit?
On my Mac the default max number of descriptors was something like 255, I raised it to the max allowed number of 10560 or something around that. I tried running with --scale 11 which created 8624 jobs instead of default 766 (approximate number, don't recall exactly), which isn't even the limit.. I could run with a higher scale factor. And I experience no problems whatsoever - and the report numbers are looking much more promising.
We could detect the max allowed number, print out a suggestion how to increase it and or include it in the doc, and use all available connections instead of some default. Potentially it would increase "productivity" ten-folds.
idea based on this comment https://github.com/Arriven/db1000n/issues/491#issuecomment-1090686644
that would probably be something more fit for https://github.com/Arriven/db1000n/discussions/500
I meant the app itself assessing available resources and taking up as much as is possible. But after a bit of experimentation I see problems there - e.g. it was a piece of cake for my Mac, but somehow connections started breaking much quicker on Win instances - they couldn't hold even 10 000 connections even though on Win the number of file handles (which I believe are also used for connections) is in millions by default. Maybe it has to do with default C runtime limits which are said to be much lower..
And the number of file descriptors could be much higher than net's resources or VPN resources - which even seem to differ between servers.
Anyway, a corrected idea - assess the maximum number of available file descriptors, begin starting new jobs in batches until problems arise - like inability to open a connection - then scale a bit down. Sounds more complicated, but I guess doable. I'll be thinking it over more thoroughly.