perf(clustering): optionally use privileged agent for control plane connection

[bungle]

Summary

Data plane's connection to control plane is moved to a privileged agent worker process, including:

• maintaining websocket (wrpc) connection and data transfer
• decompression of received data
• json decoding of the received data
• validation and flattening of received data
• inserting data to lmdb

(so that these won't affect latencies / rps on proxy workers)

This time behind configuration flag, and disabled by default.

See previous attempts: #9432 #8971

KAG-114

[bungle]

TODO:

• try to optimize memory usage of privileged worker to minimum.
• add some unit/integration testing (e.g. that it really starts extra worker)

[oowl]

This is so amazing, I will look forward to Kong offloading all no proxy-related logic to privileged worker, And I have another concern, Is there any need for us to consider the downgrade permissions of privileged workers, Some control plane logic doesn't need root privileged, Using root user to do these things maybe have some security issues?

[bungle]

Some control plane logic doesn't need root privileged, Using root user to do these things maybe have some security issues?

That is the case currently that none of the logic needs (root or the user that started kong) privileges (obviously because we have not had it before). Though we might need it later, and it certainly opens new possibilities to have those privileges. For example we have been discussing about auto-updating data planes and stuff like that. Perhaps multiple processes as side cars. Also there seems to not be a way to run privileged agent in non-privileged way or there is not non-privileged agent. But as the tasks that we put there are fairly isolated (mostly internal logistics and background tasks), I don't see that as a huge problem. We already run some of our code in init phase (which is a privileged phase, even without privileged worker).

In many environments you can use port mapping and stuff and not run or even start Kong with privileged rights (e.g. root). Usually root is needed only when we need to bind listening ports below 1024.

[bungle]

I managed to reduce memory usage a lot.

Without plugins and vaults:

With bundled plugins and vaults:

So on Kong CE the privileged agent takes something between 20 and 50 MB (depending on how many plugins / vaults you have, schemas seem to take some 20+ MB of its memory).