lib: Memory spike reduction for sh cmds at scale

[raja-rajasekar]

The output buffer vty->obuf is a linked list where each element is of 4KB. Currently, when a huge sh command like is executed on a large scale, all the vty_outs are processed and the entire data is accumulated. After the entire vty execution, vtysh_flush proceeses and puts this data in the socket (131KB at a time).

Problem here is the memory spike for such heavy duty show commands.

The fix here is to chunkify the output on VTY shell by flushing it intermediately for every 128 KB of output accumulated and free the memory allocated for the buffer data.

This way, we achieve ~25-30% reduction in the memory spike.

Fixes: #16498 Note: This is a continuation of MR #16498

[donaldsharp]

As a note this is a resubmittal of #16498 because that work was done by an intern and they have gone back to school and we do not expect them to continue this work at this moment. We just wanted to get it in. Rajasekar has addressed all the comments from that PR.

[eqvinox]

I don't think this change is correct, we can't call vtysh_flush() from vty_out() as that can unexpectedly close the vty. This may need to be reverted and rethought / reworked.

(Still digging through/evaluating, may still change my view on further reading/input.)