Adding the capability of reading an entire folder for running knuu instances

[staheri14]

We are about to run a large scale test with 100 nodes, where each of them collect traced data locally inside a specific folder, and we wish to retrieve it from the instances all at once. Since the number of files in that folder varies based on the current state of the codebase (celstia-app), and the number of files can be high, it is less error prone and more reliable to read the entire folder at once, needless to keep track of the exact file names. This feature is not currently supported in knuu and its addition would be really appreciated!

[smuu]

I would propose something like this:

• Having a sidecar that mounts the same PVCs as the main container (specified by using AddVolume())
• The sidecar has logic installed to take files/folders and upload them to a S3 bucket (minio or external)
• The sidecar exposes some CLI or API
• Adding a function to knuu, like BackupFileToS3() that forwards that request to the sidecar

[tty47]

we have discussed this topic to find the best way to solve it, we will add the following feature:

we'll create a new release as soon as it'll be done

cheers!

[smuu]

Ref: https://github.com/celestiaorg/knuu/issues/272