docs for configuration files

[maha-hajja]

Description

some documentation for configurations files, how to use them, and yaml file format

Quick checks:

• [ ] I have followed the Code Guidelines.
• [ ] There is no other pull request for the same update/change.
• [ ] I have written unit tests.
• [ ] I have made sure that the PR is of reasonable size and can be easily reviewed.

[neovintage]

@maha-hajja I don't understand the status field in the schema. Why would I ever do anything besides "running"?

[maha-hajja]

@neovintage you're mostly going to have your pipeline as running, but in some cases, it could be helpful to have it as stopped too:

• maybe the pipeline is only one part of a bigger project, so you could prepare the pipeline and have it ready in conduit, then set the status to running once everything else is in place
• maybe the pipeline needs to be stopped, but should stay in conduit because you want to resume it at a later point in time
• maybe you have a pipeline that you want to only run on demand (e.g. a pipeline that processes DLQ messages)

credit to @lovromazgon for giving these examples.

[lovromazgon]

Whoops, submitted the review a tad too early - I wanted to point out that you did a nice job, I think the doc gives a nice overview. I personally would prefer if we provided more info (e.g. what happens if a file is invalid, what happens if multiple pipelines have the same ID, what if the pipeline folder is not found, the fact that Conduit searches only for files with extensions .yml and .yaml recursively in all subfolders, that env variables are supported, that Conduit won't reload changed files unless it's restarted, that connector IDs and processor IDs will get the parent ID prefixed...). But even the current info in the doc is already very valuable.