OPAL should have JSON-Schema validator capability

[avosepp]

Describe the solution you'd like A bad OPAL fetch can result in unexpected and hard to troubleshoot errors. Especially if the maintainers are not as familiar with the original system. (Interns, New Hires, etc.) I would like to be able to supply a JSON-Schema along with any fetcher configs. OPAL will validate the output of process() from whatever fetcher got the data against the JSON-Schema. If the validation fails, OPAL should report "unhealthy" on its health check endpoint. Preventing Kubernetes rollouts from continuing a rollout.

[orweis]

Hi @avo-sepp :) This is a great idea - makes total sense.

I think we can start with a guide in the docs on how to do it per data-fetcher you create with Pydantic (you can fork a fetcher for a specific case and enforce it's schema in the process code).

We'll need to think on how to best make this generic- I'm guessing this can be another field in the SourceEntry.

[avosepp]

Yeah, that's exactly what I'm going to do for now in the Cosmos fetcher to get it out the door now. But I think it should be generic eventually. Thanks @orweis!