fix(scheduler): Skip bad experiments when restoring from cache

[sakoush]

In some cases we ended up with a experiment db state that contains experiments that fail validation (unclear how yet). This PR skips loading experiments if they fail validation as they cannot be loaded. Importantly it will not fail the scheduler from starting.

Note that for pipelines we do not have a validation step when restoring from disk.

fixes: INFRA-1055 (internal)