Create a Way to Validate the Integrety of a Catalog

[jbouffard]

It would be good to have a way to check the integrity of a catalog; as bad ingests can result in a lot of problems later on.

Possible ways this could be implemented in the API

// validate the files after they are written
def write(uri: URI, store: AttributeStore, validateLayer: Boolean = true)

// validate a layer
def validateLayer(layerId: LayerID)

// validate all layers in a catalog
def validateCatalog(uri: URI)

// validate layer before reading
def read(id: ID, validate: Boolean = true)

[echeipesh]

Some validation would certainly be useful. What failure cases have you seen that prompted this issue?

[jbouffard]

The most recent issue I had was during the GDDP ingest. Certain files would time out when being read, but something would still be written to catalog. When you looked at the tasks for each stage you'd sometimes see 2/2 (1 failed), so I wasn't sure if it completed okay or if there was a problem. Having a way to verify that everything was written correctly would have been useful.