`esm_datastore_v2`: A rewrite of `esm_datastore` with new features + improvements

[andersy005]

TO DO

• Introduce pydantic models that allow us to validate the catalog against a pre-defined schema. This has the potential of reducing code complexity in the existing esm_datastore catalog object.
  • [x] #347
  • [x] https://github.com/intake/intake-esm/pull/367
• Public methods to expose (https://github.com/intake/intake-esm/pull/368)
  • [x] __init__
  • [x] __len__
  • [x] __getitem__
  • [x] __contains__
  • [x] to_dataset_dict()
  • [x] nunique()
  • [x] unique()
  • [x] search()
  • [x] https://github.com/intake/intake-esm/pull/370
  • [x] https://github.com/intake/intake-esm/pull/373
  • [x] https://github.com/intake/intake-esm/pull/374
  • [x] https://github.com/intake/intake-esm/pull/375
• Merge intake_esm.source:ESMGroupDataSource and intake_esm.source:ESMDataSource into a single data source intake_esm.source:ESMDataSource: https://github.com/intake/intake-esm/pull/372
• #352 proposes creating a dict like object and returning this object as the output of to_dataset_dict(). This object will look and smell like a dictionary but will have additional functionality to facilitate applying operators on all returned datasets and/or a subset of these datasets
• Add Derived Variable functionality
  • https://github.com/intake/intake-esm/pull/379
  • https://github.com/intake/intake-esm/issues/388
  • https://github.com/intake/intake-esm/issues/387
• #163 proposes providing users with functionality to control how dataset aggregations are done. Currently, the aggregation process is rigid. A rewrite of esm_datastore will allow us to explore exposing aggregation logic to users via a registry of operators/preprocessors.
• [ ] Update documentations to reflect new changes

[mgrover1]

@andersy005 what still need to get done here? Are we still missing the docs?

[andersy005]

For this particular issue, there is one pending issue that needs to be addressed

• https://github.com/intake/intake-esm/issues/511

Regarding the docs, everything looks good. I plan to organize what we currently have in a separate PR.

For the release, here's a list of outstanding issues that i think are worth addressing before the next release: https://github.com/intake/intake-esm/milestone/11

[mgrover1]

@andersy005 Can we close this now that #511 is good to go?