devsjc

## 19-04-2024 In progress on VM. Expected to take ~2 weeks.

Currently at 16,000/~175,000.

Seems the vast majority of this is now available on huggingface: https://huggingface.co/datasets/openclimatefix/eumetsat-iodc. This might be enough to mark the task as done @peterdudfield? Also, for future reference as part of...

I seem to recall that the images for this dataset were chunked using a 4x4 grid? If x and y are only split into 4 respectively on the large image...

A couple of encounters leaning me in favour of a separation of base library code and import scripts that use the library: - Feel uncomfortable adding package requirements to the...

I'll leave it in here for now, to get my ducks in a row before I start making organisational changes! And will bear the above in mind during any further...

This is still a first pass I would say, can definitely be improved upon if it's use case changes from the occasional non-critical manual import, and once I've a more...

Began script import this morning pulling 2014 onwards.

@jacobbieker I would be grateful at some point if I could get some more info on what happens to this `.hdf` data - so far my understanding from Peter is...

Okay thanks! So it is netcdf that is the format you're importing in the end to train with? And I guess that's via mounting the volume and reading in the...