earthkit-data icon indicating copy to clipboard operation
earthkit-data copied to clipboard

Add option to retrieve data into user specified target files without any parsing

Open sandorkertesz opened this issue 11 months ago • 0 comments

Is your feature request related to a problem? Please describe.

Currently we cannot retrieve data directly into a user defined file, because we cannot pass a file target to from_source, but we have to use e.g.

ds = eathkit.data.from_source("cds", ....)
ds.save("my_file")

Here, the data is first downloaded into a tmp/cache, unpacked when needed, then parsed/interpreted. This all happens before calling save(), which actually performs a file copy operation.

Describe the solution you'd like

A possible API:

ds = eathkit.data.from_source("cds", ...., target="my_file")

where the data will be save into file my_file.

Some questions immediately arise:

  • what would ds be? (it has to be a source)
  • what if multiple files are retrieved? (with the cds source it is possible)
  • what if the user wants to preserve the remote file name?
  • sometimes the type of the retrieved file is not know in advance e.g. it can be a "zip" file. It would be nice to have an option to automatically add the right suffix to the target file. E.g. when this option is used we just need to specify target="my_file" and the data would be saved into my_file.zip.

Describe alternatives you've considered

No response

Additional context

No response

Organisation

ECMWF

sandorkertesz avatar Mar 08 '24 13:03 sandorkertesz