satpy icon indicating copy to clipboard operation
satpy copied to clipboard

Add Gld360 ualf2 reader

Open seenno opened this issue 2 years ago • 8 comments

VAISALA GLD360 ualf2 lightning data reader added.

  • [x] Tests added
  • [x] Fully documented
  • [x] Add your name to AUTHORS.md if not there already

seenno avatar Dec 02 '21 16:12 seenno

Suggestions on how to speed up performance are welcomed. I have compared the performance with the EUM internal equivalent of this reader. The only difference is that the EUM internal reader uses Pandas everywhere where this reader uses dask. Processing 24 hourly files with a total of ~2.5 million records. The EUM internal reader does it in 3 minutes whereas the dask version here takes 6 minutes for the same task.

Played around a bit with the blocksize argument in dask read_csv() as suggested by @sjoro

  • blocksize not defined => runtime 6 minutes
  • blocksize=1e6 => runtime 9 minutes
  • blocksize=10e6 => runtime 6 minutes
  • blocksize=25e6 => runtime 6 minutes
  • blocksize='default' => runtime 7 minutes

seenno avatar Dec 02 '21 16:12 seenno

good job on the first PR @seenno ! maybe other can comment on the unit tests... maybe to chop it up in smaller pieces and tests? test data creation could be done outside of test_vaisala_gld360-method?

sjoro avatar Dec 03 '21 14:12 sjoro

@seenno Any chance on going through the review comments and resolving the merge conflicts?

gerritholl avatar Jul 29 '22 08:07 gerritholl