satpy
satpy copied to clipboard
Add Gld360 ualf2 reader
VAISALA GLD360 ualf2 lightning data reader added.
- [x] Tests added
- [x] Fully documented
- [x] Add your name to
AUTHORS.md
if not there already
Suggestions on how to speed up performance are welcomed. I have compared the performance with the EUM internal equivalent of this reader. The only difference is that the EUM internal reader uses Pandas everywhere where this reader uses dask. Processing 24 hourly files with a total of ~2.5 million records. The EUM internal reader does it in 3 minutes whereas the dask version here takes 6 minutes for the same task.
Played around a bit with the blocksize
argument in dask read_csv() as suggested by @sjoro
-
blocksize
not defined => runtime 6 minutes -
blocksize=1e6
=> runtime 9 minutes -
blocksize=10e6
=> runtime 6 minutes -
blocksize=25e6
=> runtime 6 minutes -
blocksize='default'
=> runtime 7 minutes
good job on the first PR @seenno ! maybe other can comment on the unit tests... maybe to chop it up in smaller pieces and tests? test data creation could be done outside of test_vaisala_gld360
-method?
@seenno Any chance on going through the review comments and resolving the merge conflicts?