sea explorer white space in payload files

[clayton33]

We have a number of missions from year 2018 with navigation firmware 2.1.5 and payload firmware 2.1.1. In the raw payload files there are leading whitespaces in both the data and the header which resulted in an error being thrown when converting the data to Float64. This fix resolves that issue.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 54.98%. Comparing base (85011e5) to head (0ca4ef7). Report is 12 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #215      +/-   ##
==========================================
+ Coverage   54.90%   54.98%   +0.07%     
==========================================
  Files           9        9              
  Lines        1692     1695       +3     
==========================================
+ Hits          929      932       +3     
  Misses        763      763              

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jklymak]

Can you explain the data that causes this error a little more explicitly. The added code looks potentially slow and I wonder if there is a better way to strip leading white space available in polars.

[clayton33]

exampleData.zip Attached is some example data.

[jklymak]

Thanks, but what about this data is causing the error? Can you just extract the string that is causing the problem so we can look for solutions?

[clayton33]

There are leading whitespaces throughout the entire file and it's in every payload file, so I think looking at every column that is a str type makes sense. Any files after that navigation and payload firmware don't have that issue, and they'll read in as float64, so these newly added lines will ultimately do nothing in those cases since there won't be any columns of str type.

[jklymak]

Can you include here a string that has a leading white space? I opened a couple of those files and I'm not seeing any leading whitespaces, but maybe we are talking about different things.

[jklymak]

Also, maybe this is a windows/Unix thing? I'm on a Mac and don't see any leading spaces. Perhaps we need to specify the encoding of the files when we open them. It's super unlikely that polars can't handle white space in float conversion, so I think it must be something like that.

[clayton33]

Sorry for the confusion. I sent data for the wrong mission (we had mission 32 in the same year for two different gliders).

exampleData.zip

[richardsc]

@clayton33 can you also post the error that you get when running with our yaml?

[clayton33]

This is the error that I get

polars.exceptions.InvalidOperationError: conversion from `str` to `f64` failed in column ' NAV_RESOURCE' for 58 out of 58 values: [" 105", " 105", … " 117"]

[jklymak]

https://github.com/pola-rs/polars/issues/10587 maybe

I'm actually pretty surprised polars has decided not to have an option for this. @callumrollo ?

[clayton33]

Yes, I had come across that issue. I followed the suggestion from this comment https://github.com/pola-rs/polars/issues/10587#issuecomment-2116459477

[clayton33]

Checking in to see if there has been any thought into incorporating this change.

[jklymak]

Its fine by me, but would be good if @callumrollo chimed in

[richardsc]

Hi @jklymak, @callumrollo! Any thoughts on merging this PR? We've been using it in production and it all seems good on our files (with and without white space).

[richardsc]

Thanks!

[jklymak]

Yes sorry to be slow. Always feel free to ping!