MHKiT-Python icon indicating copy to clipboard operation
MHKiT-Python copied to clipboard

Read issue with Signature ADCP dataset

Open hevgyrt opened this issue 6 months ago • 7 comments
trafficstars

Describe the bug:

I get an OverflowError when reading a subset (using the nens argument) of a large (>10Gb) Signature 500 .ad2cp file. The adcp was measuring in a concurrent mode including the "burst and waves" + the echosounder option.

To Reproduce:

Provide a minimal working example that reproduces the error.

ds = dolfyn.read('my_sig500_file.ad2cp',nens=[2000,20000])

prompts

`File ~/miniconda3/lib/python3.10/site-packages/mhkit/dolfyn/io/api.py:114, in read(fname, userdata, nens, **kwargs) 112 func_map = dict(RDI=read_rdi, nortek=read_nortek, signature=read_signature) 113 func = func_map[file_type] --> 114 return func(fname, userdata=userdata, nens=nens, **kwargs)

File ~/miniconda3/lib/python3.10/site-packages/mhkit/dolfyn/io/nortek2.py:79, in read_signature(filename, userdata, nens, rebuild_index, debug, dual_profile, **kwargs) 75 raise TypeError("nens must be: None (), int, or len 2") 77 userdata = _find_userdata(filename, userdata) ---> 79 rdr = _Ad2cpReader( 80 filename, rebuild_index=rebuild_index, debug=debug, dual_profile=dual_profile 81 ) 82 d = rdr.readfile(nens[0], nens[1]) 83 rdr.sci_data(d)

File ~/miniconda3/lib/python3.10/site-packages/mhkit/dolfyn/io/nortek2.py:157, in _Ad2cpReader.init(self, fname, endian, bufsize, rebuild_index, debug, dual_profile) 155 self.f.seek(0, 2) # Seek to end 156 self._eof = self.f.tell() --> 157 self.start_pos = self._check_header() 158 self._index, self._dp = lib.get_index( 159 fname, 160 pos=self.start_pos, (...) 164 dp=dual_profile, 165 ) 166 self._reopen(bufsize)

File ~/miniconda3/lib/python3.10/site-packages/mhkit/dolfyn/io/nortek2.py:209, in _Ad2cpReader._check_header(self) 206 idx = s.find(c, idx + 1) 208 # Open the entire file --> 209 self._reopen(self._eof) 210 pk = self.f.peek(1) 211 # Search for multiple saved headers

File ~/miniconda3/lib/python3.10/site-packages/mhkit/dolfyn/io/nortek2.py:226, in _Ad2cpReader._reopen(self, bufsize) 224 except AttributeError: 225 pass --> 226 self.f = open(_abspath(self.fname), "rb", bufsize)

OverflowError: Python int too large to convert to C int`

Expected behavior:

Expected a subset of the file to be read

Screenshots:

Desktop (please complete the following information):

  • OS: Ubuntu, Jammy
  • MHKiT Version: 0.9.0

Additional context:

hevgyrt avatar Apr 25 '25 08:04 hevgyrt