Unable to load very large recordings

[kevincroissant]

I'm unable to open some recordings I made recently. This is the error:

kevin@[hostname]:/media/Quark/recordings$ inspectrum 20160413-195414_COMPLEX_filtered_xlated_costas.cfile 
terminate called after throwing an instance of 'std::runtime_error'
  what():  Error opening file
Aborted

This particular recording is from GNURadio, from a USRP N210 with a sample rate of 250000. It is 247GB, which I believe to be the issue. I also tested with a 443GB recording made the same way with the same results. Interestingly, if I use 'dd' to make a new file with only the first 50MB, it loads fine.

The recordings are complex float32 straight from gnuradio, made with a "file sink" block. This is with the latest inspectrum code as of about 30 minutes ago (~9:00PM EDT, 2016-04-22).

Inspectrum's main appeal to me is its ability to support "large (multi-gigabyte) files". Is this simply too large to deal with? Eventually I will have more recordings like this, and likely even larger.

[miek]

I've pushed up 2bf0b1a5c792a7fd5cbdd7a460583b5569a93d85 which should give a more detailed error, could you try that and let me know what it says?

[kevincroissant]

kevin@[hostname]:/media/Quark/recordings$ inspectrum 20160413-195414_COMPLEX_filtered_xlated_costas.cfile 
terminate called after throwing an instance of 'std::runtime_error'
  what():  Error opening file: Value too large for defined data type (75)
Aborted

[miek]

Cheers. This suggests that inspectrum has been compiled as a 32bit program so fopen is refusing to open any file larger than 2GB. Do you think that could be the case? What arch is the system compiling/running inspectrum?

[kevincroissant]

Hmm. Certainly possible I suppose. Here's my uname -a:

$ uname -a Linux [hostname] 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2 (2016-04-08) x86_64 GNU/Linux

This started out with a i686-pae kernel (by accident) but I switched it over to amd64. I suppose all my packages are still 32 bit though... give me a little time and I'll set up a new system that is truly 64 bit and try again.

[miek]

Also note that while #35 is outstanding, there's an effective limit of about 17GB since scrolling is limited to the first 2^31 - 1 samples

[kevincroissant]

Got it...Would like to eventually be able to see all (or most) of this, but even the first 17GB is helpful. I think I have another machine that has 64 bit packages, so I'll test it later today.

[kevincroissant]

I was able to copy the file to my laptop (definitely 64 bit) and although it opened with no errors, there is an odd visual glitch. See screenshot below. Once again, trimming the file down to 50MB with dd makes it open normally. Changing the power sliders has no discernible effect with the full size file, nor does zooming in/out and changing the FFT size.

Also, enabling cursors and selecting part of the recording shows totally incorrect rate and period data. This makes no difference on the visual glitch, so I have enabled it in the screenshot so you can see it.

[kevincroissant]

I've been thinking about this glitch some more, and I think my recording is actually messed up since I can't get my decoder to work with these files. I will likely be making another recording sometime this week, and will try that one again.