Extraction fails with "struct.error: unpack requires a bytes object of length 80"

[VasoVV]

At first, thank you very much for the great script!

But some directories are failed to be processed.

Here is the debug output and the index file.

./hikstract
2018-07-19 11:58:42,743 [15476] DEBUG: Finished loading db: {'cur_datadir': 'datadir0',
 'datadirs': {'datadir0': {'cur_section': 1,
                           'last_vrec': 0,
                           'revision': 15787},
              'datadir1': {'cur_section': 131,
                           'last_vrec': 1,
                           'revision': 16269},
              'datadir2': {'cur_section': 0,
                           'last_vrec': -1,
                           'revision': 15683}}}
2018-07-19 11:58:42,743 [15476] INFO: Entering datadir0
2018-07-19 11:58:42,743 [15476] DEBUG: Making Header at /mnt/transport/vol03/datadir0/index00.bin:0
2018-07-19 11:58:42,744 [15476] DEBUG: cur_sec_idx: 2, rec_sec_num: 186, revision: 14228, total_sec: 2
2018-07-19 11:58:42,744 [15476] INFO: Index revision is 14228
2018-07-19 11:58:42,744 [15476] DEBUG: Making Section at /mnt/transport/vol03/datadir0/index00.bin:500
2018-07-19 11:58:42,744 [15476] DEBUG: end_dt: 2018-02-14 16:14:15, idx: 0, last_vrec_idx: 0, start_dt: 2018-02-14 15:52:02
2018-07-19 11:58:42,744 [15476] DEBUG: Making Section at /mnt/transport/vol03/datadir0/index00.bin:520
2018-07-19 11:58:42,745 [15476] DEBUG: end_dt: 2018-02-22 15:47:50, idx: 1, last_vrec_idx: 0, start_dt: 2018-02-22 15:39:22
2018-07-19 11:58:42,745 [15476] DEBUG: Making CurrentSection at /mnt/transport/vol03/datadir0/index00.bin:30
2018-07-19 11:58:42,745 [15476] DEBUG: end_dt: 1970-01-01 00:00:00, idx: 255, last_vrec_idx: 0, start_dt: 1970-01-01 00:00:00
2018-07-19 11:58:42,745 [15476] DEBUG: Entering section 1
2018-07-19 11:58:42,746 [15476] DEBUG: Making VideoRecord at /mnt/transport/vol03/datadir0/index00.bin:6c40
2018-07-19 11:58:42,746 [15476] DEBUG: duration: 508, end_dt: 2018-02-22 15:47:50, length: 260466216, start_dt: 2018-02-22 15:39:22, start_offset: 0
2018-07-19 11:58:42,746 [15476] DEBUG: Entering section 255
2018-07-19 11:58:42,746 [15476] DEBUG: Making VideoRecord at /mnt/transport/vol03/datadir0/index00.bin:4fcc40
Traceback (most recent call last):
  File "./hikstract", line 50, in <module>
    run()
  File "./hikstract", line 42, in run
    parser.update()
  File "/home/user/git/hikstract/parser.py", line 32, in update
    self.update_datadir(datadir)
  File "/home/user/git/hikstract/parser.py", line 61, in update_datadir
    next_vrecs = u.islice_from(sec.video_records, next_vrec_idx)
  File "/home/user/git/hikstract/items.py", line 104, in video_records
    vrec = VideoRecord.make(self._h_idx_file, idx, start)
  File "/home/user/git/hikstract/items.py", line 58, in make
    fields = struct.unpack(cls.fmt, buf)
struct.error: unpack requires a bytes object of length 80

-- in this case only one video file is created. index00.bin.gz

[thehiddenbit]

Same problem here. Maybe the video record is corrupt, therefor skipping this part would be a good solution.

@ @2018-12-12 10:57:22,780 [18550] DEBUG: Entering section 255
2018-12-12 10:57:22,781 [18550] DEBUG: Making VideoRecord at /srv/dvr/datadir0/index00.bin:4fd460
Traceback (most recent call last):
  File "./hikstract", line 50, in <module>
    run()
  File "./hikstract", line 42, in run
    parser.update()
  File "/opt/hikstract/parser.py", line 32, in update
    self.update_datadir(datadir)
  File "/opt/hikstract/parser.py", line 61, in update_datadir
    next_vrecs = u.islice_from(sec.video_records, next_vrec_idx)
  File "/opt/hikstract/items.py", line 104, in video_records
    vrec = VideoRecord.make(self._h_idx_file, idx, start)
  File "/opt/hikstract/items.py", line 58, in make
    fields = struct.unpack(cls.fmt, buf)
struct.error: unpack requires a bytes object of length 80

[iafilatov]

Wow, I have no Idea how I missed this issue, and necroposting probably won't make any difference. I'd say it looks like your cam maintains its index in a slightly different way. My DS-2CD2012-I has a section at 0x30 which describes where the latest videos are in the hiv*.mp4 files. In your case that part looks uninitialized. That's why the parser fails. Unfortunately, there isn't much I can do without knowing the index format of your cam better. You can maybe try commenting out these lines to see if it helps.

[palfrey]

I've just hit this one with a DS-2CD2365G1. Is there any useful info I can provide to help debug this?

[palfrey]

Notably, it was working fine until today for some reason...

[iafilatov]

It might help to see your hikstract log with the stack trace you're getting and the index00.bin file it's reading at that moment. You should see it in the log (there are index00.bin files in datadir0, datadir1 etc.) The index files have only metadata and none of the actual videos or pictures. Also whether the clock on your cam is set correctly and if you remember roughly when hikstract last worked fine and when it started crashing.

[palfrey]

I've got it running on an hourly cronjob, but (un)fortunately it's only failed once like this (06:00 on the 14th), and I'm not currently able to reproduce it. I'll try and remember to get that data if it does it again though.

[iafilatov]

I had had similar issues when the index file was read exactly during the time a video was being written. But the actual error came from a different place in the index file. It's just a guess but it seems like when a cam starts recording, it initializes a part of the index structure and when it's done, it fills our the rest. It's this in-between state that kills the parser. Perhaps, as @thehiddenbit said above, broken records should just be skipped and retried. Also, if it only failed once, that means hikstract must have tried again the next hour and ripped out that record successfully.