warcio

warcio copied to clipboard

One of the most likely problems we see is failed transfers leading to truncated WARC.GZ files. We can spot this with gunzip -t but it would be good if warcio check also raised this as a validation error. My tests so far have indicated that the warcio and cdxj-indexer etc. tools all skip over these errors silently.

This came up recently in IIPC Slack when trying to diagnose why warcheology was reporting a corrupted WARC file, and warcio was not. It appeared that the WARC file was truncated as a result of a browsertrix-crawler container exiting abnormally, and not closing the GZIP file properly...

In case it's helpful to have a test script (which doesn't emit a warning that I can see):

from warcio.archiveiterator import ArchiveIterator

with open('test.warc.gz', 'rb') as stream:
    for i, record in enumerate(ArchiveIterator(stream)):
        print(i, record.rec_headers.get_header('WARC-Target-URI'))
        if record.rec_type == 'response':
            content = record.content_stream().read()

And here's a test file: test.warc.gz

gunzip on the other hand does notice:

$ gunzip --test test.warc.gz
gunzip: truncated input
gunzip: test.warc.gz: uncompress failed

Wow, I'd totally forgotten about this!

Seems like there's a hook in the underlying Python library to spot this case:: https://docs.python.org/3/library/zlib.html#zlib.Decompress.eof

Decompress.eof A boolean indicating whether the end of the compressed data stream has been reached. This makes it possible to distinguish between a properly formed compressed stream, and an incomplete or truncated one. New in version 3.3.

But it's not clear to me how to weave that in here...

https://github.com/webrecorder/warcio/blob/aa702cb321621b233c6e5d2a4780151282a778be/warcio/archiveiterator.py#L108-L140

@edsu what record in test.warc.gz is the truncated one? And where can I find warcheology? Thanks.

I believe it's the last record. If you try to gunzip the file, you should see the error error right at the end?

I'm not really familiar with it but here is the warchaeology repo: https://github.com/nlnwa/warchaeology

@edsu thanks for adding a simple test and @anjackson for looking up the .eof property!

With that, I think detecting this case can be done as follows:

diff --git a/warcio/archiveiterator.py b/warcio/archiveiterator.py
index 484b7f0..451f182 100644
--- a/warcio/archiveiterator.py
+++ b/warcio/archiveiterator.py
@@ -113,7 +113,13 @@ class ArchiveIterator(six.Iterator):
 
                 yield self.record
 
-            except EOFError:
+            except EOFError as e:
+                if self.reader.decompressor:
+                    if not self.reader.decompressor.eof:
+                        sys.stderr.write("warning: final record appears to be truncated")
+
                 empty_record = True
 
             self.read_to_end()

But, what is the desired behavior be more generally?

• for warcio check, seems like it should return an error
• seems like the gunzip behavior is definitely not desirable, as that fails to unzip any record even if only last one is invalid.
• for indexing, it seems like the indexing should still succeed, and maybe print the warning? there are other recoverable errors that are also logged, such as Content-Length mismatches. Should it still return a 1

It sort of depends on how the WARC is being used:

• If the goal is to detect if WARC is valid after transfer, this is definitely an error and should be detected.
• If the goal is to index a WARC that already exists, this is more of a warning since not much be done at that point, and we definitely don't want to invalid the whole WARC just because of the last record.