WARC Parsing sometimes results in truncated records.

[ikreymer]

The WARC parsing sometimes results in records being truncated.

This might be due to the parser continuing to look for newlines/read one line at a time, even when parsing the content body, and might be happening if there is a \r\n\r\n encountered in the body of the record.

The issue can be seen by running:

const { AutoWARCParser } = require('node-warc');
  
(async () => {
  for await (const record of new AutoWARCParser('test1.warc.gz')) {
    console.log(record.content.toString('utf-8'));
  }
})();

With these example files: test1.warc.gz (last couple of bytes are cut-off)

test2.warc.gz (most of the file is cut-off after initial comment)

For comparison, the warcio version prints the full record:

from warcio import ArchiveIterator
  
for record in ArchiveIterator(open('./test1.warc.gz', 'rb')):
    print(record.content_stream().read().decode('utf-8'))

[comatory]

I'm trying to parse the contents as well (especially HTML) and most of the sites return the HTML truncated.

[chichicuervo]

I'm experiencing this also. Seems like the issue is warcRecord/builder.js .. the function consumeLine() ... basically you're using a solo CRLF as the boundary between WARC Records... seems to me that you should be ignoring this until you reach the WARC Record's Content-Length

[chichicuervo]

seems that commenting out the line this._parsingState = parsingStates.consumeCRLFContent2

inside the switch statement resolves this specific problem. Is there a specific protocol reason it's there, like trailers or something, or were you just trying to avoid trailing CRLFs?

[N0taN3rd]

The aim of crlf content two was to skip the trailing CRLFs between records. For more details see the spec.

If that fixes the issue for all cases feel free to open a PR. I am working on this albeit slowly on the updates branch. Also feel free to add to the effort via a PR.

[N0taN3rd]

Also @ikreymer still waiting on that PR fixing this you said you had

[ikreymer]

Hey, here is the fix that I have in builder.js: https://github.com/ikreymer/node-warc/commit/e5bfd2fd3e14a8debba11d85b732252a1cb343f0

I could isolate that and make it into a PR, not 100% sure it fixes it, but think so.. I had a bunch of other changes on that branch though.

[ikreymer]

@chichicuervo in case this helps you, I wanted to mention I've been working on a new WARC reader library focusing on streaming WARCs in the browser: https://github.com/ikreymer/warcio.js It works comparable to the python warcio library and should not have the truncation issue.

It's not as far along as this library and doesn't yet have support for writing WARCs, though.