Output filtered data to WARC Format

[greebie]

Users may desire outputs in WARC format after filtering their RDD[ArchiveRecord].

[greebie]

@dportabella has an example to implement this here: https://gist.github.com/dportabella/3caf261c218a4448a03a14dbc06fe730 .

The other alternative is the more detailed WARCWriter class from iipc: https://github.com/iipc/webarchive-commons/blob/master/src/main/java/org/archive/io/warc/WARCWriter.java which has me confused, honestly.

This feature has potential to be dangerous, as there is no real way to test the total size of the request. Take for example this pseudocode:

var record = RecordLoader('filePath', sc) .map(x => SaveToWARC(record))

which would save the entire Warc for every ArchiveRecord in record. It would be a juggernaut that will not stop until the server explodes due to lack of fileSpace.

I have to admit to being a little lost to the finer details of producing and saving a WARC files here, and it's Monday, so am prone to laziness. Advice @ruebot, @lintool and @ianmilligan1 ?

[ianmilligan1]

The example looks promising, @greebie! I'm not too hung up on the danger, as long as the feature is well documented. But maybe I'm naive. The others may think differently, but my gut is that taking a stab at using @dportabella's example and seeing if it can play with AUT is probably the most fruitful way forward?

We can also discuss tomorrow.

[dportabella]

it would be nice to also create the cdx index at the same time.

[greebie]

Producing the cdx would be a safe start for testing purposes, actually. Thanks dportabella!

[greebie]

Backing away from this issue for now until we find someone with better understanding of the iipc toolkit.

[ianmilligan1]

I think our conversations have largely moved away from the idea of creating new WARC files, and really focusing on derivative datasets. I think given this move in the project, we could consider closing this?

[dportabella]

I still think that filtering WARC files is an important task that AUT can solve.

[ianmilligan1]

Thanks @dportabella! My sense is that our team's time is limited to make this a short or medium-term issue for us, but any chance you'd be interested in opening up a PR based on the example code that @greebie shared up above?

[dportabella]

I shared a gist on achieving this task (included in @greebie comment above), and I am currently using this approach. I don't know much the details of the AUT library, and I don't have time to get into it, sorry :(

[sepastian]

Adding writing of WARC records to the current AUT is one way of solving this.

But I would rather go with @ruebot's suggestion of hooking into existing infrastructure and creating a Spark Data Source, see #371.