Greg Lindahl

Migrate CI

3

travis-ci has basically ended free builds for open source projects. Migrate our CI to something that's still free: [GitHub actions](https://github.com/features/actions) [Azure PIpelines](https://docs.microsoft.com/en-us/azure/devops/organizations/billing/buy-more-build-vs?view=azure-devops) [appveyor free tier](https://www.appveyor.com/pricing/) -- we have a working...

NOT READY: warcio test

3

An opinionated WARC standards-conformance tool. Ready for review - I have yet to work on test coverage. ``` $ warcio test test/data/*.warc.gz test/data/*.warc test/data/example-bad-non-chunked.warc.gz saw exception ERROR: non-chunked gzip file...

error checking around record creation?

4

Given [this whitespace-related header bug](https://github.com/commoncrawl/nutch/issues/5) that crept into the August 2018 Common Crawl crawl , it would be nice if it was somewhat difficult to create broken WARC files using...

Brought up in issue https://github.com/webrecorder/warcio/issues/74, it appears that most tools and the WARC standard disagree about how to compute digests when there is a transfer-encoding (i.e. chunked). "warcio check" should...

Warc tester

1

I built a thing that tests a warc for standards conformance. The cli is similar to "warcio check". It's 440 lines of code so far, likely to be around 1,000...

method '*' treated differently from others

9

I'm trying to use aiohttp-cors with aiohttp-graphl, which registers a route with the '*' method. I get `ValueError: already has a '*' handler for all methods` Here is a small...

The docs link source code in a few places, but incorrectly. A correct source url is: `https://github.com/webrecorder/warcio/blob/master/test/test_writer.py`

Well, this is the trivial part of updating to py3 :-)

Section 8 in conflict with section 6 for warc-fields

1

In addition to these MIME types not being registered (#33), there is an inconsistency about whether the WARC-Type of warcinfo and metadata SHALL BE, MAY, or is recommended to be...

'WARC-Identified-Payload-Type' allowed for request, revisit, continuation?

2

In the 1.1 spec, section 5.19, 'WARC-Identified-Payload-Type' is allowed for anything with a well-defined payload. That makes sense for response, resource, and conversion. That doesn't make sense for ~~request~~, ~~revisit~~,...