warchaeology
warchaeology copied to clipboard
nlnwa
Command line tool for digging into WARC files
The [gowarc.WarcFileReader](https://pkg.go.dev/github.com/nlnwa/gowarc#WarcFileReader) does not provide a simple way to get the size of the next record. This PR introduces an iterator abstraction for iterating over WARC records. In addition to...
I was wondering if it would be possible to allow cat with wildcard: when there is a folder with some warcs inside a command line usuage like: ` warc cat...
Tried out the latest release on Mac, and am getting this segfault with Browsertrix Crawler WARCs: ``` panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation...
Using viper outside command initialization and configuration makes it difficult to see what options gets applied. This commit refactors the initialization of the Filter struct in the filter package to...
Similar to https://github.com/nlnwa/warchaeology/issues/89, we should also run our tests on MacOS
Based on real world testing on data from the early 2000's we need to replace the following code https://github.com/nlnwa/warchaeology/blob/ab32b5f0340bc6b44ded32960c8fe1dbbcefe29c/nedlibreader/nedlibreader.go#L58-L66 with a function that can parse as many date formats as...
We currently run no tests on `Windows`, which we should in theory support. It should be relatively straightforward to enable testing for `Windows`, unless the tests fails.
Commit https://github.com/nlnwa/warchaeology/commit/a72acd4d469cb612f24e37c1b75af6553fb7ee7e possibly introduced a close before use issue (maybe nil pointer exception).
Not all of the sub-linters of `golang-ci` should be used, and this issue needs to determine what errors should be fixed, and which ones should be left disabled.
https://github.com/nlnwa/warchaeology/blob/606d0dcc55be3134b6c819c1ba87907a37a71264/main.go#L33 This setting should be localized to the cases where badger is used as it is badger who recommends the value to be 128 or higher (see https://dgraph.io/docs/badger/get-started/ and https://dgraph.io/docs/badger/faq/#are-there-any-go-specific-settings-that-i-should-use).