Use compression for transfer of JSON

[ties]

The file with validated objects is growing and starting. Depending on the output style, I see 43MB (current rpki-client output), 57MB (jq or new rpki-client builds, one attribute per line, no colour, indented), while compact JSON is ~37M (jq -c).

It compresses pretty well.

Because the client trusts the server (w.r.t. content) I do not think gzip bombs are an issue. So, as a user, I want the HTTP client should use HTTP compression when loading the prefixes.

[benjojo]

The golang http library already does transparent gzip decompression, so long as your webserver does it. it will be sent over the wire compressed!

I think that should be enough to allow people to have compressed pulls, rather than adding logic in place to actively decompress files

On Tue, 2 May 2023, 08:49 Ties de Kock, @.***> wrote:

The file with validated objects is growing and starting. Depending on the output style, I see 43MB (current rpki-client output), 57MB (jq or new rpki-client builds, one attribute per line, no colour, indented), while compact JSON is ~37M (jq -c).

It compresses pretty well.

Because the client trusts the server (w.r.t. content) I do not think gzip bombs are an issue. So, as a user, I want the HTTP client should use HTTP compression when loading the prefixes.

— Reply to this email directly, view it on GitHub https://github.com/bgp/stayrtr/issues/100, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALPK4WAL7CAZY5SKFSW2EDXEC37ZANCNFSM6AAAAAAXSVUP3A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[ties]

Nice feature. Did not have time to check this at the time. I've validated that the content is compressed (request to default console.rpki-client.org URL on Fastly):

Adding a metric for the underlying length is a bit hard because chunked responses have fhttp.ContentLength == -1 and the decompression is transparent. I will skip that one.