[ENHANCEMENT]: Serve wordlists from an external URL

[injust]

Description

I would like to serve large wordlists from an S3 bucket instead of directly from the server.

Context: I'd like to switch my wordlists from 7z to gzip, since hashcat supports on-the-fly gzip decompression. However, in my experience, gzip has a much lower compression ratio, so I expect the compressed wordlists to become much larger.

[s3inlc]

Typically, this difference would not be that high, maybe some percents but not like multiple times larger.

[injust]

I'll benchmark and come back with some numbers.

[injust]

I tested with Top1pt8Billion-WPA-probable-v2.txt from https://github.com/berzerk0/Probable-Wordlists/tree/master/Real-Passwords/WPA-Length.

Lines: 1,802,908,173 Decompressed: 20.6 GB Original 7z file: 3.5 GB gzip 1.14 with --best: 6.5 GB

[injust]

An example with a smaller wordlist, rockyou.txt:

Lines: 14,344,391 Decompressed: 140 MB 7-Zip 23.01 with -mx9: 39 MB gzip 1.14 with --best: 53 MB

[injust]

I think there would still be benefits to serving wordlists from an external URL, e.g. S3 or a similar service.