Update checker can block usage

[kousu]

LevelDB Dumper version

3.0.0-beta1

Describe the bug

If you run LevelDBDumper too much (or are otherwise on a busy network) it will get rate-limited by GitHub and stop working.

To Reproduce

Run it 10000 times over, I guess? It's hitting https://api.github.com/repos/mdawsonuk/LevelDBDumper/releases which is returning

{"message":"API rate limit exceeded for 45.45.148.196. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}

Expected behavior

It would be best if it didn't need to be online to function!

Screenshots

[kousu@bomp leveldb]$ LevelDBDumper -t csv -o /tmp/ -d .


LevelDB Dumper 3.0.0-beta.1

Author: Matt Dawson

Command Line: -t csv -o /tmp/ -d .

panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
main.checkUpdatePreleaseStream({0x5606f55e9db8, 0xc})
	github.com/mdawsonuk/LevelDBDumper/src/LevelDBDumper/Updates.go:41 +0x2b2
main.checkUpdate({0x5606f55e9db8, 0xc})
	github.com/mdawsonuk/LevelDBDumper/src/LevelDBDumper/Updates.go:16 +0x49
main.dumpDBs({0xc0000ae000?, 0x7, 0x7})
	github.com/mdawsonuk/LevelDBDumper/src/LevelDBDumper/LevelDBDumper.go:46 +0x28b
main.main()
	github.com/mdawsonuk/LevelDBDumper/src/LevelDBDumper/LevelDBDumper.go:23 +0x2e

[mdawsonuk]

Morning!

Thanks for your big report - this functionality is intended to check for updates but I didn't consider running it so frequently that it hits API limits.

From my perspective there's two things which need to be done: better error handling in the case that the connection fails, and the ability via the command line to disable checking for updates. Any additional suggestions? Would you prefer to be able to permanently disable update checks?

[OnlyBeta]

In my situation because my internet connection dropped, I was unable to run it. Personally I don't think it needs to check for an update automatically every time it executes but a command line switch would be great.

[mdawsonuk]

Hi @OnlyBeta @kousu I've hopefully resolved this (and did some much needed housekeeping for the repo!) with the addition of some checks for rate limiting, and an --offline command line argument. Could you both please test the builds from the GitHub Actions run here? Just download either Ubuntu or Windows version, they're compressed zips with all of the platform builds (Win, OSX, Linux)

[OnlyBeta]

Done some quick testing and was able to successfully use it offline now. It's getting late in my timezone now, will do some more testing tomorrow. Will report if any issues found.

Thanks for your efforts!

[mdawsonuk]

Done some quick testing and was able to successfully use it offline now. It's getting late in my timezone now, will do some more testing tomorrow. Will report if any issues found.

Thanks for your efforts!

No worries, thanks for your quick reply! If you don't find any issues tomorrow then I should be okay to publish :)

[kousu]

My internet connection cleared up (it was on a hotel wifi) so I can't really test the initial case easily now, but this seems to work for me too now! Thank you!

[mdawsonuk]

My internet connection cleared up (it was on a hotel wifi) so I can't really test the initial case easily now, but this seems to work for me too now! Thank you!

No worries - I was able to trigger the rate limit so was able to accommodate for that use case too

[OnlyBeta]

I have done some additional testing and am able to successfully run it without an active internet connection but did notice the following.

Running the app with an active internet connection and using the "-offline" switch, I notice there is still an outbound network connection to GitHub, is this expected? I used Wireshark to capture network packets.

Additionally after running the app, I see this message at the end "You are using the latest version of LevelDB Dumper" which seems to imply it did a check, despite using the 'offline' switch.

[mdawsonuk]

I have done some additional testing and am able to successfully run it without an active internet connection but did notice the following.

Running the app with an active internet connection and using the "-offline" switch, I notice there is still an outbound network connection to GitHub, is this expected? I used Wireshark to capture network packets.

Additionally after running the app, I see this message at the end "You are using the latest version of LevelDB Dumper" which seems to imply it did a check, despite using the 'offline' switch.

Hmm it shouldn't have made any network connections if you have specified "--offline"... did you use one or two dashes?

[OnlyBeta]

ok, the double dash did the trick, all is good.