db1000n
db1000n copied to clipboard
arriven
Q: How to use this tool more effectively on my laptop?
Expected Behavior
I have 64G RAM + bought a good VPN with no limitation on connections and I want to use your tool the most effectively I can on my machine. Can you recommend the way how to use it?
Actual Behavior
Right now I just run this through the docker command. Also tried the solution with a Linux image that runs 10 threads of your tools + ovpn, but that solution hasn't been updated for some time
Specifications
- Platform: Mac OS M1
I think the most effective way is to run a native binary on your MacOS, because running it inside an amd64 docker container includes x86_64 virtualization on arm + linux VM on your host machine, that for sure impacts battery life and code execution performance.
The simplest way is to download arm64 binary for Darwin
or just run
curl "https://raw.githubusercontent.com/Arriven/db1000n/main/install.sh" | bash
and then
./db1000n
And don't forget about VPN or Proxy
You can use --scale parameter to spawn more threads (i.e. ./db1000n --scale 10) but make sure your ulimit -n is configured to be high enough for that
@Arriven What's the default for threads? Do we need to use it (--scale parameter) for any multi-core instance? Or is there a sane default? Maybe it should set it to the number of cores by default?
I don't think scaling it based on the CPU capacity is a good idea since you are more likely to hit a limit on network capacity (or the amount of sockets you can open). TCP traffic generation doesn't require a lot of CPU unless the target is very quick to respond
I guess adding information about increasing descriptors / sockets maximum number for different platforms would be very useful - on my Mac it was something like 256 by default, when the allowed maximum is 10240.
@Arriven as for CPU - I noticed the app loads CPU a lot when used with IPv6, like 100%. But when it's IPv4 only - it generates a low CPU load like maybe 12%. Mind I used different VPNs for that. And the one which has only IPv6 (it seems) makes the app load the CPU to 100% and then there are multiple issues with updating the targets list - like EOFs, timeouts, etc.
Maybe it's a separate issue with IPv6?
could be, I don't think I've tested ipv6 functionality well enough, can you tell which specific resources you tried with ipv6?
could be, I don't think I've tested ipv6 functionality well enough, can you tell which specific resources you tried with ipv6?
Hotspot Shield VPN seems to have only IPv6 for their servers. And I always get very high CPU load with that one.
Also I noticed that targets list update failures happen even with IPv4 sometimes, though less often - I'll post examples below.
2022/04/07 11:35:19.784759 config.go:69: Failed to fetch config from "https://raw.githubusercontent.com/db1000n-coordinators/LoadTestConfig/main/config.v0.7.json": Get "https://raw.githubusercontent.com/db1000n-coordinators/LoadTestConfig/main/config.v0.7.json": dial tcp: lookup [raw.githubusercontent.com](http://raw.githubusercontent.com/): no such host
2022/04/07 11:33:39.101158 config.go:69: Failed to fetch config from "https://raw.githubusercontent.com/db1000n-coordinators/LoadTestConfig/main/config.v0.7.json": Get "https://raw.githubusercontent.com/db1000n-coordinators/LoadTestConfig/main/config.v0.7.json": context deadline exceeded
and the same with just EOF. I think I also saw something like connection timeout.
hmm, I have one idea of a potential cause, does it manage to get the proper config at least once in the configuration where you get that 100% CPU load? There is a config embedded into the app as a backup and it could become outdated, I'm not 100% sure whether I've updated it after latest breaking changes
yep, confirm, when I used Hotspot VPN and I ran the app, it was able to get an initial list and I think it updated a few times ok, but then started having those issues with updating the list. But the traffic report keeps growing anyway.
If it was able to fetch the config at least once it won't get back to embedded config and that means that the problem is somewhere else, could you maybe send me some debug logs (--debug)?
Hotspotshield is a transparent proxy for TCP (L4). Hotspotshield has a limited connection tracking table for concurrent TCP streams that are terminated and proxied.. So probably it can be an issue.
In my case, no VPN used. I start the docker container with clean command from the docs and it stops after 3-10 seconds. The output freezes at this stage:
2022/04/07 09:06:18.878204 http.go:147: Attacking https://***hidden sample***
2022/04/07 09:06:18.878822 http.go:147: Attacking https://***hidden sample***
2022/04/07 09:06:18.879576 http.go:147: Attacking https://***hidden sample***
2022/04/07 09:06:18.880271 http.go:147: Attacking https://***hidden sample***
2022/04/07 09:06:18.881085 http.go:147: Attacking https://***hidden sample***
If I run the container with --restart unless-stopped argument, then docker stats shows, that it consumes 90%++ then is killed, started again, again 90%++, again killed, etc
@Arriven it's a standard output w/o debug. Is there a simple way to pass debug argument inside the docker container?
Yup, just add --debug to the end of the commandline
Easy enough, thanks The output: https://pastebin.com/3zYdaSDw
The container stops at this point
hmm, I don't see anything suspicious in the log. The only cases where something like this has happened before involved the process being killed due to OOM. I'm not sure that it's the case here but just to eliminate that as an option, how much memory is available to the container in your configuration?
Seems to be memory issue, indeed. The container was 1GB when the issue appeared. After I resized it to 2GB, it worked. But according to stats, the regular memory consumption is always lower than 700MB. It seems, some peaks of consumption above 1GB happen irregularly, maybe on start.
I might need to debug that additionally because I've only been seeing spikes to ~700MB with general usage being under 200MB (there was a bug that could lead to a spike of 2+ GB but it's been fixed long ago). The only "valid" reason I can see right now would be you using custom encryption keys but I don't think that's the case and even then it still shouldn't spike that much unless go garbage collector fails to free up the memory in time (I could see couple memory-heavy decryption operations happening one after the other in quick succession but still)
So, generally 1GB is more than enough, right? That's good, as running 2GB can be costly for multiple instances. I don't use any custom configuration, just the plain command from the docs. If you get new ideas about the issue, I'm eager to test them on my side.
If it was able to fetch the config at least once it won't get back to embedded config and that means that the problem is somewhere else, could you maybe send me some debug logs (--debug)?
The highest CPU usage was by db100n itself (around 60-70%), by Antimalware Service Executable (Windows defender, around 25-30%), but then I turned the defender off right while still running the app, and by Hotspot VPN itself (~25%).
I ran for several minutes, but got ~450mb of logs, so I had to cut it by half and archive. If it doesn't suit, you prefer me to use pastebin or if I need to re-run with Defender off - let me know.
@roman-kruglov there was an issue in the config that could lead to some increased CPU usage but it shouldn't be that significant. Considering the amount of cpu being used by hotspot shield I wonder whether it could be just the normal usage putting a lot of strain on your system? You could try using the commandline from #499 to artificially reduce the amount of traffic generated and if my assumption is correct it should also reduce the load from hotspot shield
@roman-kruglov there was an issue in the config that could lead to some increased CPU usage but it shouldn't be that significant. Considering the amount of cpu being used by hotspot shield I wonder whether it could be just the normal usage putting a lot of strain on your system? You could try using the commandline from #499 to artificially reduce the amount of traffic generated and if my assumption is correct it should also reduce the load from hotspot shield
Trying with the latest version, 21. Now it has spikes down sometimes for a short time. I confirm that if I increase the interval, the CPU load goes down - rather the spikes become less frequent.
What concerns me is that the amount of traffic with this VPN is not always so much significantly higher (sometimes it is much higher than with VPN Unlimited e.g.). But in general - if the traffic is much higher with Hotspot - should I consider it more effective? Taking into account the high CPU load.
P.S. maybe you should announce releases sometimes in the tg channgel - the last one there is so outdated and I think the app improved a lot, right?