charts
charts copied to clipboard
Haste-server rate limit causees http 429 error
App Name
haste-server
SCALE Version
22.02.3
App Version
latest 9.0.20
Application Events
I corrected that myself and no longer have events with errors
Application Logs
this is not revelant here
Application Configuration
default with some pv configured
Describe the bug
I found out that after installation app is restarted each 40-50 minutes, this increases pod restart counter. The reason for that was easy to find in logs - configured liveness probe is executed every 10s and that triggers built in rate limiting mechanism so after that time it start to return http 429 code (rate limiting) and pod restarts. Haste server has configuration to configure rate limiting (as well as white/black lists), but easiest solution for that problem is to change probe time parameters, for sure 10s changed to 100s is just ok.
To Reproduce
Deploy chart with default values and wait about 45mins. Pod will be restarted due to rate limit.
Expected Behavior
It should not restart each hour just because of internal probing.
Screenshots
not important here
Additional Context
not important here
I've read and agree with the following
- [X] I've checked all open and closed issues and my issue is not there.
This isn't much to work with...post everything you got, config, events, logs, etc and specifically the error / log
@StevenMcElligott it's just about launching helm with default chart values and leaving it for about 1h, but just for You I reverted my quick fix to get errors again:
$ kubectl get events -n haste
LAST SEEN TYPE REASON OBJECT MESSAGE
9m50s Warning Unhealthy pod/haste-server-f4d4964fd-gtggv Liveness probe failed: HTTP probe failed with statuscode: 429
9m50s Warning Unhealthy pod/haste-server-f4d4964fd-gtggv Readiness probe failed: HTTP probe failed with statuscode: 429
haste-server-config:
kah@haste-server-f4d4964fd-gtggv:/app$ cat config.js
{"host":"0.0.0.0","port":"7777","keyLength":"10","maxLength":"400000","staticMaxAge":"86400","recompressStaticAssets":"true","logging":[{"level":"verbose","type":"Console","colorize":"true"}],"keyGenerator":{"type":"phonetic"},"rateLimits":{"whitelist":[],"blacklist":[],"categories":{"normal":{},"whitelist":null,"blacklist":null}},"storage":{"type":"file","host":"127.0.0.1","port":"11211","expire":"2592000","bucket":"","region":"","connectionUrl":"postgres://undefined:@127.0.0.1:11211/2","db":"2","password":"","path":"/config"},"documents":{"about":"./about.md"}}
I did not find any helm values to alter rate limiting as well as health probes. All I know for now is that with healthcheck changed to 100s (which is ok for me) pod is not restarted every 42 min. default haste config has empty rate limiting configuration here so I guess that defaults is something about 2500 hits per hour. haste rate limiting is module is described here: https://github.com/dharmafly/connect-ratelimit
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.
This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.