charts icon indicating copy to clipboard operation
charts copied to clipboard

Haste-server rate limit causees http 429 error

Open dominch opened this issue 2 years ago • 3 comments

App Name

haste-server

SCALE Version

22.02.3

App Version

latest 9.0.20

Application Events

I corrected that myself and no longer have events with errors

Application Logs

this is not revelant here

Application Configuration

default with some pv configured

Describe the bug

I found out that after installation app is restarted each 40-50 minutes, this increases pod restart counter. The reason for that was easy to find in logs - configured liveness probe is executed every 10s and that triggers built in rate limiting mechanism so after that time it start to return http 429 code (rate limiting) and pod restarts. Haste server has configuration to configure rate limiting (as well as white/black lists), but easiest solution for that problem is to change probe time parameters, for sure 10s changed to 100s is just ok.

To Reproduce

Deploy chart with default values and wait about 45mins. Pod will be restarted due to rate limit.

Expected Behavior

It should not restart each hour just because of internal probing.

Screenshots

not important here

Additional Context

not important here

I've read and agree with the following

  • [X] I've checked all open and closed issues and my issue is not there.

dominch avatar Jul 04 '23 10:07 dominch

This isn't much to work with...post everything you got, config, events, logs, etc and specifically the error / log

StevenMcElligott avatar Jul 04 '23 13:07 StevenMcElligott

@StevenMcElligott it's just about launching helm with default chart values and leaving it for about 1h, but just for You I reverted my quick fix to get errors again:

$ kubectl get events -n haste
LAST SEEN   TYPE      REASON      OBJECT                             MESSAGE
9m50s       Warning   Unhealthy   pod/haste-server-f4d4964fd-gtggv   Liveness probe failed: HTTP probe failed with statuscode: 429
9m50s       Warning   Unhealthy   pod/haste-server-f4d4964fd-gtggv   Readiness probe failed: HTTP probe failed with statuscode: 429

haste-server-config:

kah@haste-server-f4d4964fd-gtggv:/app$ cat config.js 
{"host":"0.0.0.0","port":"7777","keyLength":"10","maxLength":"400000","staticMaxAge":"86400","recompressStaticAssets":"true","logging":[{"level":"verbose","type":"Console","colorize":"true"}],"keyGenerator":{"type":"phonetic"},"rateLimits":{"whitelist":[],"blacklist":[],"categories":{"normal":{},"whitelist":null,"blacklist":null}},"storage":{"type":"file","host":"127.0.0.1","port":"11211","expire":"2592000","bucket":"","region":"","connectionUrl":"postgres://undefined:@127.0.0.1:11211/2","db":"2","password":"","path":"/config"},"documents":{"about":"./about.md"}}

I did not find any helm values to alter rate limiting as well as health probes. All I know for now is that with healthcheck changed to 100s (which is ok for me) pod is not restarted every 42 min. default haste config has empty rate limiting configuration here so I guess that defaults is something about 2500 hits per hour. haste rate limiting is module is described here: https://github.com/dharmafly/connect-ratelimit

dominch avatar Jul 05 '23 09:07 dominch

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.

truecharts-admin avatar Jun 25 '24 20:06 truecharts-admin

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

truecharts-admin avatar Jul 10 '24 20:07 truecharts-admin