komf icon indicating copy to clipboard operation
komf copied to clipboard
verified publisher icon Snd-R

Add configurable ratelimit for providers

Open GyD opened this issue 11 months ago • 2 comments

Hello,

First thanks a lot for the software, I especially like the fact that I can save comic info.

I just have a small issue with HenTag provider, it sometime fail when I add multiples manga because of rate limit: Rate limited. (1 request per 5 seconds due to high server load)

Could it be possible, in a future update, to exposes those settings in the config, this way you don't have to update the app when services changes the limits.

Thanks a lot

GyD avatar Jan 18 '25 16:01 GyD

Same issue on my end

Image

xra1m1 avatar Feb 17 '25 21:02 xra1m1

@Snd-R 78bf8d81b0a409acfa3ef1f6ef892d1bc3b0659e didn't solve this issue, at least for me.

Below is a snip of the relevant section from my docker logs:

komf   | 04:04:18.094 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:04:18.395 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (300ms, unknown-length body)
komf   | 04:04:21.038 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:04:21.115 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (76ms, unknown-length body)
komf   | 04:04:25.241 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:04:25.318 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (77ms, unknown-length body)
komf   | 04:04:33.617 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:04:33.701 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (83ms, unknown-length body)
komf   | 04:04:33.702 [DefaultDispatcher-worker-8] ERROR snd.komf.app.api.MetadataRoutes -- catching
komf   | io.ktor.client.plugins.ClientRequestException: Client request(POST https://hentag.com/api/v1/search/vault/title) invalid: 429 Too Many Requests. Text: "Rate limited. (1 request per 5 seconds due to high server load)"
komf   |        at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:54)
komf   |        at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
komf   |        at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
komf   |        at io.ktor.client.plugins.HttpCallValidatorKt.HttpCallValidator$lambda$2$validateResponse(HttpCallValidator.kt:110)
komf   |        at io.ktor.client.plugins.HttpCallValidatorKt.access$HttpCallValidator$lambda$2$validateResponse(HttpCallValidator.kt:1)
komf   |        at io.ktor.client.plugins.HttpCallValidatorKt$HttpCallValidator$2$2.invokeSuspend(HttpCallValidator.kt:129)
komf   |        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
komf   |        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
komf   | 04:05:32.177 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:05:32.261 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (84ms, unknown-length body)
komf   | 04:05:34.588 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:05:34.669 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (80ms, unknown-length body)
komf   | 04:05:37.177 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:05:37.259 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (82ms, unknown-length body)
komf   | 04:05:39.321 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:05:39.401 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (80ms, unknown-length body)
komf   | 04:05:39.567 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:05:39.649 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (81ms, unknown-length body)
komf   | 04:05:43.727 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:05:43.809 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (81ms, unknown-length body)
komf   | 04:05:48.535 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:05:48.615 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (80ms, unknown-length body)
komf   | 04:05:48.616 [DefaultDispatcher-worker-8] ERROR snd.komf.app.api.MetadataRoutes -- catching
komf   | io.ktor.client.plugins.ClientRequestException: Client request(POST https://hentag.com/api/v1/search/vault/title) invalid: 429 Too Many Requests. Text: "Rate limited. (1 request per 5 seconds due to high server load)"
komf   |        at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:54)
komf   |        at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
komf   |        at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
komf   |        at io.ktor.client.plugins.HttpCallValidatorKt.HttpCallValidator$lambda$2$validateResponse(HttpCallValidator.kt:110)
komf   |        at io.ktor.client.plugins.HttpCallValidatorKt.access$HttpCallValidator$lambda$2$validateResponse(HttpCallValidator.kt:1)
komf   |        at io.ktor.client.plugins.HttpCallValidatorKt$HttpCallValidator$2$2.invokeSuspend(HttpCallValidator.kt:129)
komf   |        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
komf   |        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)
komf   | 04:05:52.654 [OkHttp https://hentag.com/...] INFO http.logging -- --> POST https://hentag.com/api/v1/search/vault/title (119-byte body)
komf   | 04:05:52.746 [OkHttp https://hentag.com/...] INFO http.logging -- <-- 429 Too Many Requests https://hentag.com/api/v1/search/vault/title (91ms, unknown-length body)
komf   | 04:05:52.746 [DefaultDispatcher-worker-1] ERROR snd.komf.app.api.MetadataRoutes -- catching
komf   | io.ktor.client.plugins.ClientRequestException: Client request(POST https://hentag.com/api/v1/search/vault/title) invalid: 429 Too Many Requests. Text: "Rate limited. (1 request per 5 seconds due to high server load)"
komf   |        at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invokeSuspend(DefaultResponseValidation.kt:54)
komf   |        at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
komf   |        at io.ktor.client.plugins.DefaultResponseValidationKt$addDefaultResponseValidation$1$1.invoke(DefaultResponseValidation.kt)
komf   |        at io.ktor.client.plugins.HttpCallValidatorKt.HttpCallValidator$lambda$2$validateResponse(HttpCallValidator.kt:110)
komf   |        at io.ktor.client.plugins.HttpCallValidatorKt.access$HttpCallValidator$lambda$2$validateResponse(HttpCallValidator.kt:1)
komf   |        at io.ktor.client.plugins.HttpCallValidatorKt$HttpCallValidator$2$2.invokeSuspend(HttpCallValidator.kt:129)
komf   |        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
komf   |        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)
komf   |        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

Judging from the timestamps, the rate limiter isn't working as it is supposed to.

I just built this docker stack today, so this should be the most recent docker version as of writing (version 1.6.0)

Honestly, it would be great to expose a setting to the user to change the rate limit per source.

In any case, it is as of yet not working, and it's causing issues with mass updating metadata for me.

templeofshadow avatar Sep 19 '25 04:09 templeofshadow