Significant performance / availability issues

[Kovah]

I am trying to use the Web API tool to access Uptime Kuma for automation of website and host checks. During a first sync, the scripts ran into dozens of issues because the API failed to respond. I run into timeouts which are currently set to 60 seconds. From about 100-120 sites added via the API, 50 failed. 😕

The load is not really heavy, 1-2 requests per second. Already tried to find out what's causing the timeouts, but can't figure it out.

Here's a log example.

packet queue is empty, aborting
packet queue is empty, aborting
packet queue is empty, aborting
packet queue is empty, aborting
CRITICAL: 
INFO:     192.168.192.1:53938 - "POST /monitors/ HTTP/1.1" 500 Internal Server Error
INFO:     192.168.192.1:53948 - "POST /monitors/ HTTP/1.1" 200 OK
packet queue is empty, aborting
INFO:     192.168.192.1:45754 - "POST /monitors/ HTTP/1.1" 200 OK
INFO:     192.168.192.1:60040 - "GET /monitors/ HTTP/1.1" 200 OK
packet queue is empty, aborting
INFO:     192.168.192.1:43932 - "POST /monitors/ HTTP/1.1" 200 OK

Example error from the scripts

cURL error 28: Operation timed out after 60000 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://my-private-domain/monitors/

Any idea what the issue could be? Are there any logs about those internal server errors as seen above?

[MedAziz11]

can i see the script if u dont mind ? ill try to test it and check the problems

[Kovah]

Well, not really a script, but one of my application queueing jobs which send HTTP requests like that:

curl --request POST \
  --url https://my-private-domain.com/monitors/ \
  --header 'Authorization: Bearer eyJ0eXA...ykLk' \
  --header 'Content-Type: application/json' \
  --header 'accept: application/json' \
  --data '{
  "name": "Site Name",
  "url": "https://duckduckgo.com",
  "type": "http",
  "method": "GET",
  "active": true,
  "maxretries": 1,
  "maxredirects": 0,
  "resendInterval": 0,
  "expiryNotification": true,
  "accepted_statuscodes": [
    "200-299"
  ]
}'

which returns something like this:

{
	"msg": "Added Successfully.",
	"monitorID": 107
}

I tested this a bit further and noticed that all successful requests take at least 2.5-3 seconds to process, and I am running this query in my local network without firewalls or the internet in between. The API and Kuma are running on a machine showing no particular spikes in CPU or RAM usage, also the NVME drive should not lead to such long times. So I am not sure what the issue could be.

[MedAziz11]

imma release a new branch later about the performance(it might be the solution idk yet) , can u try it ? and thnx alot . and btw use -L with curl for the redirection

[OneYellowLemon]

Definitely a memory leak somewhere. I'm running it in Docker.

Probably related to lucasheld/uptime-kuma-api#19.

---

Memory usage goes up for every request made it seems (about 2MB for every request to /monitors it seems) Stats after 16 hours of runtime: (side note: ~3.5GB of bandwidth in 16 hours seems to be a major issue as well)

It is sometimes taking upwards of 4-5 seconds after 16 hours of running to get results about a single monitor (even simple 404s back when you try to access a monitor that doesn't exist)

One of my requests actually took over 25s and was using over an entire CPU core during that time (it dropped back down to under 1% after finishing the request)

After restarting the container, requests take under 100ms again but gradually take longer and longer over time.

[MedAziz11]

tbh im still trying to figure out the best way to handle the socket connections (i think the leak is happening becuz of the socket)

[alienbuild]

Unfortunately running into the same issues. Any update on this ? When using the kuma out of the box package it is fine, but when requesting monitors via the api cpu/mem usage goes off the rails.

Before api request

Submitting api request

I'm only running 5 monitors

try {
    const monitors = await fetch(`http://*****.com:****/monitors`, {
        headers: {
            Authorization: `Bearer ${process.env.KUMA_TOKEN}`
        }
    })
    const result = await monitors.json()
 
    return result.monitors

} catch (e) {
    throw new GraphQLError('Unable to get monitors.')
}

It does look like there was a potential memory leak in the python wrapper that has since been patched? https://github.com/lucasheld/uptime-kuma-api/issues/19#issuecomment-1501072303

p.s also just to clarify, as above, restarting the docker container frees up the mem and runs smooth again.

[OneYellowLemon]

lucasheld/uptime-kuma-api#19 has been fixed and merged

[tigunia-martin]

Just chiming in here to verify we are running into the same issue and likely the fix is to get the newer python API version incorporated into this project.

Looks like the version here is 0.13.0 (pre May 2023) and we need at least version 1.0.1

[OneYellowLemon]

@MedAziz11 Is this something that is still on your todo?