paperd icon indicating copy to clipboard operation
paperd copied to clipboard

Saturated CPU after some daemon commands

Open TuxCoding opened this issue 5 years ago • 3 comments

Hey, I don't know if this is the right channel to report this or if I should post it somewhere else (like in Discord). However it seems to be related to JNI implementation according to thread dump below:

"paperd-Message-Response-Handler-7" #94 prio=5 os_prio=0 cpu=35830,13ms elapsed=42,75s tid=0x00007ff88c00a800 nid=0x1b1d runnable  [0x00007ff7e5c13000]
   java.lang.Thread.State: RUNNABLE
	at com.destroystokyo.paper.daemon.PaperDaemonJni.receiveMessage(Native Method)
	at com.destroystokyo.paper.daemon.PaperDaemon$MessageHandler.run(PaperDaemon.java:448)
	at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:515)
	at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:630)
	at java.lang.Thread.run([email protected]/Thread.java:832)

I noticed that the paperd daemon inside Paper fully satures the CPU after invoking some commands from the paperd CLI. It spawns multiple threads and each of them get stuck at the line above.

Setup

Maybe it's just my setup. So here is what I tried. I compiled this project natively (without Docker) from the root project directory and Paper using the daemon feature branch. Copied paperclip out of it (soft symlinks would link to the original source) and started the server using the absolute path of paperd (~/[...]/paperd start).

Testing

The issue occurs only after interacting with the daemon. I could reproduce it with the console command as well as the status command. Invoking the latter multiple times causes 100% CPU usage.

BTW: Nice project idea, especially the console feature is very useful. It extends the feature set from existing tools like tmux.

TuxCoding avatar Aug 18 '20 12:08 TuxCoding

This is interesting, I'll dig into this and see if I can spot what's going on. It uses a bounded thread pool, shouldn't be saturating the CPU. I may have set up something wrong.

DenWav avatar Aug 23 '20 04:08 DenWav

I set the socket timeout to 250 ms so it should loop on receive every 250 ms...definitely shouldn't saturate the CPU unless the impl for the recv syscall is saturating (unlikely). Which makes me think that I'm either doing something wrong setting the timeout or it's not working like I thought it was. Thanks for catching this.

DenWav avatar Aug 23 '20 04:08 DenWav

Okay it seems when paperd is running it's fine, it's when it quits that seems to get the thread stuck for some reason. Makes me think that the issue is paperd is no longer sending sockets (the socket is closed) and the code I have to detect when a socket has closed and detach isn't working. So since calling recv on a closed socket returns immediately the timeout isn't in place and is causing the receive thread to busy-wait until the server stops.

DenWav avatar Aug 23 '20 04:08 DenWav