MisterTea
100% CPU Usage 😢
I am getting 100% CPU usage because of ET : Can someone help and guide?
I had installed et v6.2.4 on an embedded debian setup (Storage device) :
root@wdmc:~# et --version
et version 6.2.4
root@wdmc:~# uname -a
Linux wdmc 4.14.4 #14 SMP Wed Dec 20 01:09:34 +05 2017 armv7l GNU/Linux
It works well and keeps the connection etc, for the reverse tunnel that I had setup to monitor the server using another on a remote, but accessible machine. After some time - the CPU usage goes up to 100% and stays there forever. This happens within 5 minutes of the connection being triggered.
etclientLogs.tar.gz
etserverLogs.tar.gz
Line in crontab which triggers the ET connection - I have started it from terminal and also without the -x option, and that gives me the same results:
#@reboot 0 * * * * /usr/bin/et -x xxxxxx@yyyyyyyy:pppp -r 8xxx:10xxxx &
While retrieving the logs:
This commands fails as I don't have these log files:
tar -cvzPhf /tmp/etclientLogs.tar.gz /tmp/etclient_err /tmp/etclient.INFO
But I had these which I have attached:
tar -cvzPhf /tmp/etclientLogs.tar.gz /tmp/etclient-2023-07-22_14-20-02.log /tmp/etclient-stderr-2023-07-22_14-20-02.log
Similarly this failed for the server as I did not have these files:
tar -cvzPhf /tmp/etserverLogs.tar.gz /tmp/etserver_err /tmp/etserver.INFO
But I had these - attached:
sudo tar -cvzPhf /tmp/etserverLogs.tar.gz /tmp/etserver-2023-07-20_11-33-10_2532693.log /tmp/etserver-stderr-2023-07-20_11-33-10_2532693.log
The etclient log file is flooded with "Got data from stdin" - Not sure if it is a bug.
Tail of the log file without "Got data" lines
root@wdmc:~# cat /tmp/etclient-2023-07-22_14-20-02.log | grep -v "Got data" | tail
[V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:23] socket 10 has data
[V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:45] Unixsocket handler read from fd: 10
[V2 2023-07-22 14:51:10,259 client-main BackedReader.cpp:68] Reading message of length: 95
[V2 2023-07-22 14:51:10,259 client-main BackedReader.cpp:71] bytes remaining: 95
[V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:23] socket 10 has data
[V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:45] Unixsocket handler read from fd: 10
[V3 2023-07-22 14:51:10,259 client-main TerminalClient.cpp:280] Got terminal buffer
[INFO 2023-07-22 14:51:10,260 client-main TerminalClient.cpp:326] Window size changed: row: 60638 column: 25787 width: 45510 height: 3856
[V2 2023-07-22 14:51:10,260 client-main BackedWriter.cpp:50] Message length with header: 37
[V4 2023-07-22 14:51:10,260 client-main UnixSocketHandler.cpp:61] Unixsocket handler write to fd: 10
The log file has 128k lines in total:
root@wdmc:~# cat /tmp/etclient-2023-07-22_14-20-02.log | wc
128004 1159042 10840264
Of these, 118k lines were the "Got data from stdin lines" - and 9.5k other lines:
root@wdmc:~# cat /tmp/etclient-2023-07-22_14-20-02.log | grep -v "Got data" | wc
9543 92893 889540
The 'Got data line looks like this:
root@wdmc:~# tail -2 /tmp/etclient-2023-07-22_14-20-02.log
[V4 2023-07-22 14:51:10,261 client-main TerminalClient.cpp:210] Got data from stdin
[V4 2023-07-22 14:51:10,261 client-main TerminalClient.cpp:210] Got data from stdin
That log message is only at verbosity level 4 and higher. You shouldn't see it in normal operation:
https://github.com/MisterTea/EternalTerminal/blob/8ce83c9239a881bf433aa29bbd7f0cef3777ea1e/src/terminal/TerminalClient.cpp#L213
Is the server cfg file set to verbose prints?
On Sat, Jul 22, 2023, 12:14 PM rahulraj80 @.***> wrote:
I am getting 100% CPU usage because of ET : Can someone help and guide?
I had installed et v6.2.4 on an embedded debian setup (Storage device) :
@.:~# et --version et version 6.2.4 @.:~# uname -a Linux wdmc 4.14.4 #14 SMP Wed Dec 20 01:09:34 +05 2017 armv7l GNU/Linux
It works well and keeps the connection etc, for the reverse tunnel that I had setup to monitor the server using another on a remote, but accessible machine. After some time - the CPU usage goes up to 100% and stays there forever. This happens within 5 minutes of the connection being triggered.
[image: image] https://user-images.githubusercontent.com/11489601/255342255-ee9c7d9a-0516-4f4c-b3bb-5ebfab03ad63.png etclientLogs.tar.gz https://github.com/MisterTea/EternalTerminal/files/12136884/etclientLogs.tar.gz etserverLogs.tar.gz https://github.com/MisterTea/EternalTerminal/files/12136885/etserverLogs.tar.gz
Line in crontab which triggers the ET connection - I have started it from terminal and also without the -x option, and that gives me the same results: @.*** 0 * * * * /usr/bin/et -x @.***:pppp -r 8xxx:10xxxx &
While retrieving the logs: This commands fails as I don't have these log files: tar -cvzPhf /tmp/etclientLogs.tar.gz /tmp/etclient_err /tmp/etclient.INFO
But I had these which I have attached: tar -cvzPhf /tmp/etclientLogs.tar.gz /tmp/etclient-2023-07-22_14-20-02.log /tmp/etclient-stderr-2023-07-22_14-20-02.log
Similarly this failed for the server as I did not have these files: tar -cvzPhf /tmp/etserverLogs.tar.gz /tmp/etserver_err /tmp/etserver.INFO
But I had these - attached: sudo tar -cvzPhf /tmp/etserverLogs.tar.gz /tmp/etserver-2023-07-20_11-33-10_2532693.log /tmp/etserver-stderr-2023-07-20_11-33-10_2532693.log
The etclient log file is flooded with "Got data from stdin" - Not sure if it is a bug.
Tail of the log file without "Got data" lines
@.***:~# cat /tmp/etclient-2023-07-22_14-20-02.log | grep -v "Got data" | tail [V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:23] socket 10 has data [V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:45] Unixsocket handler read from fd: 10 [V2 2023-07-22 14:51:10,259 client-main BackedReader.cpp:68] Reading message of length: 95 [V2 2023-07-22 14:51:10,259 client-main BackedReader.cpp:71] bytes remaining: 95 [V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:23] socket 10 has data [V4 2023-07-22 14:51:10,259 client-main UnixSocketHandler.cpp:45] Unixsocket handler read from fd: 10 [V3 2023-07-22 14:51:10,259 client-main TerminalClient.cpp:280] Got terminal buffer [INFO 2023-07-22 14:51:10,260 client-main TerminalClient.cpp:326] Window size changed: row: 60638 column: 25787 width: 45510 height: 3856 [V2 2023-07-22 14:51:10,260 client-main BackedWriter.cpp:50] Message length with header: 37 [V4 2023-07-22 14:51:10,260 client-main UnixSocketHandler.cpp:61] Unixsocket handler write to fd: 10
The log file has 128k lines in total:
@.***:~# cat /tmp/etclient-2023-07-22_14-20-02.log | wc 128004 1159042 10840264
Of these, 118k lines were the "Got data from stdin lines" - and 9.5k other lines:
@.***:~# cat /tmp/etclient-2023-07-22_14-20-02.log | grep -v "Got data" | wc 9543 92893 889540
The 'Got data line looks like this:
@.***:~# tail -2 /tmp/etclient-2023-07-22_14-20-02.log [V4 2023-07-22 14:51:10,261 client-main TerminalClient.cpp:210] Got data from stdin [V4 2023-07-22 14:51:10,261 client-main TerminalClient.cpp:210] Got data from stdin
— Reply to this email directly, view it on GitHub https://github.com/MisterTea/EternalTerminal/issues/588, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACK5P2W6CZJTLZ3MW3OWSLXRQDBBANCNFSM6AAAAAA2T6LFBU . You are receiving this because you are subscribed to this thread.Message ID: @.***>
The conf file was not changed.
; et.cfg : Config file for Eternal Terminal
;
[Networking]
port = 2022
# bind_ip = 0.0.0.0
[Debug]
verbose = 0
silent = 0
logsize = 20971520
telemetry = true
logdirectory = /tmp
I found a ubuntu/focal based firmware instead of the jessie/stretch one. Let me see if that was the reason for my issue. Will upgrade OS and report.
Sorry for the delay: I ran into a ton of issues with my embedded system due to a wrong flash. Fixed it after a ton of reading and effort.
Update: On the new OS as well, I have the same issue. The et executable starts using up 100% of the CPU a few minutes after being activated, and then stays at 100% until killed. It also overheats the device enclosure and I can hear the fan working at a higher speed.
I see no errors in the error logs, but if I change verbosity to 1 in et.cfg, I see these two lines repeated continuously.
[INFO 2023-08-03 17:26:38,830 client-main TerminalClient.cpp:326] Window size changed: row: 28811 column: 46730 width: 14152 height: 46849
[INFO 2023-08-03 17:26:38,830 client-main TerminalClient.cpp:326] Window size changed: row: 28811 column: 46730 width: 50832 height: 46848
[INFO 2023-08-03 17:26:38,899 client-main TerminalClient.cpp:326] Window size changed: row: 28811 column: 46730 width: 14152 height: 46849
[INFO 2023-08-03 17:26:38,900 client-main TerminalClient.cpp:326] Window size changed: row: 28811 column: 46730 width: 50832 height: 46848
What can I do to fix it please?
Those prints are saying that your window size is changing. The window size is measured in CHARACTERS, so a window size of 50k by 46k is bonkers and will definitely blow up your CPU. You need to find out why the window size is so large.
Those prints are saying that your window size is changing. The window size is measured in CHARACTERS, so a window size of 50k by 46k is bonkers and will definitely blow up your CPU. You need to find out why the window size is so large.
So - this is running from a cron job, and what it does is setup a reverse tunnel to forward a port (say) A from the device to a remote server on port B. It is a non-standard port, so unless I visit the https://server:B there should be no activity.
Where does the window thing come (and its size)? Also, the size is alternating for some reason. Are there two sessions running simultaneously? I am using the -x option which I understood from the docs that it will kill the other sessions, so there should be only one active at any time.
How often does the job run? The window size is based on the size of the shell that called et. If it's done non-interactively, maybe the size is arbitrary?
It is run from cron. This is the line in my crontab :
#@reboot 0 * * * * /usr/bin/et -x xxxxxx@yyyyyyyy:pppp -r 8xxx:10xxxx &
0 * * * * /usr/bin/et -x xxxxxx@yyyyyyyy:pppp -r 8xxx:10xxxx &
I am not sure how window size is defined when a job is run by the cron daemon.