bilibili-live-monitor-ts icon indicating copy to clipboard operation
bilibili-live-monitor-ts copied to clipboard

Published 20 hours ago • verified publisher icon Billyzou0741326

使用pm2运行后不到几分钟，后台查看CPU占用率为100%

Open luoshuijs opened this issue 4 years ago • 11 comments

如题，但是普通运行下运行却没有任何问题。 在/root/.pm2/logs下有个错误日记，内容如下。 [Error: EMFILE: too many open files, open '/root/bilibili-live-monitor-ts/build/db/record.json'] { errno: -24, code: 'EMFILE', syscall: 'open', path: '/root/bilibili-live-monitor-ts/build/db/record.json' }

luoshuijs avatar Mar 10 '20 03:03 luoshuijs

通过查看 https://billyzou0741326.github.io/bilibili-live-monitor-ts/#/faq/README 确认nofile上限已经设置最大，为65535

luoshuijs avatar Mar 10 '20 03:03 luoshuijs

ss -t | awk ' $5 ~ /^[0-9]/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n 或者 netstat -t | awk ' $5 ~ /^[0-9]/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

第一列是连接数 看下有多少

watch 'ss -t | awk '\'' $5 ~ /^[0-9]/ {print $5}'\'' | cut -d: -f1 | sort | uniq -c | sort -n'

Billyzou0741326 avatar Mar 10 '20 04:03 Billyzou0741326

top指令看看有没有多个监控在跑吧 只跑一个的话应该不至于超上限...

Billyzou0741326 avatar Mar 10 '20 04:03 Billyzou0741326

top查看发现只有一个监控在跑，通过pm2 logs命令对其监视，发现

PM2      | App [live:0] exited with code [0] via signal [SIGINT]
PM2      | pid=17046 msg=process killed
PM2      | App [live:0] starting in -fork mode-
PM2      | App [live:0] online
0|live   |  [2020-03-10 17:12:48]   WS server listening on 0.0.0.0:8999
0|live   |  [2020-03-10 17:12:48]   WS server listening on 0.0.0.0:8998
0|live   |  [2020-03-10 17:12:48]   TCP server listening on 0.0.0.0:8997
0|live   |  [2020-03-10 17:12:48]   Http server listening on 0.0.0.0:9001
0|live   |  [2020-03-10 17:12:51]   Setting up monitor @room 21981359     in 娱乐区
0|live   |  [2020-03-10 17:12:51]   Setting up monitor @room 21093559     in 电台区
0|live   |  [2020-03-10 17:12:51]   Setting up monitor @room 10139633     in 绘画区
0|live   |  [2020-03-10 17:12:51]   Setting up monitor @room 5884993      in 手游区
0|live   |  [2020-03-10 17:12:51]   Setting up monitor @room 293122       in 单机区
0|live   |  [2020-03-10 17:12:51]   Setting up monitor @room 5468775      in 网游区
0|live   |  [2020-03-10 17:13:04]   2172290      @5101555      guard        舰长
0|live   |  [2020-03-10 17:13:56]   2172293      @8222458      guard        舰长
0|live   |  [2020-03-10 17:14:43]   2172295      @21513000     guard        舰长
0|live   |  [2020-03-10 17:14:52]   Bilibili.getLiveCount - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:54]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:55]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:55]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:55]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:55]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:55]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:55]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:55]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:55]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:55]   Bilibili.getRoomsInArea - Http request errored - getaddrinfo EBUSY api.live.bilibili.com
0|live   |  [2020-03-10 17:14:55]   Monitoring (静态) 77 + (动态) 15597
0|live   | [Error: EMFILE: too many open files, open '/root/bilibili-live-monitor-ts/build/db/record.json'] {
0|live   |   errno: -24,
0|live   |   code: 'EMFILE',
0|live   |   syscall: 'open',
0|live   |   path: '/root/bilibili-live-monitor-ts/build/db/record.json'
0|live   | }
^X^X0|live   |  [2020-03-10 17:15:30]   2172296      @146088       guard        舰长
1PM2      | Stopping app:live id:0
PM2      | pid=18205 msg=failed to kill - retrying in 100ms
PM2      | App [live:0] exited with code [0] via signal [SIGINT]
PM2      | pid=18205 msg=process killed

当CPU占用达到100%时执行ss -t | awk ' $5 ~ /^[0-9]/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n，发现连接数就卡在这里，后面没有任何增长

   1014 120.92.78.57
   1022 120.92.112.150
   1034 120.92.158.137

重新运行几次都是这样 但是很奇怪的是，使用node ./build/main.js (正常运行)却没有任何问题，几分钟后再次执行ss -t | awk ' $5 ~ /^[0-9]/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

  1419 120.92.158.137
  1455 120.92.144.250
  1474 120.92.112.150
  1485 120.92.78.57

一脸懵逼，再次修改此issue普通模式已经正常运行40分钟左右了 我™直接疑惑.jpg

luoshuijs avatar Mar 10 '20 09:03 luoshuijs

看起来通过命令查看连接数，判断pm2模式运行下似乎是超出限制了，但是普通模式并没有出现这种情况，这很玄学了。 不知道是运行环境问题，而且我也发现这个机子似乎有点问题（？） 再次修改此issue已经再次确认/etc/security/limits.conf并没问题内容如下

root soft nofile 65535
root hard nofile 65535
* soft nofile 65535
* hard nofile 65535

luoshuijs avatar Mar 10 '20 10:03 luoshuijs

刚刚注意到一个文件夹/etc/security/limits.d里面有个文件20-nproc.conf 刚刚通过网上查询到里面也有个限制 内容如下

*          soft    nproc     4096
root       soft    nproc     unlimited

刚刚通过计算器计算得连接数正好接近4096，难道是这个配置文件的限制？但是问题是在top命令显示的时候已经以root权限运行，讲道理不应该出现这种情况。

luoshuijs avatar Mar 10 '20 10:03 luoshuijs

刚刚修改了/etc/security/limits.d/20-nproc.conf文件，修改后如下

*          soft    nproc     65535
root       soft    nproc     unlimited

然后重启系统后（如果重新链接SSH不会生效），再次执行正常运行了5分钟，使用ss -t | awk ' $5 ~ /^[0-9]/ {print $5}' | cut -d: -f1 | sort | uniq -c | sort -n查看链接数内容如下

   2228 120.92.158.137
   2260 120.92.144.250
   2282 120.92.78.57
   2309 120.92.112.150

所以就很奇怪了....讲道理pm2已经以root权限运行，top查看也是这样。应该遵循第二行的配置文件无限制才对= =

luoshuijs avatar Mar 10 '20 10:03 luoshuijs

在这里我求求大佬优化一下异步操作吧，这样有点浪费系统资源啊 虽然没吃什么CPU但是讲道理哪怕是普通用户都用不着这么多 （运行20分钟后.....）

3762 120.92.158.137
3824 120.92.78.57
3871 120.92.144.250
3880 120.92.112.150

luoshuijs avatar Mar 10 '20 11:03 luoshuijs

刚刚查看了这个文章：https://blog.csdn.net/gzliudan/article/details/51754701 （我看了好像就Centos有这覆盖之前配置的魔法操作） 里面说了守护进程的文件数量限制的修改，很详细 希望大佬在README更新一下，谢谢！ （好了去写试卷了.jpg）

luoshuijs avatar Mar 10 '20 11:03 luoshuijs

嗯 我看着改一下文档

Billyzou0741326 avatar Mar 10 '20 12:03 Billyzou0741326

有关连接数——

最开始写监控我的监听目标仅包括所有开播中的房间，当时的舰长覆盖率可能低至70% 后来在运行过程中累积静态房间（“永久”监听目标）以至现在95%以上应该

跑监控的话连接数是不可抗力 除非不要覆盖率2333

Billyzou0741326 avatar Mar 10 '20 12:03 Billyzou0741326