yi-hack-v5 icon indicating copy to clipboard operation
yi-hack-v5 copied to clipboard

Performance issues YI Outdoor (high load average 9CUS)

Open bakemonoru opened this issue 3 years ago • 9 comments

Performance issues YI Outdoor This issue related to https://github.com/TheCrypt0/yi-hack-v4/issues/173 I have Yi outdoor camera with base version 3.0.0.0D_201809111054 and firmware hack-v5 0.3.1 installed Settings: swap enabled SDcard: brand new 16Gb Sandisk Ultra 10+ class (80mb\s) Enabled services: SSH, HTTPD Other services are disabled: camera (ipc_cmd -t OFF), RTSP, Onvif, etc Camera is really warm and sdcard when inserted is very hot, looks like a heavy I\O was performed

But I have load average : ~ 4.18 4.11 3.49 As result - poor web performance and recording issues.

Here some additional info

Mem: 28888K used, 1872K free, 1608K shrd, 1824K buff, 8996K cached
CPU: 30.3% usr 10.5% sys  0.0% nic 57.0% idle  0.0% io  0.0% irq  2.0% sirq
Load average: 3.98 4.09 2.55 1/79 4567
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  973     1 root     S    23056 74.8   0 37.8 ./rmm
 4447  2391 root     R     1484  4.8   0  1.3 top
  783     2 root     SW       0  0.0   0  1.1 [RtmpTimerTask]
 2387  1657 root     S     1256  4.0   0  0.9 dropbear -R -B
  792     1 root     S     4236 13.7   0  0.6 ./dispatch
 1626     1 root     S     3996 12.9   0  0.6 ./mp4record
  784     2 root     SW       0  0.0   0  0.1 [RtmpMlmeTask]
  403     2 root     SW       0  0.0   0  0.1 [kworker/0:1]
 1174     1 root     S     1764  5.7   0  0.0 /home/base/tools/wpa_supplicant -c/tmp/wpa_supplicant.conf -g/var/run/wpa_supplicant-global -iwlan0 -B
 2391  2387 root     S     1500  4.8   0  0.0 -sh
 1348     1 root     S     1492  4.8   0  0.0 /sbin/udhcpc -i wlan0 -b -s /home/app/script/default.script -x hostname:testyi
    1     0 root     S     1480  4.8   0  0.0 init
 1642     1 root     S     1480  4.8   0  0.0 httpd -p 8080 -h /tmp/sd/yi-hack-v5/www/ -c /tmp/httpd.conf
 2014     1 root     S     1480  4.8   0  0.0 crond -c /tmp/sd/yi-hack-v5/etc/crontabs
 1696     1 root     S     1480  4.8   0  0.0 /usr/sbin/crond -c /var/spool/cron/crontabs/
 1657     1 root     S     1188  3.8   0  0.0 dropbear -R -B
  791     1 root     S     1044  3.3   0  0.0 ./log_server
  427     1 root     S<    1020  3.3   0  0.0 udevd --daemon
  475   427 root     S<    1020  3.3   0  0.0 udevd --daemon
  471   427 root     S<    1020  3.3   0  0.0 udevd --daemon
 1661     1 root     S     1004  3.2   0  0.0 ipc_multiplexer


/home/yi-hack-v5 # iostat
Linux 3.4.35 (testyi)   05/19/21        _armv5tejl_     (1 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          32.34    0.00   23.05    0.05    0.00   44.55

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
mtdblock2         0.01         0.12         0.02         40          8
mtdblock6         0.00         0.39         0.00        128          0
mmcblk0           0.80        54.92         1.16      18132        382
mmcblk0p1         0.80        54.89         1.16      18124        382

/home/yi-hack-v5 # mount
rootfs on / type rootfs (rw)
/dev/root on / type jffs2 (rw,relatime)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /dev type tmpfs (rw,relatime)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/mtdblock5 on /home type jffs2 (rw,relatime)
none on /dev/mqueue type mqueue (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime,size=16384k)
/dev/mmcblk0p1 on /tmp/sd type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mmcblk0p1 on /tmp/sd/yi-hack-v5/www/record type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

/home/yi-hack-v5 # df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 1.3M      1.2M     88.0K  93% /
tmpfs                    15.0M      4.0K     15.0M   0% /dev
/dev/mtdblock5           12.7M      8.8M      3.9M  70% /home
tmpfs                    16.0M      1.6M     14.4M  10% /tmp
/dev/mmcblk0p1           14.8G     82.8M     14.7G   1% /tmp/sd
/dev/mmcblk0p1           14.8G     82.8M     14.7G   1% /tmp/sd/yi-hack-v5/www/record
/home/yi-hack-v5 #

rmm process killed. LA reverted to normal values (above 1.00) Started again: LA backed to 4.0+, it's not depend enabled camera or not

/home/app # ./rmm
sensor_register_callback ok
linear mode
I2C_WRITE error!
addr = fd data =0
===ominivision ov2735 sensor new 1080P15fps(Parallel port) init success!=====
=========================================================
cmos_fps_set1 =20.000000
vi_start_devvi_start_dev =============

MMAP address=0xb5dbf000, size=1586752(0x183640), mode=0, try 1
cmos_fps_set1 =20.000000

***** SMART BABYCRYING - V20160905.1 - *****
  --- pmstat init_flt2ll[1] diff:1320684 us, totalTm: 1320684 us

***** SMART BABYCRYING - V20160412.2 - *****
ppmngr:0xe4fb28, len:320000

Than enabled camera (ipc_cmd -t ON) after 1 min.


stExposureAttr.stAuto.u8Compensation=40
===============u32DstStrength =======8
stExposureAttr.stAuto.u8Compensation=40
===============u32DstStrength =======8
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=50
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=47
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=45
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=47
===============u32DstStrength =======32
stExposureAttr.stAuto.u8Compensation=40
===============u32DstStrength =======8
stExposureAttr.stAuto.u8Compensation=40
===============u32DstStrength =======8
u32FullLines2 =2152
u32FullLines2 =1994
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=60
===============u32DstStrength =======128
stExposureAttr.stAuto.u8Compensation=50
===============u32DstStrength =======128

bakemonoru avatar May 19 '21 20:05 bakemonoru

@bakemonoru they suggest trying a different PSU. Have you tried that?

I can see a trend of issues with this model, so trying to eliminate them one by one. Also, I have no camera to test, so it's hard to support it fully.

alienatedsec avatar May 19 '21 22:05 alienatedsec

@alienatedsec I tried different high quality PSUs, original power adapter too. any ideas ? i can provide any help, including remote ssh, to make yi-hack better.

bakemonoru avatar May 20 '21 11:05 bakemonoru

@alienatedsec

I tried different high quality PSUs, original power adapter too.

any ideas ?

i can provide any help, including remote ssh, to make yi-hack better.

@bakemonoru Send me a message on Discord, please.

alienatedsec avatar May 20 '21 11:05 alienatedsec

I have the same issue on mine :

/home/yi-hack-v5 # uptime 
 17:28:37 up 16:51,  load average: 6.98, 6.60, 6.74
/home/yi-hack-v5 #
Firmware Version | 0.3.1
-- | --
Base Version | 3.0.0.0D_201809111054
Model Suffix | yi_outdoor
Hardware ID | 6CUS

top -H

Mem: 28912K used, 1848K free, 1704K shrd, 228K buff, 9692K cached
CPU: 60.0% usr 23.9% sys  0.0% nic 13.3% idle  0.0% io  0.0% irq  2.6% sirq
Load average: 6.38 6.46 6.65 3/101 26420
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 1049     1 root     R<   24616 79.9   0 31.8 {ai_get_lpcmstre} ./rmm
 1051     1 root     R    24616 79.9   0  8.5 {motion_proc} ./rmm
 1039     1 root     S    24616 79.9   0  7.7 ./rmm
 1012     1 root     S    24616 79.9   0  2.6 {hi_Adec0_Send} ./rmm
26358 25489 root     R     1488  4.8   0  2.4 top -H
 1047     1 root     S    24616 79.9   0  1.9 {venc_get_h264st} ./rmm
 1714     1 root     S     2696  8.7   0  1.9 rRTSPServer -r high -p 554
  771     2 root     SW       0  0.0   0  1.9 [RtmpTimerTask]
 1048     1 root     S    24616 79.9   0  1.3 {venc_get_h264st} ./rmm
25479  1680 root     S     1212  3.9   0  1.3 dropbear -R -B
  772     2 root     SW       0  0.0   0  1.1 [RtmpMlmeTask]
 1013     1 root     S    24616 79.9   0  0.9 {hi_Adec0_Dec} ./rmm
 1721     1 root     S     2488  8.0   0  0.9 h264grabber -r high -m yi_outdoor -f
 1639     1 root     S     8028 26.0   0  0.8 ./p2p_tnp
 1610     1 root     S     5236 17.0   0  0.8 ./mp4record
 1050     1 root     S<   24616 79.9   0  0.6 {ai_get_aacstrea} ./rmm
25066     1 root     S     8028 26.0   0  0.3 ./p2p_tnp
  964     1 root     S     4240 13.7   0  0.3 ./dispatch
    3     2 root     SW       0  0.0   0  0.3 [ksoftirqd/0]
  961     1 root     S    24616 79.9   0  0.1 ./rmm
 1612     1 root     S     8028 26.0   0  0.1 ./p2p_tnp
25065     1 root     S     8028 26.0   0  0.1 ./p2p_tnp
  963     1 root     S     4240 13.7   0  0.1 ./dispatch
 1752     1 root     S     2220  7.2   0  0.1 wsdd --pid_file /var/run/wsdd.pid --if_name wlan0 --type tdn:NetworkVideoTransmitter --xaddr http://%s --scope onvif://www.onvif.org/name/Unknown onvif://www.onvif.org/Profile/Streaming
 1168     1 root     S     1764  5.7   0  0.1 /home/base/tools/wpa_supplicant -c/tmp/wpa_supplicant.conf -g/var/run/wpa_supplicant-global -iwlan0 -B
 1646     1 root     S     1480  4.8   0  0.1 httpd -p 8080 -h /tmp/sd/yi-hack-v5/www/ -c /tmp/httpd.conf
  105     2 root     DW       0  0.0   0  0.1 [kusbotg]
  403     2 root     SW       0  0.0   0  0.1 [kworker/0:1]
 1045     1 root     S<   24616 79.9   0  0.0 {ao_send_lpcmstr} ./rmm
 1044     1 root     S    24616 79.9   0  0.0 {msg_proc} ./rmm
 1046     1 root     S<   24616 79.9   0  0.0 {ao_get_711strea} ./rmm
 1001     1 root     S    24616 79.9   0  0.0 {hi_Aenc0_Get} ./rmm
 1746     1 root     S     8068 26.2   0  0.0 onvif_srvd --pid_file /var/run/onvif_srvd.pid --model yi_outdoor --manufacturer Yi --firmware_ver 0.3.1 --hardware_id 6CUS --serial_num ******** --ifs wlan0 --port 80 --scope onvif:/
 1643     1 root     S     8028 26.0   0  0.0 ./p2p_tnp
 1642     1 root     S     8028 26.0   0  0.0 ./p2p_tnp
 1638     1 root     S     8028 26.0   0  0.0 ./p2p_tnp
 1611     1 root     S     5328 17.3   0  0.0 ./cloud
 1666     1 root     S     5328 17.3   0  0.0 ./cloud
 1665     1 root     S     5328 17.3   0  0.0 ./cloud
 1660     1 root     S     5328 17.3   0  0.0 ./cloud
 1661     1 root     S     5328 17.3   0  0.0 ./cloud
 1620     1 root     S     5236 17.0   0  0.0 ./mp4record
  965     1 root     S     4240 13.7   0  0.0 ./dispatch

Minims avatar May 29 '21 17:05 Minims

Please check #70 as could be related

alienatedsec avatar Jun 11 '21 12:06 alienatedsec

Seems helping a bit, but I still have a load around of 5 or 6 (without making any snapshot).

Minims avatar Jun 13 '21 08:06 Minims

I am having the same issue. The process rmm generates a high load. Whenever I tried to kill that process, the camera was rebooting.

What is this process doing? Where is it coming from ? I could not find it in the git repo

gdoumen avatar Aug 30 '21 12:08 gdoumen

@bakemonoru @Minims @gdoumen - please try v0.3.3

alienatedsec avatar Sep 26 '22 10:09 alienatedsec

Load is around 6.

Load Average | 5.51 5.55 5.44

low res snapshot helps.

this is quite stable for now. We see if it can be stable for a week.

Minims avatar Sep 26 '22 20:09 Minims

many performance issues resolved with the latest version 0.3.6

alienatedsec avatar Dec 05 '22 20:12 alienatedsec

i have the same problem, all my yi outdoor camera (9CUS) have high cpu load all the time. The same iostat as mention before. On all three cameras i have 0.3.7 version uploaded. I use only rtsp stream (rest features all off) with frigate, but stream all the time disconnecting from time to time, and reconnect.. The stream have some lost frames, and micro freezing. The low stream help a little bit, but cpu still have high cpu load.

My memory stats are: 11568/30760 KB, swap dont help.

The process eat most of cpu is rmm with top -H its {ai_get_lpcmstre} ./rmm

skupi20 avatar Jan 11 '23 09:01 skupi20

Yes I still have this behavior. Something strange too is that when you save you configuration in the web interface it takes a lot of seconds to display saved. I have not this issue with yi-hack-v4, it quite instant. Any idea

Minims avatar Jan 11 '23 20:01 Minims

Unfortunately, I have no camera to test, so it's hard to improve anything.

alienatedsec avatar Jan 11 '23 22:01 alienatedsec

Which test / debug / informations can we provide to help ?

Minims avatar Jan 11 '23 22:01 Minims

@alienatedsec , sorry to ask, but what country you are from? maybe someone could send you one camera for tests.

Second thought, what is that thread/subprocess - "ai_get_lpcmstre" from rmm process?

skupi20 avatar Jan 12 '23 07:01 skupi20

@alienatedsec , sorry to ask, but what country you are from? maybe someone could send you one camera for tests.

Thanks for the offer. Lets say that is not necessary at this moment.

Second thought, what is that thread/subprocess - "ai_get_lpcmstre" from rmm process?

It looks like anything related to additional functionality e.g., camera on/off, nightvision on/off, motion detection etc.

I was wrong - more explanation here

alienatedsec avatar Jan 12 '23 17:01 alienatedsec

I have some thoughts on why this model has constant performance issues:

  • the original firmware is running on the old busybox version
  • yi-hack-v5 relies on newer busybox to bring functionality, e.g., FTP, SSH, SwapFile etc. A full list is here
  • a newer version may conflict with the underlying system and create performance issues

There is only one way to find out, and it looks like I will have to redesign a lot before it can be achieved:

  • point to any busybox-based module and extract it to a separate folder - the same way as other hacks do
  • update libraries in the code
  • test test test :)

The above will likely fix other performance issues - it is next on my TO-DO list. However, I tried in the past and it will take me ages to do so.

alienatedsec avatar Jan 12 '23 18:01 alienatedsec

@alienatedsec i read that first yi-hack for camera run rtsc server (rtsp2301) alone without rmm. There was operation like kill rmm process before run rtsc server. @andy2301 propose use streams from /tmp/view which did rmm process. That give possibility to have rtsc server and the stock functions work together.

As i understand rmm process is only need for stock functions coming from yi, like cloud recording, two stream etc. And you hack work with that idea by using two streams from /tmp/view.

Perhaps you can make version of hack with rtsc server and 1080p stream which dont need those streams coming from rmm process only for tests. Or is it possible to start rtsc server without using rmm streams? I will kill rmm process and start rtsc alone.

As i understand the problem with rmm process could be that its run with not compatible busybox version?

skupi20 avatar Jan 13 '23 09:01 skupi20

As i understand the problem with rmm process could be that its run with not compatible busybox version?

That is my assumption and understanding. The more important is the yi_outdoor series is based on the YI/Xiaomi v3.0.0 firmware whereas other cameras are v1.9.x and v2.x.x

As i understand rmm process is only need for stock functions coming from yi, like cloud recording, two stream etc. And you hack work with that idea by using two streams from /tmp/view

The RTSP module of the hack is fed by the output of the /tmp/view https://github.com/alienatedsec/yi-hack-v5/blob/3f6d3ad32922211a5b71156a6b5cc6326a91fde2/src/h264grabber/h264grabber/h264grabber.c#L137

So, while one process (or possibly more) is reading the file, the other process is writing into it at almost the same time.

I also believe that /tmp/view is not created or even updated if the rmm process is killed.

alienatedsec avatar Jan 13 '23 09:01 alienatedsec

So, while one process (or possibly more) is reading the file, the other process is writing into it at almost the same time. I also believe that /tmp/view is not created or even updated if the rmm process is killed.

Yes. When i killed rmm process i got black screen on latest version camera crash :) Is there a rtsp version which dont use stream from /tmp/view. Andy2301 mention that first rtsp server requires to shut down the stock streaming process rmm. As we have performance problem with rmm process, before you will have time to find solution it will be nice to have stable 1080p rtsp server only :)

edited: On issue page of v4 https://github.com/TheCrypt0/yi-hack-v4/issues/294#issuecomment-833386801 there is answer that someone have also choppy, freezing problem on v5 and stay on v4 where there is no problem

I uploaded v4, as i cant enabled rtsp server as project is dead, but i check the top and i get load average 3-5, but without rtsp server running. So cant check it.

So maybe its problem of rtsp or h264grabber, cause when i download recorded files from sd card they were fluid without micro freezing. When i killed rtsp server before wd re-lanuch it i get simillar cpu load as on v4.

skupi20 avatar Jan 13 '23 10:01 skupi20

Hello, I have exactly the same problem on my Yi outdoor camera. I hope you find a solution to fix the problem.

b0b33140 avatar Jan 29 '23 18:01 b0b33140

I will be open and honest that it is hard to test or improve anything without the camera. Does anyone have a link or a source where I could still purchase one?

alienatedsec avatar Jan 29 '23 19:01 alienatedsec

Amazon or Aliexpress

b0b33140 avatar Jan 29 '23 20:01 b0b33140

Amazon or Aliexpress

Thanks @b0b33140 Those are likely newer models. We are talking about models sold in 2015

alienatedsec avatar Jan 30 '23 11:01 alienatedsec

It's strange I bought mine in 2019. image

b0b33140 avatar Jan 31 '23 08:01 b0b33140

Mine is from 2019 too and is a 9CUS

Minims avatar Jan 31 '23 20:01 Minims

We are talking about models sold in 2015

I should quote the above and correct myself - the YI cameras with the chipset supported by this hack were available from late 2015 in China. I bought my 1080p model in 2017 and managed to get a 720p model in 2018 - both from Amazon.

Now, yi_outdoor was a later model (I think) and you got it in 2019, which is fine, but recent models are QFUS supported by https://github.com/roleoroleo/yi-hack-Allwinner-v2

image

alienatedsec avatar Feb 02 '23 08:02 alienatedsec

This issue has been stale for 30 days - it will be closed within the next 7 days if not updated

github-actions[bot] avatar Jul 11 '23 06:07 github-actions[bot]