yi-hack-v5 icon indicating copy to clipboard operation
yi-hack-v5 copied to clipboard

[YI_Outdoor 6CUS and YI_Dome 43US] Snapshot Freeze - Swap Issue

Open Minims opened this issue 3 years ago • 27 comments

Hello,

When I got the the snapshot page, the camera is unresponsive, but t still ping.

http://192.168.x.x:8080/?page=snapshot

it seems load increase very fast so snapshot can't be used.

Mem: 29952K used, 808K free, 1616K shrd, 20K buff, 3596K cached
CPU: 28.2% usr 68.3% sys  0.0% nic  0.0% idle  0.0% io  0.0% irq  3.3% sirq
Load average: 19.80 10.35 4.27 13/107 2433
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  956     1 root     S    24608 79.9   0 47.2 ./rmm
 1604     1 root     S     8028 26.0   0  7.2 ./p2p_tnp
 2418  2379 root     R    13488 43.8   0  5.6 imggrabber -m yi_outdoor -r high -w
 1703     1 root     S     2696  8.7   0  3.8 rRTSPServer -r high -p 554
  517     2 root     SW       0  0.0   0  3.4 [mmcqd/1]
 1602     1 root     S     3936 12.7   0  3.2 ./mp4record
  772     2 root     SW       0  0.0   0  3.2 [RtmpMlmeTask]
 1712     1 root     R     2488  8.0   0  3.0 h264grabber -r high -m yi_outdoor -f
  780     1 root     S     4240 13.7   0  2.8 ./dispatch
 2431  2195 root     R     1488  4.8   0  2.6 top
  771     2 root     RW       0  0.0   0  2.0 [RtmpTimerTask]
 1603     1 root     S     5328 17.3   0  1.6 ./cloud
  199     2 root     DW       0  0.0   0  1.2 [kswapd0]
 1607     1 root     S     2348  7.6   0  1.1 ./oss
 1743     1 root     S     2224  7.2   0  0.5 wsdd --pid_file /var/run/wsdd.pid --if_name wlan0 --type tdn:NetworkVideoTransmitter --xaddr http://%s --scope onvif://www.onvif.org/name/Unknown onvif://www.onvif.org/Profile/Streaming
 1689     1 root     S     2124  6.9   0  0.5 mqttv4
 2187  1678 root     S     1212  3.9   0  0.5 dropbear -R -B
 1608     1 root     S     1024  3.3   0  0.5 ./watch_process
 1152     1 root     S     1768  5.7   0  0.4 /home/base/tools/wpa_supplicant -c/tmp/wpa_supplicant.conf -g/var/run/wpa_supplicant-global -iwlan0 -B
    3     2 root     SW       0  0.0   0  0.3 [ksoftirqd/0]
   88     2 root     SW       0  0.0   0  0.2 [sync_supers]
 1765     1 root     S     1480  4.8   0  0.1 ntpd -p pool.ntp.org
 1683     1 root     S     1004  3.2   0  0.1 ipc_multiplexer
  779     1 root     S     1044  3.3   0  0.1 ./log_server
 1842     1 root     S     1020  3.3   0  0.1 ./arp_test
  403     2 root     SW       0  0.0   0  0.1 [kworker/0:1]
  105     2 root     DW       0  0.0   0  0.1 [kusbotg]
 1734     1 root     S     8068 26.2   0  0.0 onvif_srvd --pid_file /var/run/onvif_srvd.pid --model yi_outdoor --manufacturer Yi --firmware_ver 0.3.1 --hardware_id 6CUS --serial_num ************* --ifs wlan0 --port 80 --scope onvif://w
 2378  1642 root     S     1528  4.9   0  0.0 httpd -p 8080 -h /tmp/sd/yi-hack-v5/www/ -c /tmp/httpd.conf
 2114     1 root     S     1496  4.8   0  0.0 crond -c /tmp/sd/yi-hack-v5/etc/crontabs
 1464     1 root     S     1492  4.8   0  0.0 /sbin/udhcpc -i wlan0 -b -s /home/app/script/default.script -x hostname:YI-OutDoor
 2195  2187 root     S     1488  4.8   0  0.0 -sh
    1     0 root     S     1480  4.8   0  0.0 init
 1642     1 root     S     1480  4.8   0  0.0 httpd -p 8080 -h /tmp/sd/yi-hack-v5/www/ -c /tmp/httpd.conf
 1753     1 root     S     1480  4.8   0  0.0 /usr/sbin/crond -c /var/spool/cron/crontabs/
 2379  2378 root     S     1480  4.8   0  0.0 {snapshot.sh} /bin/sh snapshot.sh
 2433   780 root     R     1476  4.7   0  0.0 [sh]
 2419  2379 root     S     1472  4.7   0  0.0 base64
 1678     1 root     S     1188  3.8   0  0.0 dropbear -R -B
  427     1 root     S<    1020  3.3   0  0.0 udevd --daemon
  477   427 root     S<    1020  3.3   0  0.0 udevd --daemon
  480   427 root     S<    1020  3.3   0  0.0 udevd --daemon
``

Minims avatar May 29 '21 20:05 Minims

I hope that @roleoroleo can grab the Hisilicon model at some point and compile an app that will read directly from the kernel (at least that was roleo's plan at some point)

Anyone would like to donate a device?

alienatedsec avatar Jun 11 '21 12:06 alienatedsec

Yes, I could try it. But probably with a swap file you can solve the problem.

roleoroleo avatar Jun 11 '21 12:06 roleoroleo

I already have a swap file on my device as it is recommended, it doesn't help :-/. I have change my SD Card too without any Success.

Minims avatar Jun 13 '21 08:06 Minims

Please check if the swap file is really enabled. Your top command is doubtful.

roleoroleo avatar Jun 13 '21 10:06 roleoroleo

It seems really enabled :

/home/yi-hack-v5 # uptime
 10:22:35 up  1:30,  load average: 5.77, 5.61, 5.45
/home/yi-hack-v5 # ls -la /tmp/sd/swapfile 
-rwxr-xr-x    1 root     root      67108864 Jun 13 08:37 /tmp/sd/swapfile
/home/yi-hack-v5 # 

New Top without snapshot:

Mem: 29636K used, 1124K free, 1668K shrd, 340K buff, 9196K cached
CPU: 48.6% usr  8.3% sys  0.0% nic 41.8% idle  0.0% io  0.0% irq  1.0% sirq
Load average: 5.54 5.59 5.46 2/98 7141
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  976     1 root     S    24616 79.9   0 48.3 ./rmm
 1725     1 root     S     2356  7.6   0  1.8 rRTSPServer -r high -p 554
 7131  7068 root     R     1488  4.8   0  1.5 top
 1626     1 root     S     8028 26.0   0  1.3 ./p2p_tnp
  783     2 root     SW       0  0.0   0  1.0 [RtmpTimerTask]
 7063  1692 root     S     1212  3.9   0  0.9 dropbear -R -B
  784     2 root     SW       0  0.0   0  0.9 [RtmpMlmeTask]
 1624     1 root     S     5368 17.4   0  0.7 ./mp4record
  792     1 root     S     4240 13.7   0  0.7 ./dispatch
  403     2 root     SW       0  0.0   0  0.3 [kworker/0:1]
 1766     1 root     S     2224  7.2   0  0.1 wsdd --pid_file /var/run/wsdd.pid --if_name wlan0 --type tdn:NetworkVideoTransmitter --xaddr http://%s --scope onvif://www.onvif.org/name/Unknown onvif://www.onvif.org/Profile/Streaming
 1175     1 root     S     1768  5.7   0  0.1 /home/base/tools/wpa_supplicant -c/tmp/wpa_supplicant.conf -g/var/run/wpa_supplicant-global -iwlan0 -B
    3     2 root     SW       0  0.0   0  0.1 [ksoftirqd/0]
 1753     1 root     S     8068 26.2   0  0.0 onvif_srvd --pid_file /var/run/onvif_srvd.pid --model yi_outdoor --manufacturer Yi --firmware_ver 0.3.1 --hardware_id 6CUS --serial_num Y3SITD3M6S190419 --ifs wlan0 --port 80 --scope onvif:/
 1625     1 root     S     5328 17.3   0  0.0 ./cloud
 1732     1 root     S     2488  8.0   0  0.0 h264grabber -r high -m yi_outdoor -f
 1628     1 root     S     2348  7.6   0  0.0 ./oss
 1778     1 root     S     1504  4.8   0  0.0 /usr/sbin/crond -c /var/spool/cron/crontabs/
 7068  7063 root     S     1496  4.8   0  0.0 -sh
 1354     1 root     S     1492  4.8   0  0.0 /sbin/udhcpc -i wlan0 -b -s /home/app/script/default.script -x hostname:Yi-OutDoor
    1     0 root     S     1480  4.8   0  0.0 init
 1772     1 root     S     1480  4.8   0  0.0 ntpd -p pool.ntp.org
 2130     1 root     S     1480  4.8   0  0.0 crond -c /tmp/sd/yi-hack-v5/etc/crontabs
 1660     1 root     S     1480  4.8   0  0.0 httpd -p 8080 -h /tmp/sd/yi-hack-v5/www/ -c /tmp/httpd.conf
 1692     1 root     S     1188  3.8   0  0.0 dropbear -R -B
  791     1 root     S     1044  3.3   0  0.0 ./log_server
 1629     1 root     S     1024  3.3   0  0.0 ./watch_process
 1824     1 root     S     1020  3.3   0  0.0 ./arp_test
  427     1 root     S<    1020  3.3   0  0.0 udevd --daemon
  468   427 root     S<    1020  3.3   0  0.0 udevd --daemon
  470   427 root     S<    1020  3.3   0  0.0 udevd --daemon
 1702     1 root     S     1004  3.2   0  0.0 ipc_multiplexer

New Top with snapshot on going: (low res seems ok, high res load ans does not work) http://192.168.xx.xx:8080/cgi-bin/snapshot.sh?res=high&watermark=yes

Mem: 29680K used, 1080K free, 1660K shrd, 4K buff, 2884K cached
CPU:  3.9% usr 93.6% sys  0.0% nic  0.0% idle  0.0% io  0.0% irq  2.4% sirq
Load average: 13.76 7.66 6.17 24/105 7516
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
  976     1 root     R    24616 79.9   0 40.3 ./rmm
 1626     1 root     S     8028 26.0   0 12.5 ./p2p_tnp
  792     1 root     S     4240 13.7   0  4.5 ./dispatch
 1625     1 root     R     5328 17.3   0  3.9 ./cloud
  540     2 root     RW       0  0.0   0  3.5 [mmcqd/1]
 7513   792 root     R      844  2.7   0  3.4 [wpa_cli]
 1624     1 root     R     5368 17.4   0  3.2 ./mp4record
  784     2 root     RW       0  0.0   0  2.8 [RtmpMlmeTask]
  199     2 root     DW       0  0.0   0  2.7 [kswapd0]
 7306  7068 root     R     1488  4.8   0  2.5 top
  783     2 root     SW       0  0.0   0  2.3 [RtmpTimerTask]
 1725     1 root     R     2356  7.6   0  2.2 rRTSPServer -r high -p 554
 7514  1660 root     S     1524  4.9   0  1.7 httpd -p 8080 -h /tmp/sd/yi-hack-v5/www/ -c /tmp/httpd.conf
 1628     1 root     S     2348  7.6   0  1.6 ./oss
    3     2 root     SW       0  0.0   0  0.9 [ksoftirqd/0]
  791     1 root     S     1044  3.3   0  0.7 ./log_server
   88     2 root     SW       0  0.0   0  0.6 [sync_supers]
 1175     1 root     S     1768  5.7   0  0.6 /home/base/tools/wpa_supplicant -c/tmp/wpa_supplicant.conf -g/var/run/wpa_supplicant-global -iwlan0 -B
  105     2 root     DW       0  0.0   0  0.5 [kusbotg]
 1766     1 root     D     2224  7.2   0  0.5 wsdd --pid_file /var/run/wsdd.pid --if_name wlan0 --type tdn:NetworkVideoTransmitter --xaddr http://%s --scope onvif://www.onvif.org/name/Unknown onvif://www.onvif.org/Profile/Streaming
 7509  7462 root     D    13852 44.9   0  0.4 imggrabber -m yi_outdoor -r high -w
 1824     1 root     S     1020  3.3   0  0.4 ./arp_test
 1702     1 root     S     1004  3.2   0  0.4 ipc_multiplexer
 7063  1692 root     S     1212  3.9   0  0.3 dropbear -R -B
 1629     1 root     S     1024  3.3   0  0.2 ./watch_process
  403     2 root     SW       0  0.0   0  0.0 [kworker/0:1]
  987     2 root     SW       0  0.0   0  0.0 [hidog]
 1753     1 root     S     8068 26.2   0  0.0 onvif_srvd --pid_file /var/run/onvif_srvd.pid --model yi_outdoor --manufacturer Yi --firmware_ver 0.3.1 --hardware_id 6CUS --serial_num Y3SITD3M6S190419 --ifs wlan0 --port 80 --scope onvif:/
 1732     1 root     S     2488  8.0   0  0.0 h264grabber -r high -m yi_outdoor -f
 7460  1660 root     S     1524  4.9   0  0.0 httpd -p 8080 -h /tmp/sd/yi-hack-v5/www/ -c /tmp/httpd.conf
 1778     1 root     S     1504  4.8   0  0.0 /usr/sbin/crond -c /var/spool/cron/crontabs/
 7068  7063 root     S     1496  4.8   0  0.0 -sh
 1354     1 root     S     1492  4.8   0  0.0 /sbin/udhcpc -i wlan0 -b -s /home/app/script/default.script -x hostname:Yi-OutDoor
    1     0 root     S     1480  4.8   0  0.0 init
 7515  7514 root     R     1480  4.8   0  0.0 {status.json} /bin/sh status.json
 1660     1 root     S     1480  4.8   0  0.0 httpd -p 8080 -h /tmp/sd/yi-hack-v5/www/ -c /tmp/httpd.conf
 1772     1 root     S     1480  4.8   0  0.0 ntpd -p pool.ntp.org
 2130     1 root     S     1480  4.8   0  0.0 crond -c /tmp/sd/yi-hack-v5/etc/crontabs
 7462  7460 root     S     1480  4.8   0  0.0 {snapshot.sh} /bin/sh snapshot.sh
 1692     1 root     S     1188  3.8   0  0.0 dropbear -R -B
  427     1 root     S<    1020  3.3   0  0.0 udevd --daemon
  468   427 root     S<    1020  3.3   0  0.0 udevd --daemon
  470   427 root     S<    1020  3.3   0  0.0 udevd --daemon
  678     2 root     SWN      0  0.0   0  0.0 [jffs2_gcd_mtd5]
  413     2 root     SWN      0  0.0   0  0.0 [jffs2_gcd_mtd4]
    7     2 root     SW       0  0.0   0  0.0 [kworker/u:1]
  103     2 root     SW       0  0.0   0  0.0 [khubd]
  335     2 root     SW       0  0.0   0  0.0 [mtdblock2]

Minims avatar Jun 13 '21 10:06 Minims

Try: cat /proc/meminfo | grep Swap

roleoroleo avatar Jun 13 '21 17:06 roleoroleo

/home/yi-hack-v5 # free
              total        used        free      shared  buff/cache   available
Mem:          30760       19880        1128           0        9752           0
-/+ buffers/cache:        19880       10880
Swap:         65532          84       65448
/home/yi-hack-v5 # cat /proc/meminfo | grep -i Swap
SwapCached:           80 kB
SwapTotal:         65532 kB
SwapFree:          65448 kB
/home/yi-hack-v5 # 

Minims avatar Jun 13 '21 18:06 Minims

/home/yi-hack-v5 # cat /proc/meminfo | grep -i Swap SwapCached: 80 kB SwapTotal: 65532 kB SwapFree: 65448 kB

@roleoroleo do you think that Swap is not utilised enough to manage the memory allocation? I find the same on yi_dome where Swap has minimal usage even the main memory is almost fully used.

alienatedsec avatar Jun 14 '21 10:06 alienatedsec

Maybe some proc parameter. For example, what's the value of swappiness?

cat /proc/sys/vm/swappiness

roleoroleo avatar Jun 14 '21 10:06 roleoroleo

/home/yi-hack-v5 # cat /proc/sys/vm/swappiness
0
/home/yi-hack-v5 #

alienatedsec avatar Jun 14 '21 10:06 alienatedsec

I changed it to 60 to see the difference

/home/yi-hack-v5 # echo 60 >> /proc/sys/vm/swappiness
/home/yi-hack-v5 # cat /proc/sys/vm/swappiness
60
/home/yi-hack-v5 # reboot

alienatedsec avatar Jun 14 '21 10:06 alienatedsec

Probably there is a line that forces this value during the boot. init.sh or other startup scripts. If you reboot it will come back to 0.

roleoroleo avatar Jun 14 '21 10:06 roleoroleo

Probably there is a line that force this value during the boot. init.sh or other startup scripts. If you reboot it will come back to 0.

Correct, it was 0, I am changing it to the following:


sysctl -w vm.dirty_background_ratio=2
sysctl -w vm.dirty_ratio=5
sysctl -w vm.dirty_writeback_centisecs=100
sysctl -w vm.dirty_expire_centisecs=500
sysctl -w vm.vfs_cache_pressure=200
sysctl -w vm.swappiness=40
echo 5 > /proc/sys/vm/laptop_mode
echo 60 > /proc/sys/vm/swappiness

alienatedsec avatar Jun 14 '21 10:06 alienatedsec

It works - it took 14 seconds for a snapshot but no hanging anymore. It could resolve many issues moving forward. I tested it with the v0.3.1 so before even your commits addressing memory allocation.

Thanks @roleoroleo

/home/yi-hack-v5 # cat /proc/meminfo | grep -i Swap
SwapCached:          716 kB
SwapTotal:         65532 kB
SwapFree:          63324 kB
/home/yi-hack-v5 # free
              total        used        free      shared  buff/cache   available
Mem:          18592       12096         828           0        5668           0
-/+ buffers/cache:        12096        6496
Swap:         65532        2188       63344
/home/yi-hack-v5 #

alienatedsec avatar Jun 14 '21 11:06 alienatedsec

/home/yi-hack-v5 # cat /proc/sys/vm/swappiness 0 /home/yi-hack-v5 #

Same on my camera.


sysctl -w vm.dirty_background_ratio=2
sysctl -w vm.dirty_ratio=5
sysctl -w vm.dirty_writeback_centisecs=100
sysctl -w vm.dirty_expire_centisecs=500
sysctl -w vm.vfs_cache_pressure=200
sysctl -w vm.swappiness=40
echo 5 > /proc/sys/vm/laptop_mode
echo 60 > /proc/sys/vm/swappiness

trying

Minims avatar Jun 14 '21 11:06 Minims

Took around 15s to get high res snapshot. Seems really better. Thx ! Done on 0.3.1 BETA. Load still around 6, but camera works great.

/home/yi-hack-v5 # uptime 
 11:08:59 up 16:32,  load average: 6.07, 6.33, 6.25

Minims avatar Jun 14 '21 11:06 Minims

Probably there is a line that force this value during the boot. init.sh or other startup scripts. If you reboot it will come back to 0.

Correct, it was 0, I am changing it to the following:

sysctl -w vm.dirty_background_ratio=2
sysctl -w vm.dirty_ratio=5
sysctl -w vm.dirty_writeback_centisecs=100
sysctl -w vm.dirty_expire_centisecs=500
sysctl -w vm.vfs_cache_pressure=200
sysctl -w vm.swappiness=40
echo 5 > /proc/sys/vm/laptop_mode
echo 60 > /proc/sys/vm/swappiness

Can confirm this on my 6CUS camera as well. The high quality snapshot now works, even if it takes a little while.

ping-localhost avatar Aug 18 '21 23:08 ping-localhost

Same for YI Outdoor 4CUS. Swap was enabled but not used without this extra sysctl settings.

dawidmalina avatar Sep 20 '21 20:09 dawidmalina

Probably there is a line that force this value during the boot. init.sh or other startup scripts. If you reboot it will come back to 0.

Correct, it was 0, I am changing it to the following:


sysctl -w vm.dirty_background_ratio=2
sysctl -w vm.dirty_ratio=5
sysctl -w vm.dirty_writeback_centisecs=100
sysctl -w vm.dirty_expire_centisecs=500
sysctl -w vm.vfs_cache_pressure=200
sysctl -w vm.swappiness=40
echo 5 > /proc/sys/vm/laptop_mode
echo 60 > /proc/sys/vm/swappiness

hi, sorry, where do I enter this text in order to make it work on camera boot? thanks!

bednim avatar Sep 29 '21 18:09 bednim

hi, sorry, where do I enter this text in order to make it work on camera boot? thanks!

/home/app/init.sh

line 19 - yi_dome

Have a look as it is called the same

@bednim

alienatedsec avatar Sep 30 '21 19:09 alienatedsec

hi, sorry, where do I enter this text in order to make it work on camera boot? thanks!

/home/app/init.sh

line 19 - yi_dome

Have a look as it is called the same

@bednim

thanks - it works :)

bednim avatar Sep 30 '21 19:09 bednim

hi, sorry, where do I enter this text in order to make it work on camera boot? thanks!

/home/app/init.sh

line 19 - yi_dome

Have a look as it is called the same

@bednim

Excuse me but.. How to edit this .sh file ? A mean, what is the command ? :/

Lecl0ne avatar Oct 01 '21 19:10 Lecl0ne

You can use default shell editor vi - it's difficult at start but it's doable.

On Fri, 1 Oct 2021, 21:51 Lecl0ne, @.***> wrote:

hi, sorry, where do I enter this text in order to make it work on camera boot? thanks!

/home/app/init.sh

line 19 - yi_dome

Have a look as it is called the same

@bednim https://github.com/bednim

Excuse me but.. How to edit this .sh file ? A mean, what is the command ? :/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/alienatedsec/yi-hack-v5/issues/63#issuecomment-932507988, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKOYRFFGMASAHBX55YUBOWLUEYGKNANCNFSM45YT3M2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

bednim avatar Oct 01 '21 20:10 bednim

Like vi /home/app/init.sh ?

Lecl0ne avatar Oct 01 '21 20:10 Lecl0ne

Yep

On Fri, 1 Oct 2021, 22:45 Lecl0ne, @.***> wrote:

Like vi /home/app/init.sh ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/alienatedsec/yi-hack-v5/issues/63#issuecomment-932539748, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKOYRFB6QXQOQ4IHJSFW7D3UEYMYLANCNFSM45YT3M2A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

bednim avatar Oct 01 '21 21:10 bednim

Doesn't seem to work for me... added a couple of those lines but the rstp feed is never accessible from vlc or any other app.

smnhdy avatar Jan 20 '22 12:01 smnhdy

@smnhdy v0.3.4 should fix your problems (partially at least)

alienatedsec avatar Sep 30 '22 11:09 alienatedsec

This issue has been stale for 30 days - it will be closed within the next 7 days if not updated

github-actions[bot] avatar Jul 10 '23 06:07 github-actions[bot]