go2rtc icon indicating copy to clipboard operation
go2rtc copied to clipboard

Add healthcheck

Open felipecrs opened this issue 2 years ago • 11 comments

This is work in progress, I will write more details before I mark it as ready for review.

Benefits:

  • Hass Supervisor watchdog (if enabled) will restart the container once it is unhealthy

For now, this is blocked by:

  • https://github.com/moby/buildkit/issues/4355

felipecrs avatar Oct 20 '23 01:10 felipecrs

API port may be closed or using another number

AlexxIT avatar Oct 20 '23 04:10 AlexxIT

Right, I have an idea to fix that.

felipecrs avatar Oct 20 '23 14:10 felipecrs

@felipecrs as go2rtc has static config path in docker, you could parse config file and use it in healthcheck sh script. It's possibility describes here https://stackoverflow.com/questions/55375371/using-a-sh-script-for-docker-healthchecks Anyway I've never seen go2rtc hanging... As variant, you could parse output existence of "go2rtc --version"

bonuzzz avatar Nov 23 '23 18:11 bonuzzz

Anyway I've never seen go2rtc hanging...

People in Frigate's world have seen go2rtc hanging many times. That's why Frigate (myself) built its own healthcheck for go2rtc to restart it when it hangs.

I, for one, have an issue open that describes a hanging situation happening in latest version 1.8.4:

  • https://github.com/AlexxIT/go2rtc/issues/762 (read last two paragraphs)

As variant, you could parse output existence of "go2rtc --version"

I don't see how invoking a new go2rtc process would help to assess the functionality of the existing one.

felipecrs avatar Nov 23 '23 19:11 felipecrs

@felipecrs as go2rtc has static config path in docker, you could parse config file and use it in healthcheck sh script. It's possibility describes here https://stackoverflow.com/questions/55375371/using-a-sh-script-for-docker-healthchecks

That's a very good idea, and it was what I was intending to do too. But then I forgot about this PR. lol

felipecrs avatar Nov 23 '23 19:11 felipecrs

I mean I had never seen hanging go2rtc myself and I don't know what exactly needs to check like rtsp port or web port. What is staying accessed when process hangs.

bonuzzz avatar Nov 23 '23 19:11 bonuzzz

Probably it could be related to this pr https://github.com/AlexxIT/go2rtc/pull/440 to track for example sighup signal

bonuzzz avatar Nov 23 '23 22:11 bonuzzz

@bonuzzz, can you elaborate?

felipecrs avatar Nov 23 '23 22:11 felipecrs

@felipecrs unfortunately I don't know how to reproduce go2rtc hanging in my environment. I think more complex solution is needed here than just to check open ports, because we don't know how the process behaves itself in hanging state. I saw your homeassistant automation to check stream status and in my opinion this is the best solution at the moment.

bonuzzz avatar Nov 23 '23 23:11 bonuzzz

Very dirty solution for frigate is parsing /dev/shm/logs/go2rtc/current for phrase "sync.(*Mutex).Lock(...)", "goroutine" of "WRN [streams] json locked" like tail -f /dev/shm/logs/go2rtc/current | grep "json locked"

bonuzzz avatar Nov 24 '23 00:11 bonuzzz

I think the current solution Frigate uses is already very good. :)

felipecrs avatar Nov 24 '23 00:11 felipecrs