remote detection using zeromq

[dennispg]

I took a stab at reworking a lot of things in order to enable detections to happen over zeromq sockets using the majordomo protocol. By default everything will happen with IPC sockets, but I've enabled the ability to run over TCP sockets as well.

I also added some configuration to be able to start frigate running only the configured detectors to connect to the main frigate instance. Detectors are the "workers", the detect process from track_camera is the "client"

Please share your feedback!

[netlify[bot]]

✅ Deploy Preview for frigate-docs canceled.

Name	Link
🔨 Latest commit	800105fbffe8ccfbdd137be72d1b9ee1c8e1cf64
🔍 Latest deploy log	https://app.netlify.com/sites/frigate-docs/deploys/63ae0da595deaa0009ff6494

[blakeblackshear]

This isn't quite what I had in mind. I want to consider zeromq as a message bus for all interprocess communications, not just detection.

[dennispg]

This isn't quite what I had in mind. I want to consider zeromq as a message bus for all interprocess communications, not just detection.

I had the same thought too, but wanted to start here first. Initially I was thinking we could use the single broker and send everything through it with different service names.

But then I had to extend the majordomo broker a bit to enable sending different replies based on whether the worker could used shared memory or not. Since the client has no idea about which worker still handle the request it has to send the request consistently, but then I didn't want to send the tensor input over the wire when SHM is much quicker when available. Those makes the broker detection specific.

So then my idea was that we could just add more brokers for the non detection based ipc but the downside is still that it's a little more complicated my way just because I modified the majortomo stuff.

Maybe to simplify so that we can reuse the unmodified majortomo broker for all ipc, we could add an additional proxy client/broker mechanism for non shm TCP communications, or something like that?

I just really wanted to submit what I had so far to get feedback on the approach itself and the decisions that have to be made behind them.

[dennispg]

I've reworked the majordomo broker to be completely generic and usable for all interprocess communication.

[dennispg]

I am facing a handful of issues here that I think will require some deeper discussion:

1. Right now a default CPU detector is created if no detectors are defined. If we wanted to open up the scenario for all detectors to be remote, this should probably be set to an empty set?
2. That opens up the question of what should happen if there are not any detectors connected.. As it is right now the frame queue would just fill up but then what?
3. With different detector workers being able to be connected and disconnected over time, we can never really know what the maxmimum frame size would be.. can we change the size of the SharedMemory for each camera to some default maximum and let the maximum be configurable?
4. I've set things up using zeromq and the majordomo protocol pattern. I started by referencing the python library named majortomo, but quickly hit up against some deficiencies. I ended up changing quite a big of the source in that library, and for now I've added it to a subfolder containing all my modifications. I imagine theres probably a better way..
5. One limitation of the majordomo protocol is that the client request will wait until it is fulfilled. In our case the client is the object detect process. I think the preference would be to wait for a certain amount of time, and then give up and move on, but this will probably require a few more modifications to the library.
6. Is there any other mechanism we should be considering? I think if somehow we were to split up the request for a detection and the receiving of the result of a detection into two different processes, we could try a different protocol than majordomo.

[NickM-27]

Right now a default CPU detector is created if no detectors are defined. If we wanted to open up the scenario for all detectors to be remote, this should probably be set to an empty set?

I really don't think that's a good idea, just like before when the coral was the default I imagine we'll get tons of support requests from users who simply didn't set any detectors up. also I expect remote detectors to be a somewhat niche usecase relative to the local coral. I'm not sure what benefit there is to a remote CPU detector.

[dennispg]

Right now a default CPU detector is created if no detectors are defined. If we wanted to open up the scenario for all detectors to be remote, this should probably be set to an empty set?

I really don't think that's a good idea, just like before when the coral was the default I imagine we'll get tons of support requests from users who simply didn't set any detectors up. also I expect remote detectors to be a somewhat niche usecase relative to the local coral. I'm not sure what benefit there is to a remote CPU detector.

I do agree about the default use case needing to make available some default, so I take that back. But, though the use case may be niche, I don't think its so unreasonable. The benefit of remote detectors is that the load can be distributed among multiple compute resources. Especially since corals are still so difficult to source.

[NickM-27]

Right, I see the benefit of remote detectors but specifically a remote CPU detector I'm not sure. The problem would be inference speeds and anyone with multiple devices to spread the load would likely be doing so on an Intel platform or something with a GPU which would work with openvino or another detector anyway. I may be missing something but just wanted to elaborate on my thinking.

[blakeblackshear]

I appreciate the initial work done here, but as you said this will require a good bit of planning and discussion. I will need to spend some time reading about zeromq patterns to see what is the best approach for frigate. I was initially thinking that this could offer a nice mqtt-like message bus system for communication between frigate processes and even allow some parts to be written in other more performant languages. But my knowledge of zeromq is very basic at this point, so I could be off base.

[dennispg]

I appreciate the initial work done here, but as you said this will require a good bit of planning and discussion. I will need to spend some time reading about zeromq patterns to see what is the best approach for frigate. I was initially thinking that this could offer a nice mqtt-like message bus system for communication between frigate processes and even allow some parts to be written in other more performant languages. But my knowledge of zeromq is very basic at this point, so I could be off base.

Yea, a generic message bus enabling any language is what I was going for with this, but I think there might need to be a few more foundational architectural changes in frigate itself to enable it in more of frigate's processes.

Anyway, I am happy to discuss more whenever the time is right, just let me know! I have enjoyed contributing so I'll be around to see if there are any other ways I can be helpful.

[dennispg]

Oh actually, would you like me to explore an implementation using rabbitmq or something like that instead?

[blakeblackshear]

My understanding is that zeromq is incredibly performant. I have seen benchmarks showing that it's even faster to use zeromq to communicate between python processes than the native python multiprocessing queues. I also like that it seems to have a very small footprint. There may be other alternatives that I haven't considered, but those are my criteria. Small footprint (i am fond of go projects for this reason) and at least matches the performance of native python multiprocessing queues. I dont think rabbitmq will meet that criteria.

[blakeblackshear]

Closing this as we have started down this path already. In 0.15, we will take this quite a bit further.