FlowFuse
Support for Real-Time Streaming Protocol
As a FlowFuse user with RTSP camera feeds I want to seamlessly ingest RTSP streams into video analytics tools and data stores So that I can easily use Node-RED in FlowFuse to collect and transmit RTSP data
Description
Enable support for RTSP ingestion in video analytics. The prospect uses RTSP camera feeds and wants to connect these to Node-RED, do some data collection and transformation, and send them to an analytics tool for functions like object detection.
This will likely involve the creation of a Certified Node. To make the process very easy for the user, the node would need to ask for a server address (where the video stream is), a username and password, and then output. In Rob's demo, the node would likely appear where the 'grab a frame' exec node is, and replaces FFMPEG node.
Context
This is a prospect request, presented at 33:58 https://fathom.video/calls/364063251
Summary of request from Fathom: Based on the meeting transcript, COMPANY is looking to integrate RTSP (Real-Time Streaming Protocol) video feeds into their IoT solutions. Specifically:
PROSPECT mentioned that a big portion of COMPANY'S projects involve video analytics, and RTSP integration is a "deal breaker" for them. They want to be able to ingest RTSP video streams and process them, potentially using containerized applications. COMPANY is looking for FlowFuse to provide enterprise-level support and integration for RTSP, as it is a critical requirement for their use cases.
- They want to be able to ingest RTSP video streams from cameras into the FlowFuse platform.
- They want to then route those RTSP streams to separate containers/applications that can perform video analytics processing.
- A specific example use case: take an RTSP stream, process it through FlowFuse, and then send it to a container that can perform object recognition (e.g. detect if a person is present).
- The core need is for RTSP integration and the ability to feed the video streams to custom analytics.
The POV criteria may imply that the data volume is significant, with one line mentioning 10M records/sec. Document Node-RED's ability to handle video ingestion.
###Acceptability Requirements
- [ ] A new node is created that ingests RTSP video streams
- [ ] The node enables processing of data in video analytics tools
Epic/Story
No response
Have you provided an initial effort estimate for this issue?
I have provided an initial effort estimate
As I recall, they need to be able to extract X frames per second from an RTSP server. I don't think X has been defined yet. Once they capture the still image (frame) they will want to be able to send that data elsewhere, I suspect to a file system or API. I built a prototype of this a couple of months ago which I can demo to anyone if that's useful. It's very much a PoC but they seemed happy when I showed them. You can see the demo here (skip to 10:20) - https://fathom.video/calls/372933715
@Steve-Mcl and @knolleary , I understand from conversation with @robmarcer that you may have some additional context about our ability to support RTSP ingestion, and that Steve worked on one of the nodes that provided for video. Please weigh in on this issue.
@ZJvandeWeg You've mentioned that a customer has a node/nodes that provide RTSP. Can you fill in details of that here so we have a direction on where we are taking this and what may already be available?
ZJ has clarified that one of our customers (the largest one) has nodes that take streams from Lucid cameras – not exactly the same as RTSP. Still, it could be a starting point, and we can ask them for code.
@Steve-Mcl just following up: do you have additional context about our ability to support RTSP ingestion?
Following a conversation with @robmarcer:
- Need to sketch out what this will and won't do, and surface that to the customer
Basic function: this is really a series of photos in a specific order, and delivers some number of photos per second.
Have provided documentation to share with the prospect. Will get their feedback the week of October 13 and provide next steps.
Update Oct 13: After a meeting with the prospect, @robmarcer has revised the spec to support ingestion at 60fps and will test that the proposed flow meets this requirement, and then relay the results.
From experience, 60fps will depend a lot on what they want to do with the individual frames. A node that can convert an RTSP stream into 60 messages/per/sec is feasible. A flow that can sustain processing of those images at 60 fromes/per/sec will depend entirely on what that flow is doing.
@knolleary would it be useful for me to benchmark using FFMPEG to see how high (FPS) my PI5 will go? I can test outputting the files to disk, an api, database etc. I've already got the flows from the demo I did here - https://fathom.video/calls/372933715
@robmarcer yes - 60fps is going to be a big stretch - will also depend a lot on what resolution the stream is at. Will this be running in cloud or on device?
How about: make a very simple flow where ingestion runs smoothly at 60fps. The flow terminates with a project node, validating that 60fps is possible, and what comes after is up to the customer?
How about: make a very simple flow where ingestion runs smoothly at 60fps. The flow terminates with a project node, validating that 60fps is possible, and what comes after is up to the customer?
Err, who said anything about Project Nodes (this is the first time it has been mentioned)?
Using MQTT to stream video really isn't a good use case.
This also implies that a device agent (or more than one) ingesting video and sending it to a cloud/hosted instance that then needs to do the processing (as nothing else can consume the Project Nodes messages). This also has implications for the MQTT broker, we need to know how many of these 60hz message streams it needs to handle.
Before we can comment on this at all we need a full description of the problem we are actually trying to solve.
There may be a better solution here. e.g. At home I run a system call Frigate
Frigate uses RTSP to ingest video feeds from a webcam and runs Machine learning models to identify things in the video (e.g. people, cars...) it then publishes MQTT messages with events based on what it has seen. These events are consumed by Node-RED to trigger things (and can either include the images in the events, or a URL to download them)
Following up chat with Rob: how about we propose 1080p, 60fps, and tell them it runs in FFC. Our goal should be to identify the minimal (reasonable) parameters needed to hit their 60fps goal, then I can add it to the spec sheet and Kasheef can forward to the prospect.
@knolleary @hardillb
@gstout52 NO, we can't just say that
Before we can comment on this at all we need a full description of the problem we are actually trying to solve.
We can't do anthing without ^
Following up on conversation with @hardillb: we need to understand the architecture that the prospect is trying to fit this into. There are much better and much worse ways of delivering this. I will connect with @Kasheef13 to get this over the line.
Just a follow up, 1080p would also be a lot larger than I would usually expect fed to a machine learning model (classifier), they tend to consume much lower resolution images, rather than the raw feed.
Update from October 29 meeting with the prospect. Prospect will send specs on the cameras they are using, to understand what exactly they want to consume, at what size and what frame rate, so we can ensure we can deliver what they want.