Make timestamp (and potentially other metadata) available for python3 tensor_filter?

[bveldhoen]

Hello,

It would be helpful if the gstreamer timestamp (and potentially other metadata) would be available for CustomFilter implementations used in python3 tensor_filter.

Is there a way to get the timestamp of (a) tensor(s) in the python3 implementation? Or are there any plans to add this functionality?

[taos-ci]

:octocat: cibot: Thank you for posting issue #4524. The person in charge will reply soon.

[myungjoo]

I do not have a plan to provide timestamp for filters becauase I didn't even know that it is needed.

Anyway, GSTBUF has timestamp information; thus, we will need to open up an interface to access GSTBUF for tensor_filter's subplugins and customfilters. Right now, we do not provide such interfaces and subplugins (including python3) get the striped input buffers (without anything related to metadata of GSTBUF).

For such capabilities, tensor_filter's APIs might need some rework. I'd like to suggest:

1. add a property "supply-metadata=1 | 0" to tensor_fitler
2. update the base class, "tensor_filter_subplugin" c++ class:

virtual void invoke (const GstTensorMemory *input, GstTensorMemory *output) = 0;

==>

virtual void invoke (const GstTensorMemory *input, GstTensorMemory *output) = 0;
virtual void invoke (const GstTensorMemory *input, GstTensorMemory *output, const GstTensorBufferMetadata *meta) {
  invoke(input, output);
}

and let invoke (input, output, meta) called instead of invoke (input, output) if supply-metadata = 1 (by updating tensor_filter_subplugin::cpp_invoke() ).

typedef struct {
  GstClockTime *pts;
  GstClockTime *dts;
  GstClockTime *duration;
  // May be extended in the future
  // Everything will be freed after invoke() is returned.
} meta; // suggestion only.

3. supporting tensor-filter subplugin will override invoke(input, output, meta) function. (i.e., python3).
4. Updating tensor_filter_subplugin::cpp_invoke() requires updating tensor_filter's main routine. Because cpp_invoke() (the invoke callback of tensor_filter) gets GstTensorMemory, not GstBuffer.
   • For easier implemention with minimum impact (but dirtier), we can expand GstTensorFilterProperties and let it hold pts/dts/duration at the time of calling invoke (tensor_filter.c's "step 3")
     • We may add entiers for "temporal values" in GstTensorFilterProperties for later similar cases
   • Or... update invoke infrastructure, including GST_TF_FW_INVOKE_COMPAT macro and all the related structures.

===

If you want to hack without updating the whole framework as above (but working in your local env only), you may let it include another dummy input tensor (but large enough to hold timestamp values: e.g., X:Y ==> X:Y,3) and fill the additional tensor with timestamp values with a hack in tensor_filter.c at "step 3. Call the filter-subplugin callback, "invoke" ".

[bveldhoen]

@myungjoo Thanks for your feedback, the proposed changes seem good.

I assume there will be a python class corresponding to GstTensorBufferMetadata, which will be made available in the python tensor_filter CustomFilter implementation(s)?

[jaeyun-jung]

suggestion : adding timestamp info into GstTensorFilterProperties invoke function of filter subplugin has GstTensorFilterProperties arg, so it is easy to pass information.