Implement new conditional/optional Non Maximum Processor Pad Probe Handler

[rjhowell44]

This will be an optional build component - similar to the WebRtcSink. The component will be excluded by default.

Steps to include in DSL

• clone the NumCpp repository git clone https://github.com/dpilger26/NumCpp.git
• set the makefile include variable to true - BUILD_NMS_PPH:=true
• set the makefile path variable to the NumCpp root directory - NUM_CPP_PATH:=<path-to-numcpp>

New symbolic constants

/**
 * @brief Non Maximim Processor (NMP) process methods
 */
#define DSL_NMP_PROCESS_METHOD_SURPRESS                             0
#define DSL_NMP_PROCESS_METHOD_MERGE                                1

/**
 * @brief Non Maximim Processor (NMP) object match determination methods
 */
#define DSL_NMP_MATCH_METHOD_IOU                                    0
#define DSL_NMP_MATCH_METHOD_IOS                                    1

New conditional API

/**
 * @brief Creates a new, uniquely named Non-Maximum Processor (NMP) Pad 
 * Probe Handler (PPH) component.
 * @param[in] name unique name for the new Pad Probe Handler.
 * @param[in] label_file absolute or relative path to inference model label file.
 * Set "label_file" to NULL to perform class agnostic NMP.
 * @param[in] process_method method of processing non-maximum predictions. One
 * of DSL_NMP_PROCESS_METHOD_SUPRESS or DSL_NMP_PROCESS_METHOD_MERGE. 
 * @param[in] match_method method for object match determination, either 
 * DSL_NMP_MATCH_METHOD_IOU or DSL_NMP_MATCH_METHOD_IOS.
 * @param[in] match_threshold threshold for object match determination.
 * @return DSL_RESULT_SUCCESS on success, DSL_RESULT_PPH_RESULT otherwise.
 */
DslReturnType dsl_pph_nmp_new(const wchar_t* name, const wchar_t* label_file,
    uint process_method, uint match_method, float match_threshold);

/**
 * @brief Gets the current inference model label file in use by the Non-Maximum 
 * Processor (NMP) Pad Probe Handler component.  
 * @param[in] name unique name of the Pad Probe Handler to query.
 * @param[out] label_file path to the inference model label file in use. NULL
 * indicates class agnostic NMP. 
 * @return DSL_RESULT_SUCCESS on success, DSL_RESULT_PPH_RESULT otherwise.
 */
DslReturnType dsl_pph_nmp_label_file_get(const wchar_t* name, 
     const wchar_t** label_file);

/**
 * @brief Sets the inference model label file for the Non-Maximum 
 * Processor (NMP) Pad Probe Handler component to use.  
 * @param[in] name unique name of the Pad Probe Handler to update.
 * @param[in] label_file absolute or relative path to the inference model 
 * label file to use. Set "label_file" to NULL to perform class agnostic NMP.
 * @return DSL_RESULT_SUCCESS on success, DSL_RESULT_PPH_RESULT otherwise.
 */
DslReturnType dsl_pph_nmp_label_file_set(const wchar_t* name, 
     const wchar_t* label_file);

/**
 * @brief Gets the current process mode in use by the Non-Maximum 
 * Processor (NMP) Pad Probe Handler component.  
 * @param[in] name unique name of the Pad Probe Handler to query.
 * @param[out] process_method current method of processing non-maximum predictions. 
 * One of DSL_NMP_PROCESS_METHOD_SUPRESS or DSL_NMP_PROCESS_METHOD_MERGE. 
 * @return DSL_RESULT_SUCCESS on success, DSL_RESULT_PPH_RESULT otherwise.
 */
DslReturnType dsl_pph_nmp_process_method_get(const wchar_t* name, 
     const uint* process_mode);

/**
 * @brief Sets the process mode for the Non-Maximum Processor (NMP) 
 * Pad Probe Handler component to use.  
 * @param[in] name unique name of the Pad Probe Handler to update.
 * @param[in] process_method new method of processing non-maximum predictions. 
 * One of DSL_NMP_PROCESS_METHOD_SUPRESS or DSL_NMP_PROCESS_METHOD_MERGE. 
 * @return DSL_RESULT_SUCCESS on success, DSL_RESULT_PPH_RESULT otherwise.
 */
DslReturnType dsl_pph_nmp_process_method_set(const wchar_t* name, 
     uint process_mode);

/**
 * @brief Gets the current match settings in use by the named Non-Maximum 
 * Processor (NMP) Pad Probe Handler component.
 * @param[in] name unique name of the Pad Probe Handler to query.
 * @param[out] match_method current method of object match determination, 
 * either DSL_NMP_MATCH_METHOD_IOU or DSL_NMP_MATCH_METHOD_IOS.
 * @param[out] match_threshold current threshold for object match determination
 * currently in use.
 * @return DSL_RESULT_SUCCESS on success, DSL_RESULT_PPH_RESULT otherwise.
 */
DslReturnType dsl_pph_nmp_match_settings_get(const wchar_t* name,
    uint* match_method, float* match_threshold);

/**
 * @brief Sets the match settings for the named Non-Maximum Processor (NMP)
 * Pad Probe Handler component to use.
 * @param[in] name unique name of the Pad Probe Handler to update.
 * @param[in] match_method new method for object match determination, either 
 * DSL_NMP_MATCH_METHOD_IOU or DSL_NMP_MATCH_METHOD_IOS.
 * @param[in] match_threshold new threshold for object match determination.
 * @return DSL_RESULT_SUCCESS on success, DSL_RESULT_PPH_RESULT otherwise.
 */
DslReturnType dsl_pph_nmp_match_settings_set(const wchar_t* name,
    uint match_method, float match_threshold);

[rjhowell44]

@youngjae-avikus I've pushed my preliminary work to a new branch nms. The updated cpp example is at non_maximum_suppression.cpp.

I've gotten as far as building you code into the new NMS-PPH and I'm able to execute the above example. See your code added here. https://github.com/prominenceai/deepstream-services-library/blob/nms_pph/src/nms/DslPadProbeHandlerNms.cpp - just look for

  GstPadProbeReturn NmsPadProbeHandler::HandlePadData(GstPadProbeInfo* pInfo)

You will need to update the make file as discussed. Mine looks like

# To enable the Non Maximum Suppression (NMS) Pad Probe Handler (PPH)
# - set BUILD_NMS_PPH:=true and NUM_CPP_PATH:=<path-to-numcpp-include-folder>
BUILD_NMS_PPH:=true
NUM_CPP_PATH:=../NumCpp/include

The next steps will be to break up your code a little so that I can add some more unit testing... but I'm very positive on where this could go. Let me know what you think.

[rjhowell44]

@youngjae-avikus the above API definition has bee refactored to support both NMS and NMM

[YoungjaeDev]

@rjhowell44

Did you commit it?

[YoungjaeDev]

Oh, and can you put the class agostic option as a constructor parameter? Depending on the class agostic option, user will decide whether to make num_labels 1 or existing num_labels

[YoungjaeDev]

In summary, nms and nmm are all post-processing processes related to bbox The nms is already in nvinfer, but the reason I implemented them separately is to flexibly add class agnostic options. Conventional nms divide classes and proceed with the suppression, but the class agnostic option divides all boxes into one class and suppresses them
And nmm is useful to combine the tiling results that have been performed through the prerpoc module. If the tiling result apply nms, the box will be strangely expressed, but if you do nmm, it will come out naturally because it is combined.

Thank you!

@rjhowell44

[YoungjaeDev]

The code structure for refactoring is so clean that it looks good. Thank you!

[rjhowell44]

@youngjae-avikus I've pushed new updates to the nms_pph branch. Still a work in progress, but the basic functionality is there. Still needs more refactoring so I can add more unit test coverage.

[YoungjaeDev]

I'm off vacation, too. I'll test it as soon as I can

[rjhowell44]

@youngjae-avikus new updates have been pushed. Just for your information API Tests are done. Unit Tests are still a work in progress.

[rjhowell44]

@youngjae-avikus you can run the test cases as follows.

API

./dsl-test-app.exe -s [pph-nmp-api]

Unit

./dsl-test-app.exe -s [NmsPph]

[YoungjaeDev]

@rjhowell44

I'm looking at the code right now. I think it would be good to add the class agostic option using param to the dsl_pph_nmp_new I mentioned before. If class agostic=true, this option will be treated as num_labels=1 even if num_labels>1

Oh, and can you put the class agostic option as a constructor parameter? Depending on the class agostic option, user will decide whether to make num_labels 1 or existing num_labels

Thank you!

[rjhowell44]

@youngjae-avikus currently, the class agnostic option is controlled by the label_file parameter. Just set the parameter to NULL for class agnostic operation.

From the API comments above

 * @param[in] label_file absolute or relative path to inference model label file.
 * Set "label_file" to NULL to perform class agnostic NMP.

[YoungjaeDev]

@rjhowell44

So, do you Implement that label_file is a parameter for parsing num_labels? The labels.txt used in osd is parsing what is in the infer config?

[rjhowell44]

@youngjae-avikus yes, for non agnostic operation, you use the same label_file as passed to the infer plugin via the config file

[YoungjaeDev]

Yes, I understand. Thank you.

[rjhowell44]

@youngjae-avikus ... although not complete, I've merged this work into the v0.26.alpha branch so that others can review/test. I have the API documented.... the introduction still needs work though.