Enhance OTEL testing to capture and verify Cancellation Requests and Non-Decoupled model inference.

[indrajit96]

Added tests for

1. For request cancellations, make sure spans are ended properly in various scenarios
2. Verify tracing behavior, after this PR: https://github.com/triton-inference-server/server/pull/6017

Tests added

1. test_non_decoupled() : Verify tracing for a non-decoupled inference
2. test_grpc_trace_all_input_required_model_cancel() : Verify trace after an inference request is cancelled in COMPUTE Phase
3. test_grpc_trace_model_cancel_in_queue() : Verify trace after an inference request is cancelled in QUEUE before COMPUTE Phase

[oandreeva-nv]

Could you please add a description, clarifying what tests were added?

[oandreeva-nv]

Could you please attach a picture of a trace as displayed in jaeger with cancelled request.

Another question, have you considered cases when request was cancelled, when it was in a queue and when it was already in a compute stage?

[indrajit96]

Could you please attach a picture of a trace as displayed in jaeger with cancelled request.

Another question, have you considered cases when request was cancelled, when it was in a queue and when it was already in a compute stage?

Could you please attach a picture of a trace as displayed in jaeger with cancelled request.

Another question, have you considered cases when request was cancelled, when it was in a queue and when it was already in a compute stage?

Fixed this, added a model which waits for a delay before executing hence cancellation request is recieved before the execution starts

[rmccorm4]

Can you update the PR title to be more descriptive? (cancellation, decoupled responses, etc. rather than JIRA ticket number)

[oandreeva-nv]

I think it's in a good state. Let's address Ryan's comment and provide a clear description for 3 tests, added in this PR, to the PR description