Hemant Jain

icon indicating a website link www.linkedin.com/in/hemantjain1 icon indicating an email [email protected]

icon company @cohere-ai icon location Engineering Manager, Finetuning @cohere-ai | ex-@NVIDIA | University of Washington

Results 15 comments of Hemant Jain

Adding DtoD support to CUDA shared memory feature

@rarzumanyan on second thought your arguments for not contaminating the namespace / enum are valid. You can just make the new changes I suggested for handling the input_values field specially...

Pull repository from different s3 accounts (multiple credentials)

@aras7 Here is a proposal on how we intend to support multiple profiles. Please share your feedback on the same: Use the following format to specify the _profile_ argument.  **s3//[email protected]:80/bucket-name/path/to/model**...

Python InferenceServerClient (http) should not call close() from __del__

@damonmaria thank you for filing this issue. Can you provide a detailed repro with a minimal example so that we can add additional handling for such cases?

Python InferenceServerClient (http) should not call close() from __del__

@jbkyang-nvi can you have a look at this?

tensorrt slower than onnx

@naor2013 do you mean you used the TensorRT backend or the TensorRT optimization option for OnnxRuntime backend? Can you share the model and perf analyzer configuration you used?

tensorrt slower than onnx

@naor2013 thank you for sharing the same. Can you also share the logs from perf analyzer that show the throughout and detailed breakdown of latency?

tensorrt slower than onnx

@naor2013 thank you for sharing all the information. We will investigate this once we have the bandwidth to do so.

Remove nvidia-ml deps for multi_server

cc @nv-kmcgill53 can you try to build a version of 22.02 on Jetson with this branch instead of r22.02?

Is python backend going to support asyncio?

@ZhuYuJin we have made a note of your request for adding such a feature.

Option for adding or overriding model config attributes at server startup

@ashrafguitoni (and other users requesting this feature) are you looking to only modify the instance group via this functionality? if so Triton would need to re-work the model control workflow...