Automating Image and Payload Upload for ONNX Backend Inference - Image Shape and Data payload

[eanmikale]

Description We would like to automate the image and payload upload, however, we are having issues with the shape and image data payload in our JSON payload script.

Triton Information What version of Triton are you using?

Jetpack 4.6 2.17.0 for Client - Ubuntu 18 Jetpack 5.0.0 2.22.0 for Server Ubuntu 20 (Trying not to dial the server back to 2.17.0)

Are you using the Triton container or did you build it yourself? Built from Source

To Reproduce Steps to reproduce the behavior.

SCRIPT FOR PAYLOAD AUTOMATED CREATION

#!/usr/bin/env python3.7

import json
import requests

# install required packages before running
# pip install pillow numpy --upgrade
from PIL import Image
import numpy as np

# method to generate payload from image url
def generate_payload(image_url):
    # download image from url and resize
    image_inputs = Image.open(requests.get(image_url, stream=True).raw)
#    image_inputs = image_inputs.resize((200, 200))

    # convert image to numpy array
    image_tensor = np.asarray(image_inputs)
    # derive image shape
    image_shape = [1] + list(image_tensor.shape)

    # create payload request
    payload = {
        "name" : "densenet_onnx",
	"versions" : [1],
	"platform" : "onnxruntime_onnx", 
	"id": "0",
        "inputs": [
            {
                "name": "data_0",
                "shape": image_shape,
                 #"shape" : [3, 224, 224]
                "datatype": "FP32",
                "data": image_tensor.tolist()
            }
        ],
      "outputs" : [
    {
      "name" : "fc6_1",
      "datatype" : "FP32",                                   ###Question: Does the output and classification section go in the config file or payload? 
      "shape" : image_shape,
      "parameters" : { "classification" : 2 }
    }
  ]
}

    # save payload as json file
    payload_file = "/home/XXXXX/instances.json"
    with open(payload_file, "w") as f:
        json.dump(payload, f)
    print(f"Payload generated at {payload_file}")

    return payload_file

if __name__ == '__main__':
  image_url = "https://cdn3.volusion.com/rfnpv.kyqhb/v/vspfiles/photos/MUG12M-D-2.jpg?v-cache=1531401419"
  payload_file = generate_payload(image_url)

JSON PAYLOAD FILE w/ Shape Field Predetermined @ [3, 224, 224]

{"name": "densenet_onnx", "versions": [1], "platform": "onnxruntime_onnx", "id": "0", "inputs": [{"name": "data_0", "shape": [3, 224, 224], "datatype": "FP32", "data": [[[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255]...}

JSON PAYLOAD FILE w/ Shape Field Non-Predetermined

{"name": "densenet_onnx", "versions": [1], "platform": "onnxruntime_onnx", "id": "0", "inputs": [{"name": "data_0", "shape": [1, 576, 625, 3], "datatype": "FP32", "parameters": {}, "data": [[[255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255], [255, 255, 255]...}

OUTPUT 1

+ curl -v --http2 --ignore-content-length --tr-encoding -X POST -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: application/x-www-form-urlencoded' -H 'Authorization: Bearer KEY' -d @instances.json https://aidoctor.tunnelto.dev/v2/models/densenet_onnx/versions/1/infer
+ jq .
Note: Unnecessary use of -X or --request, POST is already inferred.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2a09:8280:1:48de:ca2b:6d39:9790:f0da...
* TCP_NODELAY set
* Connected to aidoctor.tunnelto.dev (2a09:8280:1:48de:ca2b:6d39:9790:f0da) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
100 5334k    0     0  100 5334k      0   333k  0:00:16  0:00:16 --:--:--  354k* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
< HTTP/2 400 
< content-type: application/json
< content-length: 114
< date: Thu, 07 Jul 2022 18:38:10 GMT
< server: Fly/9ece5bcd (2022-06-21)
< via: 2 fly.io
< fly-request-id: KEY-chi
< 
{ [114 bytes data]
100 5334k    0   114  100 5334k      6   317k  0:00:16  0:00:16 --:--:--  278k
* Connection #0 to host aidoctor.tunnelto.dev left intact
{
  "error": "unexpected shape for input 'data_0' for model 'densenet_onnx'. Expected [3,224,224], got [1,576,625,3]"
}

OUTPUT 2

+ curl -v --http2 --ignore-content-length --tr-encoding -X POST -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: application/x-www-form-urlencoded' -H 'Authorization: Bearer KEY' -d @instances.json https://aidoctor.tunnelto.dev/v2/models/densenet_onnx/versions/1/infer
+ jq .
Note: Unnecessary use of -X or --request, POST is already inferred.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2a09:8280:1:48de:ca2b:6d39:9790:f0da...
* TCP_NODELAY set
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to aidoctor.tunnelto.dev (2a09:8280:1:48de:ca2b:6d39:9790:f0da) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3824 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.tunnelto.dev
*  start date: May 27 11:42:29 2022 GMT
*  expire date: Aug 25 11:42:28 2022 GMT
*  subjectAltName: host "aidoctor.tunnelto.dev" matched cert's "*.tunnelto.dev"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
* Using Stream ID: 1 (easy handle 0x559f1299b0)
} [5 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
> POST /v2/models/densenet_onnx/versions/1/infer HTTP/2
> Host: aidoctor.tunnelto.dev
> User-Agent: curl/7.58.0
> Connection: TE
> TE: gzip
> Content-Type: application/x-www-form-urlencoded
> Accept: application/x-www-form-urlencoded
> Authorization: Bearer KEY
> Content-Length: 5462365
> 
} [5 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [81 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
...
* We are completely uploaded and fine
{ [5 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
< HTTP/2 400 
< content-type: application/json
< content-length: 83
< date: Thu, 07 Jul 2022 19:49:24 GMT
< server: Fly/9ece5bcd (2022-06-21)
< via: 2 fly.io
< fly-request-id: KEY-chi
< 
{ [83 bytes data]
100 5334k    0    83  100 5334k      3   217k  0:00:24  0:00:24 --:--:--  140k
* Connection #0 to host aidoctor.tunnelto.dev left intact
{
  "error": "Unable to parse 'data': Shape does not match true shape of 'data' field"
}

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

DENSENET_ONNX CONFIG FILE

name: "densenet_onnx"
platform: "onnxruntime_onnx"
max_batch_size : 0
input [
  {
    name: "data_0"
    data_type: TYPE_FP32
    format: FORMAT_NCHW
    dims: [ 3, 224, 224 ]
    reshape { shape: [ 1, 3, 224, 224 ] }
  }
]
output [
  {
    name: "fc6_1"
    data_type: TYPE_FP32
    dims: [ 1000 ]
    reshape { shape: [ 1, 1000, 1, 1 ] }
    label_filename: "densenet_labels.txt"
  }
]

Expected behavior A clear and concise description of what you expected to happen.

We are just looking to get the classes output using the classification extension. Thanks!

[rmccorm4]

Hi @eanmikale ,

From a glance at your code, it looks like maybe the actual image input data you are sending is not of shape [224,224,3] but is instead [1,576,625,3].

1. Since you set the request's input shape to [224,224,3] but the actual image shape is [1,576,625,3], I believe that's the reason for the error in "OUTPUT1".
2. And when you set the request's input shape to [1,576,625,3] manually when forming the request, this shape does not match the expected input shape in the config file [222,224,3], so I believe that's why you get the error in "OUTPUT2".

Seems like for the field "data": image_tensor.tolist(), the image_tensor.tolist() is not of correct shape [224,224,3]

Can you verify that you are correctly pre-processing the image to this expected input shape of the model ([224,224,3] in this case)? You can see a similar example of this pre-processing in our image_client.py example.

[krishung5]

Closing issue due to lack of activity. Please re-open the issue if you would like to follow up with this.