[Web] dynamic batch size doesn't work when use webgl provider

[MengjiaoH]

trafficstars

Describe the issue

Ask a Question

Question

I have converted a PyTorch model to ONNX with the following code to support the dynamic batch size.

    model_dir = "../models/hurricane/num_models_long_0_20_hurricane_2M_network2_2048_3_6/model_100.pth"
    batch_size = 1
    dim = 3
    model = Network2(dim, 3, 6, 2048)
    # Initialize model with the pretrained weights
    map_location = lambda storage, loc: storage
    if torch.cuda.is_available():
        print("map location none")
        map_location = None
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model.load_state_dict(torch.load(model_dir, map_location=map_location))
    model.to(device)
    model.eval()
    # Input to the model 
    points = torch.rand(batch_size, dim, requires_grad=True).to(device)
    t = torch.rand(batch_size, 1, requires_grad=True).to(device)
    torch_out = model(points, t)
    # Export the model
    torch.onnx.export(model,               # model being run
                     args=(points, t),                         # model input (or a tuple for multiple inputs)
                     f = "Hurricane1.onnx",   # where to save the model (can be a file or file-like object)
                     export_params=True,        # store the trained parameter weights inside the model file
                     verbose=False,
                     opset_version=10,          # the ONNX version to export the model to
                     do_constant_folding=True,  # whether to execute constant folding for optimization
                     input_names = ["input_1", "input_2"],   # the model's input names
                     output_names = ['output1'], # the model's output names
                     dynamic_axes={'input_1' : {0 : 'batch_size'},    # variable length axes
                                    'input_2' : {0 : 'batch_size'},
                                    'output1' : {0 : 'batch_size'}})

The model has two inputs: three-dimensional point locations and one-dimensional time steps. The input size is [batch_size, 3] for the point locations and [batch_size, 1] for the time steps. My developed app takes the number of points $n$ from the users and uses the number of points as the batch size for inference. Users can modify the number of points $n$ for new inference. During the inference, I converted the input data to the size of [n, 3] and [n, 1] and then fed them into the model.

    const data_1 = Float32Array.from(points); 
    const input_1 = new Tensor( "float32", data_1, [parseInt(num_seeds), 3]);
    const data_2 = Float32Array.from(times)
    const input_2 = new Tensor( "float32", data_2, [parseInt(num_seeds), 1]);
    const feeds = { input_1: input_1, input_2: input_2};
    const output= await model.run(feeds);

I warm up the session by running random input with the batch size 1000 [https://github.com/MengjiaoH/FlowMap_Web_Viewer/blob/3809fd45cd75476fb491f6e06ea2eab25f1d595c/src/LeftPanel/components/ModelLoader.js#L94].

It works well when I use wasm as the provider. I can change the number of seeds to make new inferences. However, if I use WebGL, it releases the error showing the input tensor size is incorrect. As the following screenshot shows, I first put 10 points and then 110 points. There was no error if the same batch sizes (number of points) were used. The batch size (number of seeds) has to be the same as the previous inference process.

I would like to ask if the ONNX Runtime Web with WebGL supports dynamic batch size. Have I done anything wrong? I appreciate any help you can provide. Let me know any other information I need to provide.

Thanks, Mengjiao

Further information

• Relevant Area:

• Is this issue related to a specific model?
  Model name:
  Model opset:

Notes

To reproduce

To reproduce the error, Step 1: Clone the github repo https://github.com/MengjiaoH/FlowMap_Web_Viewer. Step 2: Download the folder models and put it under folder public [https://drive.google.com/file/d/1IjHm6vBhEoBgFh-vh_HoYq1YSsqX8Me-/view?usp=sharing] Step 3: Run npm start to start a local server for the webpage. Step 4: Select the Hurricane model from the Model drop-down list. Step 5: Input 10 random seeds under Seed Placement, then click Add Seeds button. Step 6: Click the Start Tracing button under the Particle Tracing section.

Urgency

No response

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.12.1

Execution Provider

WebGL

[manish181192]

any plans to support this?

[rajkundu]

I have the same issue; +1!

[microsoft-github-policy-service[bot]]

Applying stale label due to no activity in 30 days