Request the feature to send metadata in BLS

[ikhyun5538]

Is your feature request related to a problem? Please describe. I want to change the Model according to the BLS input. But BLS requires the same input bach size. So I have to make input like ["model1", "model1"]. If my batch size is 100, should I send 100 same data like ["model1", "model1", "model1"...].

Describe the solution you'd like It would be nice to have the feature to send metadata other than batch data.

[dyastremsky]

Thanks for submitting your request. I've filed a ticket to look into this enhancement.

[ikhyun5538]

Thank you for consideration.

[Tabrizian]

@ikhyun5538 Can you clarify more on the feature request? You can create a larger batch by providing more elements in the first batch of your model. For example, if the model accepts variable batches in the first dimension, you can send a batch of size [2, 1] instead of [1,1].

[ikhyun5538]

@ikhyun5538 Can you clarify more on the feature request? You can create a larger batch by providing more elements in the first batch of your model. For example, if the model accepts variable batches in the first dimension, you can send a batch of size [2, 1] instead of [1,1].

I'm sorry for the unclear question.

I making TTS(Text To Speech) pipeline using BLS. My pipeline consists of 3 models [1. Preprocessor] -> [2.Spec.Generator] -> [3. Vocoder]. But [2.Spec.Genenrator] has various checkpoints(onnx) versions depending on speaker.

The reason why I'm using BLS is that I want to change [2.Spec.Generator] checkpoint according to input speaker value. (If I used ensemble, too many ensembles are created depending on the speaker) So I configured BLS as follows. However, since the speaker information has to be delivered in the form of batch data, same speaker name must be duplicated like ["Ikhyun", "Ikhyun"] (batch size 2 as bellow). I think this limits the flexibility of BLS and makes response time slower.

BLS input config

max_batch_size:20
input [
  {
    name: "CHARS_INPUT"
    data_type: TYPE_STRING
    dims: [1]
  },
  {
    name: "SPEAKER_INPUT"
    data_type: TYPE_STRING
    dims: [1]
  }
]
...

BLS input Example

{
  "inputs": [
    {
      "name": "CHARS_INPUT",
      "shape": [2, 1],
      "datatype": "BYTES",
      "data": ["Hi.", "Hello"]
    },
    {
      "name": "SPEAKER_INPUT",
      "shape": [2,1],
      "datatype": "BYTES",
      "data": ["Ikhyun", "Ikhyun"]
    }
  ]
}

Does your answer mean to send various data as object type like as follow?

max_batch_size:20
input [
  {
    name: "ANY_INPUT"
    data_type: TYPE_STRING
    dims: [1]
  }
...

I think It will work, but there seems to be a problem that it is hard to know the input information through the config.

For this reason, It would be nice to have the feature to send metadata other than batch data.

[Tabrizian]

Sorry for the delay in responding to this thread. I'm still not completely clear on the feature request. Why the additional metadata that you want to send cannot be a separate input?

[Tabrizian]

Closing due to in-activity.