Slow processing times - can it run faster?

[TylerStewart1998]

Question

Slow processing times on a complex context with one intent containing 8 entities with hundreds of different context structures. Can it run faster and more reliable than the current processing time range of 0.01s and 0.8s ?

Intent 1 has 8 variable entities each with thousands some millions of possible permutations.

A small dataset (a few thousand samples) was trained and slot extraction is perfect. Processing time is very slow - between 0.01s and 0.8s per inference.

With more data, should processing times decrease?

In theory, if the dataset contained all possible permutations/variations/samples would this maximise inference speed?

Thanks

[adrienball]

Intent 1 has 8 variable entities each with thousands some millions of possible permutations.

It's likely that what takes most time in the processing is just the lookup of entity values in the input. One million possible values per entity (x 8 entities) ends up being a lot of data, I'm not that surprised about the processing time. Any way you can reduce this number of values, or split the intent into several ones, each of them containing a subset of the entities ?

[TylerStewart1998]

The intents could be split. doing so could require ongoing work/maintenance.

Aiming for a more general model.

What interests me with current processing times, is how test samples process in such diverse time ranges. Look at the context from one sample in the low range and you would think it processes in the same time as a sample from the higher ranges.

I thought maybe the samples in the high processing time ranges didn't have enough training data so inference was slower. Adding more data would help? But processing times only seem to increase.