Slow inference using TARSClassifier base model on GPU

[akshaykekuda]

The inference time during evaluation using dev/test set is very slow. While training happens very fast as expected the evaluation time is too large for GPU. Running the same experiments using TransformerDocumentEmbeddings model results in the expected lightning training and inference speed using the same datasets(I understand that the embedding model is much smaller than that of tars base)

Screenshots The below image is for training and evaluation using TransformerDocumentEmbeddings: The below image is for training and evaluation using tars-base model:

[helpmefindaname]

Hi @akshaykekuda how many different labels do you have? Is num_negative_labels_to_sample set to the default? How many positive labels does one sentence have on average?

The training will create a sample for each positive label plus num_negative_labels_to_sample per sample. The evaluation will do one prediction per sample and label, hence it could be way more.

[akshaykekuda]

I do have around 50 labels. Everything is set to default. The same dataset on a similar distil bert model finishes inference in 2 mins, while with tars base it takes 1 hour. Looks like something is off.

[helpmefindaname]

2 minutes * 50labels = 100minutes (1.66hrs). I suppose the 2 minutes are rounded up? But the numbers sound fine to me.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.