No predictions from setfit

[simran-padam]

I ran SetFitTrainer to train the model with 5 classes (used multi_target_strategy="one-vs-rest"). My target variable was one hot encoded. When I get my predictions on the test data, I observe for most of the rows (close to 40% of test data), I have no class predictions meaning it is not predicting any class.

Has any experienced this issue before and can provide any suggestions.

Thanks

[kgourgou]

Hello!

That's interesting. Do you have any small example (e.g., with toy sklearn data) that reproduces the issue or do you think it's specific to your data?

[simran-padam]

Hii! I think it is specific to my data. I ran few examples earlier on toy data and saw no issues there. Doesn't Setfit uses softmax on the top layer, it is strange it did not predict any class.

[kgourgou]

Yeah, very weird. May I ask which version of SetFit you are using? Is it the latest?

Also, what does the loss look like during training? Is that also weird?

[leavoigt]

Hi Simran, I experienced a similar issue using setfit. Did you figure out what the problem is?

[kgourgou]

Doesn't Setfit uses softmax on the top layer, it is strange it did not predict any class.

Missed this question. Vanilla SetFit uses a scikit-learn LogisticRegression model on top of the sentence-transformer embeddings.