How to weight loss function

[freecui]

I have an extreme imbalanced dataset for muticlass task . To handle the imbalance, I want to weight loss function

[kgourgou]

How extreme? Does it help to pass an option to the head model, i.e., LogisticRegression? For example, you can pass class_weight='balanced', I have found this to work well.

[freecui]

an extreme imbalanced dataset： for example total data is 30000. class A has 12000, class B has 1, class C has 2000, class D has 13 ...

[freecui]

I have tried class_weight='balanced', but the result is just 74.57%. If I don't use class_weight='balanced, the result is 80% the code: model_body=SentenceTransformer("uer/sbert-base-chinese-nli") model_head=LogisticRegression(class_weight="balanced") model = SetFitModel(model_body=model_body,model_head=model_head)

Create trainer

trainer = SetFitTrainer( model=model, train_dataset=train_ds, eval_dataset=test_ds, loss_class=CosineSimilarityLoss, batch_size=8, num_iterations=10, # Number of text pairs to generate for contrastive learni ng num_epochs=1# Number of epochs to use for contrastive learning, )

trainer.train()

[kgourgou]

Thank you!

I have tried class_weight='balanced', but the result is just 74.57%. If I don't use class_weight='balanced, the result is 80%

Yeah, I assume this is because of the very hard imbalance.

Could you also check the accuracy per class please?

Also, do you have a way to get more data for the poorest classes, e.g., augmentation, or additional sampling?

[freecui]

I will do what you said

[Ulipenitz]

Maybe usefull for people working on a multi-label problem: You can wrap LogisticRegression inside OneVsRestClassifier like this:

model = SetFitModel(
                    model_body=SentenceTransformer('all-MiniLM-L6-v2'), 
                    model_head=OneVsRestClassifier(LogisticRegression(class_weight="balanced")),
                    multi_target_strategy="one-vs-rest"
                    )

[Matthieu-Tinycoaching]

@freecui @kgourgou where to specify LogisticRegression(class_weight="balanced") during the hyperparameters optimization phase? :

# model specfic hyperparameters
def model_init(params):
    params = params or {}
    max_iter = params.get("max_iter", 100)
    solver = params.get("solver", "liblinear")
    model_id = params.get("model_id", "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
    model_params = {
        "head_params": {
            "max_iter": max_iter,
            "solver": solver,
        }
    }
    return SetFitModel.from_pretrained(model_id, **model_params)

[kgourgou]

@Matthieu-Tinycoaching I'm only guessing, but "head_params" looks like the place to try.

[Matthieu-Tinycoaching]

@kgourgou that's what I've done experimentally also. Didn't generate any error when launching, but don't know if this works...