metrics are same(F1_score, acc, recall, precision)

[mjiangtao]

When I set the config "model_config = FTTransformerConfig( task="classification", metrics=["f1_score", "accuracy","recall","auroc"], # embedding_initialization=None, embedding_bias=True, share_embedding = True, share_embedding_strategy="fraction", shared_embedding_fraction=0.25, metrics_params=[{"num_classes": num_classes, "average": "macro"}, {},{"num_classes": num_classes, "average": "macro"},{}], metrics_prob_input=[False,False,False,True] ) trainer_config = TrainerConfig(auto_select_gpus=True, fast_dev_run=False, min_epochs=10 ,max_epochs=20, batch_size=512) experiment_config = ExperimentConfig( project_name="GATE Dev", run_name="gate_w_t_softmax_w_init", exp_watch="gradients", log_target="wandb", log_logits=True,

) optimizer_config = OptimizerConfig()

tabular_model = TabularModel( data_config=data_config, model_config=model_config, optimizer_config=optimizer_config, trainer_config=trainer_config, experiment_config=experiment_config,

) cust_loss = get_class_weighted_cross_entropy(train[target_name].values.ravel()) tabular_model.fit( train=train, validation=test, loss=cust_loss, # train_sampler=sampler, )

预测与结果保存

result = tabular_model.evaluate(test) print(result) test.drop(columns=target_name, inplace=True) pred_df = tabular_model.predict(test) print(pred_df.head()) os.makedirs('output-ft', exist_ok=True) pred_df.to_csv("output-ft/temp2.csv")" The results of metrics are same "Epoch 9/19 ----------------- 17/17 0:00:44 • 0:00:00 0.40it/s v_num: 8akb
train_loss: 0.217 valid_loss: 0.377 valid_f1_score:
0.827
valid_accuracy:
0.827
valid_recall:
0.827
valid_auroc:
0.788
train_f1_score:
0.899
train_accuracy:
0.899
train_recall:
0.899
train_auroc:
0.961
2025-04-29 11:00:17,036 - {pytorch_tabular.tabular_model:696} - INFO - Training the model completed
2025-04-29 11:00:17,044 - {pytorch_tabular.tabular_model:1537} - INFO - Loading the best model
D:\mjt\pytorch_tabular\src\pytorch_tabular\utils\python_utils.py:85: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature. return torch.load(f, map_location=map_location) C:\Users\User.conda\envs\tabr\lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py:424: The 'test_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argumenttonum_workers=19in theDataLoader` to improve performance. ┌───────────────────────────┬───────────────────────────┐ │ Test metric │ DataLoader 0 │ ├───────────────────────────┼───────────────────────────┤ │ test_accuracy │ 0.8725489974021912 │ │ test_auroc │ 0.8189974427223206 │ │ test_f1_score │ 0.8725489974021912 │ │ test_loss │ 0.2777501940727234 │ │ test_loss_0 │ 0.2777501940727234 │ │ test_recall │ 0.8725489974021912 │ └───────────────────────────┴───────────────────────────┘ Testing -------------------------------------- 3/3 0:00:01 • 0:00:00 2.79it/s [{'test_loss_0': 0.2777501940727234, 'test_loss': 0.2777501940727234, 'test_f1_score': 0.8725489974021912, 'test_accuracy': 0.8725489974021912, 'test_recall': 0.8725489974021912, 'test_auroc': 0.8189974427223206}] ",I can't find the progress of metrics evaluation, I don;t konw how to solve the problem .

[dosubot[bot]]

The issue you're experiencing, where multiple metrics like f1_score, accuracy, and recall have the same values, could be due to the configuration of the metrics_prob_input parameter in your FTTransformerConfig.

In your configuration, you have set metrics_prob_input to [False, False, False, True]. This means that the f1_score, accuracy, and recall metrics are being calculated using class inputs rather than probabilities. This can lead to these metrics having the same values, especially if the class predictions are consistent across these metrics.

To address this, consider the following:

1. Check Metrics Input Type: Ensure that the metrics_prob_input is correctly set for each metric. For metrics like f1_score and recall, which can be sensitive to the input type, you might want to try setting the corresponding entries in metrics_prob_input to True if they require probability inputs for more nuanced evaluation.

2. Review Metrics Parameters: Verify that the metrics_params are correctly configured for each metric. For example, ensure that the average parameter is set appropriately for your classification task (e.g., "macro" for multi-class).

3. Custom Loss Function: Ensure that your custom loss function, get_class_weighted_cross_entropy, is not affecting the output in a way that leads to uniform predictions. This could indirectly cause the metrics to be similar.

By adjusting these configurations, you may be able to achieve more distinct metric evaluations.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.