[QST] How to implement a model for binary classification task?

[brictobrick]

❓ Questions & Help

Details

Thanks for making this great library! I'd like to implement a model for binary classification task with Transformers4Rec, but I encountered some errors. Could you please tell me how to solve this problem, or provide tutorials for binary classification task? I will be grateful for any help you can provide.

First, I appended a target column 'is_churn' to dataset (1: churn, 0: retention), and set 'target_name' as below. Is this the correct way for binary classification task? I'm confused what I have to do with schema.pb for target column.

head = tr.Head(
    body,
    tr.BinaryClassificationTask(target_name='is_churn')
)

Secound, I used the same code for training and evaluation used in this tutorial 03-Session-based-recsys.ipynb. However, I got 'IndexError: too many indices for tensor of dimension 1'. What should I fix for evaluation process?

%%time
start_time_window_index = 1
final_time_window_index = 4
for time_index in range(start_time_window_index, final_time_window_index):
    # Set data 
    time_index_train = time_index
    time_index_eval = time_index + 1
    train_paths = glob.glob(os.path.join(OUTPUT_DIR, f"{time_index_train}/train.parquet"))
    eval_paths = glob.glob(os.path.join(OUTPUT_DIR, f"{time_index_eval}/valid.parquet"))

    # Train on day related to time_index 
    print('*'*20)
    print("Launch training for day %s are:" %time_index)
    print('*'*20 + '\n')
    trainer.train_dataset_or_path = train_paths
    trainer.reset_lr_scheduler()
    trainer.train()
    trainer.state.global_step +=1

    # Evaluate on the following day
    trainer.eval_dataset_or_path = eval_paths
    train_metrics = trainer.evaluate(metric_key_prefix='eval')
    print('*'*20)
    print("Eval results for day %s are:\t" %time_index_eval)
    print('\n' + '*'*20 + '\n')
    for key in sorted(train_metrics.keys()):
        print(" %s = %s" % (key, str(train_metrics[key]))) 
    wipe_memory()

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
File <timed exec>:21, in <module>

File /opt/conda/lib/python3.8/site-packages/transformers/trainer.py:2113, in Trainer.evaluate(self, eval_dataset, ignore_keys, metric_key_prefix)
   2110 start_time = time.time()
   2112 eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
-> 2113 output = eval_loop(
   2114     eval_dataloader,
   2115     description="Evaluation",
   2116     # No point gathering the predictions if there are no metrics, otherwise we defer to
   2117     # self.args.prediction_loss_only
   2118     prediction_loss_only=True if self.compute_metrics is None else None,
   2119     ignore_keys=ignore_keys,
   2120     metric_key_prefix=metric_key_prefix,
   2121 )
   2123 total_batch_size = self.args.eval_batch_size * self.args.world_size
   2124 output.metrics.update(
   2125     speed_metrics(
   2126         metric_key_prefix,
   (...)
   2130     )
   2131 )

File /transformers4rec/transformers4rec/torch/trainer.py:412, in Trainer.evaluation_loop(self, dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix)
    405 if (
    406     metric_key_prefix == "train"
    407     and self.args.eval_steps_on_train_set > 0
    408     and step + 1 > self.args.eval_steps_on_train_set
    409 ):
    410     break
--> 412 loss, preds, labels, outputs = self.prediction_step(
    413     model, inputs, prediction_loss_only, ignore_keys=ignore_keys
    414 )
    416 # Updates metrics
    417 # TODO: compute metrics each N eval_steps to speedup evaluation
    418 metrics_results_detailed = None

File /transformers4rec/transformers4rec/torch/trainer.py:299, in Trainer.prediction_step(self, model, inputs, prediction_loss_only, ignore_keys)
    296     else:
    297         outputs = model(inputs, training=False)
--> 299     loss = outputs["loss"].mean().detach()
    301 if prediction_loss_only:
    302     return (loss, None, None, None)

IndexError: too many indices for tensor of dimension 1

[rnyak]

@brictobrick you can refer to this issue to see an example of binary classification task: https://github.com/NVIDIA-Merlin/Transformers4Rec/issues/405

For the error IndexError: too many indices for tensor of dimension 1 . Are you using your own dataset? If yes, how are you creating the schema file? what's your preprocessing pipeline? we need a reproducible of your process to be able to repro the error. You can add your code in a gist with a simple fake dataset. thanks.

[jacobdineen]

@rnyak - poster of #405 here. I think that this issue (I had the same question but closed) is due to trying to use the overloaded trainer class for binary classification, which is currently only supported via a different pipeline as @sararb discussed - It has to be a call to the keras style model.fit(dataloader) rather than trainer.train(dataloader)

[rnyak]

@brictobrick is the conversation above solving your issue? if yes, can we close this? thanks.