Ian Benlolo

I replaced that line with ``` if isinstance(examples[0], (list, tuple, np.ndarray)): examples = [torch.tensor(e, dtype=torch.long) for e in examples] ``` and it does what (i think) is supposed to happen

+1

Loading the model for me looks something like ``` from transformers.modeling_utils import load_sharded_checkpoint tab_net = TabFormerBertLM(custom_special_tokens, vocab=vocab, field_ce=args.field_ce, flatten=args.flatten, ncols=dataset.ncols, field_hidden_size=args.field_hs ) load_sharded_checkpoint(tab_net.model, base_path+"checkpoints1/checkpoint-80/") ``` I need that i guess...

+1

I'd like an update on this as well. Some kind of data preprocessing pipeline on each fold would be great.

I would also like to add a `collate_fn` but am creating a torch dataloader with `SparkDatasetConverter.make_torch_dataloader`. the work around i am using is doing my collating in the `TransformSpec` but...

Yes this is! How would this differ to TransformSpec though? Would the input also be a dataframe or would it be a torch Tensor? Thank you for getting back to...

Interesting. I wanted to use the `converter.make_torch_dataloader` so what i ended up with was just this `TransformSpec` func: ``` def transform_row(pd_batch): """ The input and output of this function must...

Hm so it turns out what i wanted to do (pass a pad_collate function) is in fact possible with the `data_loader_fn` argument like so: ``` from petastorm.pytorch import DataLoader data_loader_fn=...

> to give somebody a hint based on my own experiments, for image only argumentations. Key to use custom transforms is based on it's interface https://github.com/albumentations-team/albumentations/blob/master/albumentations/core/transforms_interface.py > > see following...