tensor.elbow_rank_selection - input must be Tensor not numpy.ndarray error for InteractionTensor made with how='outer'

[tomthomas3000]

Hi cell2cell team,

Fantastic work on the package - it is really addressing a missing gap in current cell-cell interaction tools! I had an issue with running tensorcell2cell and was wondering whether you may have encountered it previously. The problem is with the tensor.elbow_rank_selection function. I am running it on the GPU. I have run the PBMC data in the vignette through the conda environment i created, and it worked as expected.

Furthermore, this problem is only limited to tensors made with, how='outer' in the c2c.tensor.InteractionTensor function. If how='inner', no such problems exist.

The error is as follows:

Traceback (most recent call last): File "/gpfs2/well/dendrou/shared/loom/TAURUS/TAURUS_analyses/cell2cell/run_cell2cell.py", line 84, in <module> fig, error = tensor.elbow_rank_selection(upper_rank=25, File "/gpfs2/well/dendrou/shared/loom/TAURUS/TAURUS_analyses/cell2cell/code/env/lib/python3.9/site-packages/cell2cell/tensor/tensor.py", line 333, in elbow_rank_selection all_loss = _multiple_runs_elbow_analysis(tensor=self.tensor, File "/gpfs2/well/dendrou/shared/loom/TAURUS/TAURUS_analyses/cell2cell/code/env/lib/python3.9/site-packages/cell2cell/tensor/factorization.py", line 218, in _multiple_runs_elbow_analysis tl_object, errors = _compute_tensor_factorization(tensor=tensor, File "/gpfs2/well/dendrou/shared/loom/TAURUS/TAURUS_analyses/cell2cell/code/env/lib/python3.9/site-packages/cell2cell/tensor/factorization.py", line 58, in _compute_tensor_factorization cp_tf = non_negative_parafac(tensor=tensor, File "/gpfs2/well/dendrou/shared/loom/TAURUS/TAURUS_analyses/cell2cell/code/env/lib/python3.9/site-packages/cell2cell/external/tensorly_nn_cp.py", line 178, in non_negative_parafac tensor = tensor * mask + tl.cp_to_tensor((None, factors), mask=1 - mask) File "/gpfs2/well/dendrou/shared/loom/TAURUS/TAURUS_analyses/cell2cell/code/env/lib/python3.9/site-packages/torch/_tensor.py", line 732, in __array__ return self.numpy() TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Just before this command I verified that the tensor object was of type 'cell2cell.tensor.tensor.InteractionTensor'. The object was made through command as below.

tensor = c2c.tensor.InteractionTensor(rnaseq_matrices=rnaseq_matrices, ppi_data=lr_pairs, context_names=context_names, how='outer', complex_sep='&', interaction_columns=int_columns, communication_score='expression_mean', ) It wasn't really a memory issue but I tried changing init='svd' to 'random' but it made no difference either way! I would be very grateful for any suggestions you may have! Thank you.

[earmingol]

Thanks for your comments @tomthomas3000 !

Regarding the error, which version of Tensor-cell2cell are you using? This seems to be an error we found in an older version, and should be fixed in the latest version.

You can check it with this:

import cell2cell as c2c
c2c.__version__

If the version is older than the 0.5.10, please update it with

pip install -U cell2cell

If you already have the latest version, could you try adding the parameter device='cuda:0' when creating the InteractionTensor?

[tomthomas3000]

@earmingol Thank you for the prompt response. The advice to add parameter device='cuda:0' worked, when I was using a GPU. However, if i am not using a GPU i.e. fully CPU i get stuck in the following step for elbow rank selection with no errors.

My GPU resources are limited, and was wondering whether there was any way of running "outer" on CPU? Thank you ever so much!

[earmingol]

@tomthomas3000 you can always run everything on the CPU, but depending on the size of the tensor it may be very slow.

For running it on a CPU just follow the regular example here: https://earmingol.github.io/cell2cell/tutorials/ASD/01-Tensor-Factorization-ASD/

And replace the how='inner' by how='outer'

Hope this helps

[tomthomas3000]

Point re: slower on CPU noted! I had a GPU related question, I am running cell2cell on a GPU, and I am facing memory issues given the size of the dataset even despite changing init='svd' to init='random' upon elbow_rank_selection. This is a general GPU question and forgive my ignorance, but if i supply more than 1 GPU card, will cell2cell automatically use memory from other GPU cards? Or do I specifically have to change any settings to allow this to happen? My current GPU ram/card is 48GB, but I have the option of adding more cards. The memory error I receive is below:

RuntimeError: CUDA out of memory. Tried to allocate 13.78 GiB (GPU 0; 44.49 GiB total capacity; 18.00 GiB already allocated; 12.08 GiB free; 31.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CON

Thank you for your time and advice.

[earmingol]

Wow! What is the size of your tensor? (you can check in tensor.tensor.shape)

I've never used more than 1 GPU, but it looks like that is possible from this thread: https://stackoverflow.com/questions/54216920/how-to-use-multiple-gpus-in-pytorch

If you want to use all available GPUs you should create your tensor with device='cuda' instead of device='cuda:0'

If you want to use specific GPUs, you can specify their numbers separated by commas, as for example device='cuda:0,2'. There you will use the GPUs labeled as '0' and as '2'.

Hope this helps!

[tomthomas3000]

@earmingol just circled back to tensorcell2cell following trialling a few of the other LR analysis tools. I had to downsample the size to 10% of the dataset due to memory issue. tensor.tensor.shape on this yielded {103, 1318, 35, 35}, however, I am still bumping into memory issues. Is there any chance I could email you to discuss this in detail? Thank you ever so much.

[earmingol]

@tomthomas3000 I would be happy to chat about this. You can contact me to the info in this QR, so we could schedule a zoom meeting or whatever is more convenient.