Memory error while running on CLI

[Erfan1369]

Hello, I am using CLI to run the pyscenic. and it seems it has been installed properly. I wanted to run the grn for a big dataset (17627, 23657) and after loading the expression matrix, I dealt with the following error. Does anyone have idea how i can fix it?

I play a lot with the --cpus-per-task and --mem-per-cpu, but error is repeated.

I would appreciate if anyone have an idea about it.

2022-08-21 00:52:03,871 - pyscenic.cli.pyscenic - INFO - Loading expression matrix. Traceback (most recent call last): File "/opt/venv/bin/pyscenic", line 8, in sys.exit(main()) File "/opt/venv/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 675, in main args.func(args) File "/opt/venv/lib/python3.7/site-packages/pyscenic/cli/pyscenic.py", line 61, in find_adjacencies_command args.gene_attribute, File "/opt/venv/lib/python3.7/site-packages/pyscenic/cli/utils.py", line 143, in load_exp_matrix df = pd.read_csv(fname, sep=suffixes_to_separator(extension), header=0, index_col=0) File "/opt/venv/lib/python3.7/site-packages/pandas/io/parsers.py", line 610, in read_csv return _read(filepath_or_buffer, kwds) File "/opt/venv/lib/python3.7/site-packages/pandas/io/parsers.py", line 468, in _read return parser.read(nrows) File "/opt/venv/lib/python3.7/site-packages/pandas/io/parsers.py", line 1069, in read df = DataFrame(col_dict, columns=columns, index=index) File "/opt/venv/lib/python3.7/site-packages/pandas/core/frame.py", line 529, in init mgr = init_dict(data, index, columns, dtype=dtype) File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/construction.py", line 287, in init_dict return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype) File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/construction.py", line 95, in arrays_to_mgr return create_block_manager_from_arrays(arrays, arr_names, axes) File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 1701, in create_block_manager_from_arrays blocks = _form_blocks(arrays, names, axes) File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 1759, in _form_blocks float_blocks = _multi_blockify(items_dict["FloatBlock"]) File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 1852, in _multi_blockify values, placement = _stack_arrays(list(tup_block), dtype) File "/opt/venv/lib/python3.7/site-packages/pandas/core/internals/managers.py", line 1880, in _stack_arrays stacked = np.empty(shape, dtype=dtype) numpy.core._exceptions.MemoryError: Unable to allocate 3.11 GiB for an array with shape (17627, 23657) and data type float64