alienzj

Test using `pd.read_csv`: ```python %%time pd_dfs = [] for i in cov_list: data_cov = pd.read_csv(i, index_col=0, engine="pyarrow") data_cov = data_cov.reset_index() columns_list = list(data_cov.columns) columns_list[0] = 'contig_name' data_cov.columns = columns_list part_data...

Test using `pl::read_csv`: ```python %%time pl_dfs_read = [] for i in cov_list: data_cov = pl.read_csv(i)\ .rename({"": "contig_name"}).filter(pl.col("contig_name").str.contains(sample_id)) pl_dfs_read.append(data_cov) contig_cov = pl.concat(pl_dfs_read, how="align") contig_names = [i.split(":")[1] for i in contig_cov["contig_name"]] contig_cov...

Test using `pl::scan_csv`: ```python %%time pl_dfs_scan = [] for i in cov_list: data_cov = pl.scan_csv(i)\ .rename({"": "contig_name"}).filter(pl.col("contig_name").str.contains(sample_id)).collect() pl_dfs_scan.append(data_cov) contig_cov = pl.concat(pl_dfs_scan, how="align") contig_names = [i.split(":")[1] for i in contig_cov["contig_name"]] contig_cov...

System information: ``` ##### ####### --------------------- ##O#O## OS: AlmaLinux 9.3 (Shamrock Pampas Cat) x86_64 ####### Host: VMware7,1 None ########### Kernel: 5.14.0-362.24.2.el9_3.x86_64 ############# Uptime: 8 days, 1 hour, 49 mins ###############...

Thank you @apcamargo so much for your suggestions! I am going to refine the code.

Same issue I meet recentlly. NixOs 23.11 is running on vaware platform.


!!! 2018-03-12 11:19:17 Doing step 20: Getting overlaps !!! 2018-03-12 11:19:17 Running: nucmer -maxmatch -c 100 292S_more2kb_contigs_nr.ref.seq 292S_more2kb_contigs_nr.qry.seq -p 292S_more2kb_contigs_nr 1: PREPARING DATA 2,3: RUNNING mummer AND CREATING CLUSTERS reading...

30Gb contigs vs 30Gb contigs, how much memory will be used?

I meet this issue too in `Snakemake 8.27.1`.