hifiasm
hifiasm copied to clipboard
chhylp123
HiFIasm is taking to much RAM to run with low coverage data
Hi,
I am trying to assembly some low coverage HIFI reads for a plant genome. I am running this assembly in a cluster with SLURM. It is easier for me in this cluster to have the program running if I request 200Gb or RAM or less. However, the program only can run with this RAM if I use just a few CPUs and it is not finishing in more than a week. It is possible to limit the amount of RAM that the program is using? It is possible to use temporary files instead of RAM for the assembly process?
Thanks
Could you please show the log file? I guess it is the low coverage issue, instead of the RAM issue.
It is a long file
Starts like this: [M::ha_analyze_count] lowest: count[139] = 63492 [M::ha_analyze_count] highest: count[4095] = 92450 [M::ha_hist_line] 2: ****************************************************************************************************> 204947280 [M::ha_hist_line] 3: ****************************************************************************************************> 258505393 [M::ha_hist_line] 4: ****************************************************************************************************> 272395850 [M::ha_hist_line] 5: ****************************************************************************************************> 246099528 [M::ha_hist_line] 6: ****************************************************************************************************> 199218948 [M::ha_hist_line] 7: ****************************************************************************************************> 152058370 [M::ha_hist_line] 8: ****************************************************************************************************> 114269619 [M::ha_hist_line] 9: ****************************************************************************************************> 85721596 [M::ha_hist_line] 10: ****************************************************************************************************> 64951133 [M::ha_hist_line] 11: ****************************************************************************************************> 49483601 [M::ha_hist_line] 12: ****************************************************************************************************> 37674071 [M::ha_hist_line] 13: ****************************************************************************************************> 28506219 [M::ha_hist_line] 14: ****************************************************************************************************> 21653915 [M::ha_hist_line] 15: ****************************************************************************************************> 16442757 [M::ha_hist_line] 16: ****************************************************************************************************> 12760670 [M::ha_hist_line] 17: ****************************************************************************************************> 10127605 [M::ha_hist_line] 18: ****************************************************************************************************> 8216675 [M::ha_hist_line] 19: ****************************************************************************************************> 6921313 [M::ha_hist_line] 20: ****************************************************************************************************> 5900445 [M::ha_hist_line] 21: ****************************************************************************************************> 5115707 [M::ha_hist_line] 22: ****************************************************************************************************> 4489866 [M::ha_hist_line] 23: ****************************************************************************************************> 3983721 [M::ha_hist_line] 24: ****************************************************************************************************> 3577529 [M::ha_hist_line] 25: ****************************************************************************************************> 3177434 [M::ha_hist_line] 26: ****************************************************************************************************> 2865416 [M::ha_hist_line] 27: ****************************************************************************************************> 2599756 [M::ha_hist_line] 28: ****************************************************************************************************> 2364726 [M::ha_hist_line] 29: ****************************************************************************************************> 2148807 [M::ha_hist_line] 30: ****************************************************************************************************> 1967226 [M::ha_hist_line] 31: ****************************************************************************************************> 1807515 [M::ha_hist_line] 32: ****************************************************************************************************> 1666505 [M::ha_hist_line] 33: ****************************************************************************************************> 1543132 [M::ha_hist_line] 34: ****************************************************************************************************> 1436709 [M::ha_hist_line] 35: ****************************************************************************************************> 1340605 [M::ha_hist_line] 36: ****************************************************************************************************> 1242376 [M::ha_hist_line] 37: ****************************************************************************************************> 1171813 [M::ha_hist_line] 38: ****************************************************************************************************> 1100013 [M::ha_hist_line] 39: ****************************************************************************************************> 1031438 [M::ha_hist_line] 40: ****************************************************************************************************> 973903 [M::ha_hist_line] 41: ****************************************************************************************************> 914078 [M::ha_hist_line] 42: ****************************************************************************************************> 865389 [M::ha_hist_line] 43: ****************************************************************************************************> 818943 [M::ha_hist_line] 44: ****************************************************************************************************> 772556 [M::ha_hist_line] 45: ****************************************************************************************************> 742143 [M::ha_hist_line] 46: ****************************************************************************************************> 702184 [M::ha_hist_line] 47: ****************************************************************************************************> 666981 [M::ha_hist_line] 48: ****************************************************************************************************> 637433 [M::ha_hist_line] 49: ****************************************************************************************************> 605544 [M::ha_hist_line] 50: ****************************************************************************************************> 578728 [M::ha_hist_line] 51: ****************************************************************************************************> 552969 [M::ha_hist_line] 52: ****************************************************************************************************> 528315 [M::ha_hist_line] 53: ****************************************************************************************************> 504845 [M::ha_hist_line] 54: ****************************************************************************************************> 486533 [M::ha_hist_line] 55: ****************************************************************************************************> 465041 [M::ha_hist_line] 56: ****************************************************************************************************> 445259 [M::ha_hist_line] 57: ****************************************************************************************************> 428053 [M::ha_hist_line] 58: ****************************************************************************************************> 409606 [M::ha_hist_line] 59: ****************************************************************************************************> 394805 [M::ha_hist_line] 60: ****************************************************************************************************> 382864 [M::ha_hist_line] 61: ****************************************************************************************************> 367444 [M::ha_hist_line] 62: ****************************************************************************************************> 355779 [M::ha_hist_line] 63: ****************************************************************************************************> 344111 [M::ha_hist_line] 64: ****************************************************************************************************> 330877 [M::ha_hist_line] 65: ****************************************************************************************************> 321798 [M::ha_hist_line] 66: ****************************************************************************************************> 310143 [M::ha_hist_line] 67: ****************************************************************************************************> 298314 [M::ha_hist_line] 68: ****************************************************************************************************> 290796 [M::ha_hist_line] 69: ****************************************************************************************************> 283023 [M::ha_hist_line] 70: ****************************************************************************************************> 270682 [M::ha_hist_line] 71: ****************************************************************************************************> 265370 [M::ha_hist_line] 72: ****************************************************************************************************> 257947 [M::ha_hist_line] 73: ****************************************************************************************************> 250755 [M::ha_hist_line] 74: ****************************************************************************************************> 241319 [M::ha_hist_line] 75: ****************************************************************************************************> 235403 [M::ha_hist_line] 76: ****************************************************************************************************> 228639 [M::ha_hist_line] 77: ****************************************************************************************************> 222234 [M::ha_hist_line] 78: ****************************************************************************************************> 215142 [M::ha_hist_line] 79: ****************************************************************************************************> 211065 [M::ha_hist_line] 80: ****************************************************************************************************> 205040 [M::ha_hist_line] 81: ****************************************************************************************************> 198079 [M::ha_hist_line] 82: ****************************************************************************************************> 193520 [M::ha_hist_line] 83: ****************************************************************************************************> 190783 [M::ha_hist_line] 84: ****************************************************************************************************> 184970 [M::ha_hist_line] 85: ****************************************************************************************************> 180547 [M::ha_hist_line] 86: ****************************************************************************************************> 176451 [M::ha_hist_line] 87: ****************************************************************************************************> 171616 [M::ha_hist_line] 88: ****************************************************************************************************> 168357 [M::ha_hist_line] 89: ****************************************************************************************************> 163438 [M::ha_hist_line] 90: ****************************************************************************************************> 159742 [M::ha_hist_line] 91: ****************************************************************************************************> 157462 [M::ha_hist_line] 92: ****************************************************************************************************> 152067 [M::ha_hist_line] 93: ****************************************************************************************************> 148581 [M::ha_hist_line] 94: ****************************************************************************************************> 145132 [M::ha_hist_line] 95: ****************************************************************************************************> 142308 [M::ha_hist_line] 96: ****************************************************************************************************> 139060 [M::ha_hist_line] 97: ****************************************************************************************************> 137509 [M::ha_hist_line] 98: ****************************************************************************************************> 133952 [M::ha_hist_line] 99: ****************************************************************************************************> 130582 [M::ha_hist_line] 100: ****************************************************************************************************> 128107 [M::ha_hist_line] 101: ****************************************************************************************************> 124800 [M::ha_hist_line] 102: ****************************************************************************************************> 122767 [M::ha_hist_line] 103: ****************************************************************************************************> 119491 [M::ha_hist_line] 104: ****************************************************************************************************> 117211 [M::ha_hist_line] 105: ****************************************************************************************************> 114558 [M::ha_hist_line] 106: ****************************************************************************************************> 113506 [M::ha_hist_line] 107: ****************************************************************************************************> 111489 [M::ha_hist_line] 108: ****************************************************************************************************> 109304 [M::ha_hist_line] 109: ****************************************************************************************************> 107716 [M::ha_hist_line] 110: ****************************************************************************************************> 104932 [M::ha_hist_line] 111: ****************************************************************************************************> 101927 [M::ha_hist_line] 112: ****************************************************************************************************> 100215 [M::ha_hist_line] 113: ****************************************************************************************************> 100055 [M::ha_hist_line] 114: ****************************************************************************************************> 96541 [M::ha_hist_line] 115: *************************************************************************************************> 95565 [M::ha_hist_line] 116: **************************************************************************************************** 92709 [M::ha_hist_line] 117: **************************************************************************************************** 92212 [M::ha_hist_line] 118: ************************************************************************************************** 90147 [M::ha_hist_line] 119: ************************************************************************************************ 88811 [M::ha_hist_line] 120: ********************************************************************************************** 87055 [M::ha_hist_line] 121: ********************************************************************************************* 86094 [M::ha_hist_line] 122: ******************************************************************************************* 83994 [M::ha_hist_line] 123: ****************************************************************************************** 82747 [M::ha_hist_line] 124: **************************************************************************************** 81282 [M::ha_hist_line] 125: ************************************************************************************** 79839 [M::ha_hist_line] 126: ************************************************************************************** 79525 [M::ha_hist_line] 127: ************************************************************************************ 77350 [M::ha_hist_line] 128: ********************************************************************************** 76070 [M::ha_hist_line] 129: ********************************************************************************* 75106 [M::ha_hist_line] 130: ******************************************************************************* 72815 [M::ha_hist_line] 131: ****************************************************************************** 72535 [M::ha_hist_line] 132: ****************************************************************************** 72090 [M::ha_hist_line] 133: *************************************************************************** 69666 [M::ha_hist_line] 134: *************************************************************************** 68907 [M::ha_hist_line] 135: ************************************************************************* 67844 [M::ha_hist_line] 136: ************************************************************************ 66145 [M::ha_hist_line] 137: *********************************************************************** 65505 [M::ha_hist_line] 138: ********************************************************************** 64999 [M::ha_hist_line] 139: ********************************************************************* 63492 [M::ha_hist_line] 140: ********************************************************************* 63600 [M::ha_hist_line] 141: ******************************************************************* 61877 [M::ha_hist_line] 142: ****************************************************************** 60792 [M::ha_hist_line] 143: ****************************************************************** 60767 [M::ha_hist_line] 144: **************************************************************** 59568 [M::ha_hist_line] 145: **************************************************************** 59180 [M::ha_hist_line] 146: *************************************************************** 58203 [M::ha_hist_line] 147: ************************************************************* 55966 [M::ha_hist_line] 148: ************************************************************* 56555 [M::ha_hist_line] 149: *********************************************************** 54905 [M::ha_hist_line] 150: *********************************************************** 54565 [M::ha_hist_line] 151: ********************************************************** 53772 [M::ha_hist_line] 152: ********************************************************** 53289 [M::ha_hist_line] 153: ********************************************************* 52514 [M::ha_hist_line] 154: ******************************************************** 51843 [M::ha_hist_line] 155: ******************************************************* 50742 [M::ha_hist_line] 156: ****************************************************** 50172 some lines more and we have this [M::ha_hist_line] 4066: 29 [M::ha_hist_line] 4067: 38 [M::ha_hist_line] 4068: 35 [M::ha_hist_line] 4069: 37 [M::ha_hist_line] 4070: 30 [M::ha_hist_line] 4071: 40 [M::ha_hist_line] 4072: 42 [M::ha_hist_line] 4073: 39 [M::ha_hist_line] 4074: 37 [M::ha_hist_line] 4075: 37 [M::ha_hist_line] 4076: 29 [M::ha_hist_line] 4077: 34 [M::ha_hist_line] 4078: 39 [M::ha_hist_line] 4079: 45 [M::ha_hist_line] 4080: 29 [M::ha_hist_line] 4081: 30 [M::ha_hist_line] 4082: 32 [M::ha_hist_line] 4083: 26 [M::ha_hist_line] 4084: 34 [M::ha_hist_line] 4085: 29 [M::ha_hist_line] 4086: 47 [M::ha_hist_line] 4087: 28 [M::ha_hist_line] 4088: 26 [M::ha_hist_line] 4089: 30 [M::ha_hist_line] 4090: 35 [M::ha_hist_line] 4091: 34 [M::ha_hist_line] 4092: 27 [M::ha_hist_line] 4093: 32 [M::ha_hist_line] 4094: 39 [M::ha_hist_line] 4095: **************************************************************************************************** 92450 [M::ha_hist_line] rest: 0 [M::ha_analyze_count] left: count[140] = 63600 [M::ha_analyze_count] right: none [M::ha_ft_gen] peak_hom: 4095; peak_het: 140 [M::ha_ct_shrink::925.4116.70] ==> counted 92489 distinct minimizer k-mers [M::ha_ft_gen::929.915[email protected]] ==> filtered out 92489 k-mers occurring 4094 or more times [M::ha_opt_update_cov] updated max_n_chain to 20475 [M::yak_count] collected 736046883 minimizers [M::ha_pt_gen::1515.1765.79] ==> counted 106052215 distinct minimizer k-mers [M::ha_pt_gen] count[4095] = 0 (for sanity check) [M::ha_analyze_count] lowest: count[99] = 5268 [M::ha_analyze_count] highest: count[100] = 5315 slurm-16887619.out many more lines and ends like this: [M::ha_hist_line] 901: * 70 [M::ha_hist_line] 902: * 67 [M::ha_hist_line] 903: * 56 [M::ha_hist_line] 904: * 55 [M::ha_hist_line] 905: * 43 [M::ha_hist_line] 906: * 52 [M::ha_hist_line] 907: * 52 [M::ha_hist_line] 908: * 56 [M::ha_hist_line] 909: * 59 [M::ha_hist_line] 910: * 52 [M::ha_hist_line] 911: * 61 [M::ha_hist_line] 912: * 65 [M::ha_hist_line] 913: * 56 [M::ha_hist_line] 914: * 69 [M::ha_hist_line] 915: * 51 [M::ha_hist_line] 916: * 58 [M::ha_hist_line] 917: * 57 [M::ha_hist_line] 918: * 66 [M::ha_hist_line] 919: * 59 [M::ha_hist_line] 920: * 54 [M::ha_hist_line] 921: * 65 [M::ha_hist_line] 922: * 61 [M::ha_hist_line] 923: * 46 [M::ha_hist_line] 924: * 56 [M::ha_hist_line] 925: * 69 [M::ha_hist_line] 926: * 53 [M::ha_hist_line] 927: * 47 [M::ha_hist_line] 928: * 42 [M::ha_hist_line] 929: * 55 [M::ha_hist_line] 930: * 47 [M::ha_hist_line] 931: * 59 [M::ha_hist_line] 932: * 63 [M::ha_hist_line] 933: * 61 [M::ha_hist_line] 934: * 36 [M::ha_hist_line] 935: * 52 [M::ha_hist_line] 936: * 49 [M::ha_hist_line] 937: * 56 [M::ha_hist_line] 938: * 48 [M::ha_hist_line] 939: * 62 [M::ha_hist_line] 940: * 41 [M::ha_hist_line] 941: * 54 [M::ha_hist_line] 942: * 53 [M::ha_hist_line] 943: * 53 [M::ha_hist_line] 944: * 44 [M::ha_hist_line] 945: * 43 [M::ha_hist_line] 946: * 45 [M::ha_hist_line] 947: * 56 [M::ha_hist_line] 948: * 60 [M::ha_hist_line] 949: * 41 [M::ha_hist_line] 950: * 49 [M::ha_hist_line] 951: * 64 [M::ha_hist_line] 952: * 56 [M::ha_hist_line] 953: * 60 [M::ha_hist_line] 954: * 52 [M::ha_hist_line] 955: * 52 [M::ha_hist_line] 956: * 42 [M::ha_hist_line] 957: * 46 [M::ha_hist_line] 958: * 65 [M::ha_hist_line] 959: * 44 [M::ha_hist_line] 960: * 54 [M::ha_hist_line] 961: * 40 [M::ha_hist_line] 962: * 42 [M::ha_hist_line] 963: * 38 [M::ha_hist_line] 964: * 43 [M::ha_hist_line] 965: * 54 [M::ha_hist_line] 966: * 53 [M::ha_hist_line] 967: * 44 [M::ha_hist_line] 968: * 45 [M::ha_hist_line] 969: * 57 [M::ha_hist_line] 970: * 59 [M::ha_hist_line] 971: * 47 [M::ha_hist_line] 972: * 45 [M::ha_hist_line] 973: * 58 [M::ha_hist_line] 974: * 46 [M::ha_hist_line] 975: * 53 [M::ha_hist_line] 976: * 46 [M::ha_hist_line] 977: * 43 [M::ha_hist_line] 978: * 57 [M::ha_hist_line] 979: * 53 [M::ha_hist_line] 980: * 47 [M::ha_hist_line] 981: * 46 [M::ha_hist_line] 982: * 41 [M::ha_hist_line] 983: * 49 [M::ha_hist_line] 984: * 38 [M::ha_hist_line] 985: * 58 [M::ha_hist_line] 986: * 55 [M::ha_hist_line] 987: * 49 [M::ha_hist_line] 988: * 47 [M::ha_hist_line] 989: * 44 [M::ha_hist_line] 990: * 54 [M::ha_hist_line] rest: **************************************************************************************************> 34326 [M::ha_analyze_count] left: none [M::ha_analyze_count] right: none [M::ha_pt_gen] peak_hom: 92; peak_het: -1 [M::ha_ct_shrink::122635.32519.70] ==> counted 76265932 distinct minimizer k-mers [M::ha_pt_gen::] counting in normal mode [M::yak_count] collected 735307230 minimizers [M::ha_pt_gen::122898.20819.68] ==> indexed 723860205 positions, counted 76265932 distinct minimizer k-mers /var/spool/slurmd/job16887619/slurm_script: line 9: 32107 Killed hifiasm -o ANN1372_hiafsm_HiC_moreCPUs.asm -f39 -t 20 --h1 NS.1630.004.IDT_i7_194---IDT_i5_193.ANN1372_3_HiC_R1.fastq.gz --h2 NS.1630.004.IDT_i7_194---IDT_i5_193.ANN1372_3_HiC_R2.fastq.gz Sequel.RunS142_S2.002.ANN1372-3.ccs.fastq.gz slurmstepd: error: Detected 1 oom-kill event(s) in StepId=16887619.batch cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.
The file has 6313 lines. Is this helpful? Can I send you the file by email? My sample is heterozygous, I am not sure if the histogram should look like this.
It should be caused by the low coverage issue so that hifiasm cannot find the correct homozygous peak (see: https://hifiasm.readthedocs.io/en/latest/faq.html#why-does-hifiasm-stuck-or-crash). If the coverage is too low, the results won't be good.
Yes, what makes me ask you, is that I have another data set with even more low coverage reads (6x) and Hifiasm was able to finish the work, producing the two asm.bp.hap.p_ctg.gfa files. However, the kmer plot is very similar to the kmer plot of this data (8x). I guess that I can not trust too much in the assembly that was able to finish. But I am wondering why the lowest coverage data set was able to finish and the other no. There are more contaminants in the 8X data set? How I could test that?
I guess it is relatively good. The N50 for Hap1 is 96.20 kp with a total assembly of 2.3 Gb and for Hap2 is 110.56 kb with a total assembly of 1.9 GB, with hicanu I have 76.37 for the primary assembly with a total assembly of 2.45 Gb. But I still don't understand why I can have an assembly for the 6X data and not for the 8X data.
We can say it is a "bug", but we think it is not such important so that we have fixed it. Hifi-assemblies should always have MB long N50 otherwise we lose the power of HiFi. As you can see with very low coverage, the N50 is just 100kb.
But if you think it important to make hifiasm work with low coverage data, we can fix this "bug" soon.
Many thanks for your answers. I think that could be useful to have something working on low coverage HiFi assemblies. But It is also clear that with higher coverage we could make better use of hifiasm. Thanks again!