ZeroDivisionError: float division by zero

[vishnubob]

This is my first time running BESST, have you ever seen this error before?

Warning: Orientations of library mappings not specified. Treating all libraries as fr mapped.         If this is not the case, please provide orientations with the --orientations parameter.
Warning: Orientations of library mappings not specified. Treating all libraries as fr mapped.         If this is not the case, please provide orientations with the --orientation parameter.
Number of initial contigs: 15153
Choosing mode: 319
mu_adjusted:370.086480811, sigma_adjusted:80.8728739826, skewness_adjusted:1.05043478189
mode adj: 319
median adj 352
Creating contig graph with library:  CENPK_reads_to_contigs_sorted.bam
Constructed contig graph. Start BESST algorithm for creating scaffolds.
Entering "find_all_paths_for_start_node"
iterating until maximum of 464 extensions.
Number of nodes:26534, Number of edges: 14078
enter Between scaf node:0, scaffold progression 0.0%.
enter Between scaf node:100, scaffold progression 21.6%.
enter Between scaf node:200, scaffold progression 43.1%.
enter Between scaf node:300, scaffold progression 64.7%.
enter Between scaf node:400, scaffold progression 86.2%.
Total nr of paths found: 91 with score larger than: 1.5
Elapsed time single core pathfinder:  0.110065937042
Traceback (most recent call last):
  File "/home/giles/src/BESST/runBESST", line 415, in <module>
    main(args)
  File "/home/giles/src/BESST/runBESST", line 194, in main
    MS.Algorithm(G, G_prime, Contigs, small_contigs, Scaffolds, small_scaffolds, Information, param)   # Make scaffolds, store the complex areas (consisting of contig/scaffold) in F, store the created scaffolds in Scaffolds, update Contigs
  File "/ginkgo/bitome/users/giles/src/BESST/BESST/MakeScaffolds.py", line 92, in Algorithm
    PROBetweenScaf(G_prime, Contigs, small_contigs, Scaffolds, small_scaffolds, param, dValuesTable, Information)
  File "/ginkgo/bitome/users/giles/src/BESST/BESST/MakeScaffolds.py", line 1569, in PROBetweenScaf
    G_, path = estimate_path_gaps(Contigs, small_contigs, path,Scaffolds,small_scaffolds, G_prime,param)
  File "/ginkgo/bitome/users/giles/src/BESST/BESST/MakeScaffolds.py", line 1160, in estimate_path_gaps
    final_path_instance, final_contigs_to_indexes, final_indexes_to_contigs, final_index_observations = calculate_path_LP(path,Scaffolds,small_scaffolds,observations,param)
  File "/ginkgo/bitome/users/giles/src/BESST/BESST/MakeScaffolds.py", line 767, in calculate_path_LP
    result_path = order_contigs.main(ctg_lengths_index_ordered, index_observations, param)
  File "/ginkgo/bitome/users/giles/src/BESST/BESST/order_contigs.py", line 534, in main
    optimal_LP_gaps = path.LP_solve_gaps(param)
  File "/ginkgo/bitome/users/giles/src/BESST/BESST/order_contigs.py", line 306, in LP_solve_gaps
    exp_means_gapest[(i,j,is_PE_link)] = self.observations[(i,j,is_PE_link)][0] + lnpe.GapEstimator(param.lognormal_mean, param.lognormal_sigma, self.read_len, samples, self.ctgs[i].length, c2_len=self.ctgs[j].length)
  File "/usr/local/lib/python2.7/dist-packages/mathstats/log_normal_param_est.py", line 229, in GapEstimator
    d_ML = get_d_ML_Newton_Raphson(mu, sigma, r, c_min, observations, c_max, d_lower, d_upper, cutoff_approx_level)
  File "/usr/local/lib/python2.7/dist-packages/mathstats/log_normal_param_est.py", line 282, in get_d_ML_Newton_Raphson
    g_d_ratio_x1 = n*g_prime_d_nominator_x1/g_d_denominator_x1
ZeroDivisionError: float division by zero

[ksahlin]

Thanks for reporting! I've seen a similar bug in that module and I thought I had fixed it.. What version are you running? Could you attach the statistics.txt file generated in the BESST_output folder here or email it to me so I can look into this. Thanks!

[vishnubob]

I believe I'm running from the HEAD of master. Stats file attached, thanks!

Statistics.txt

[ksahlin]

Sorry for the slow answer, the bug seems non-trivial to track given the info I have now. Would it be possible for you to add the line: print(mu, sigma, r, c_min, observations, c_max, d_lower, d_upper, cutoff_approx_level) at the line right before the error? That is, before 282 in /usr/local/lib/python2.7/dist-packages/mathstats/log_normal_param_est.py
and send me the last line generated before the error? Then I could work with these values to resolve the bug. I would need to update the mathstats library to a new version so the current one you have installed would have to be overwritten anyway.

Thanks!

[sletort]

Hi Kristoffer, I updated my files according to your recommandations. see them in https://github.com/sletort/debug_mode.storage/tree/master/besst/2016.07

[ksahlin]

Hi Sebastien & Giles,

Can you please try the latest commit on master of BESST? It requires a newer version of mathstats (0.2.5). pip will detect this dependency automatically and update it. If you don't use pip you have to manually install the latest mathstats as well.

Thanks

[sletort]

Hi Kristoffer,

I've just rerun with my data. The process terminates without problem. Issue can be closed in my opinion. thanks.

[ksahlin]

Thank you!

[ptranvan]

Hi,

I have a similar problem despite of using the last commit version:

Number of initial contigs: 546681 Choosing mode: 315 mu_adjusted:319.291063062, sigma_adjusted:55.3128021429, skewness_adjusted:0.680770613239 mode adj: 315 median adj 316 Creating contig graph with library: 350.no_contaminant.bam Traceback (most recent call last): File "/scratch/beegfs/monthly/ptranvan/Software/BESST/BESST/runBESST", line 416, in main(args) File "/scratch/beegfs/monthly/ptranvan/Software/BESST/BESST/runBESST", line 177, in main (G, G_prime) = CG.PE(Contigs, Scaffolds, Information, C_dict, param, small_contigs, small_scaffolds, bam_file) #Create graph, single out too short contigs/scaffolds and store them in F File "/scratch/beegfs/monthly/ptranvan/Software/BESST/BESST/BESST/CreateGraph.py", line 300, in PE GiveScoreOnEdges(G, Scaffolds, small_scaffolds, Contigs, param, Information, plot) File "/scratch/beegfs/monthly/ptranvan/Software/BESST/BESST/BESST/CreateGraph.py", line 517, in GiveScoreOnEdges gap = lnpe.GapEstimator(param.lognormal_mean, param.lognormal_sigma, param.read_len, samples, len1, c2_len=len2) File "/software/lib/python2.7/site-packages/mathstats/log_normal_param_est.py", line 251, in GapEstimator d_ML = get_d_ML_Newton_Raphson(mu, sigma, r, c_min, observations, c_max, d_lower, d_upper, cutoff_approx_level) File "/software/lib/python2.7/site-packages/mathstats/log_normal_param_est.py", line 305, in get_d_ML_Newton_Raphson g_d_ratio_x1 = n*g_prime_d_nominator_x1/g_d_denominator_x1 ZeroDivisionError: float division by zero

[ksahlin]

The bug was fixed in the external library mathstats - so its possible that you are running an older version of mathstats despite having the newest version of BESST. You can run pip list to see the version of your packages. One line should say mathstats (0.2.5). If not, run pip install --upgrade mathstats.

Let me know if this fixes the issue. Thanks!

[ptranvan]

Hi,

I confirm that I have the last version (or at least the one you mentioned) and still have the error.:

pip list | grep 'mathstats' mathstats (0.2.5)

[ksahlin]

Ok! I recommend running with parameter --no_score until I fix this.

[ptranvan]

Thanks ksahlin for your help but even with the --no_option, I still got the error.

Here is my command:

runBESST -c ../assembly/filter/1_Tdi_contig_filtered.fasta -f 1_Tdi_is_350.no_contaminant.bam 1_Tdi_is_550.no_contaminant.bam 1_Tdi_is_700.no_contaminant.bam 1_Tdi_is_3000.no_contaminant.bam 1_Tdi_is_5000.no_contaminant.bam -o output -orientation fr fr fr fr fr --no_score

[ksahlin]

I then recommend skipping the first library, i.e. the 350bp one. This is the library that produces the error.

[ptranvan]

Do you know what can cause this error ? is it because too few coverage ?

[ksahlin]

This bug is a tricky one to fix and I would need the exact instance that generated the error.

Would it be possible for you to:

1. uncomment the line 302, i.e. print(mu, sigma, r, c_min, observations, c_max, d_lower, d_upper, cutoff_approx_level) in File /software/lib/python2.7/site-packages/mathstats/log_normal_param_est.py
2. Run BESST as when error occured
3. Send me the last line generated before the error (the instance that causes the bug)?

Similarly suggested in an earlier reply in this issue here.

[ptranvan]

Last line is:

(250.29167297172077, 5.755742213586912, 0.056298976556232484, 122.89, 679.0, [127, 137, 139, 143, 143], 1002.0, 172.0, 263, 89480.61026514288)

[ptranvan]

Hi @ksahlin , did you have an opportunity to look at it ?

Thanks.

[ksahlin]

Hi @ptranvan,

I've performed a fix in mathstats 0.2.6. Please run pip install --upgrade mathstats.

The fix is really just catching the runtime error. The ML gap estimate is not found and an heuristic fallback value is chosen as the gap estimate for the instance caught with this error, this will be printed as warnings.

Let me know if your instance terminates without runtime error.

I also pushed a new version of besst 2.2.7 (requiring the newest mathstats). So doing a pip upgrade of BESST will also update to newest mathstats.