relion icon indicating copy to clipboard operation
relion copied to clipboard

unexpectedly small, yet non-zero sigma2 value

Open DrJesseHansen opened this issue 4 months ago • 0 comments

3D refinement with subvolumes gives the error cited. Subvolumes extracted from WarpTools 2.0.0/dev28. reconstruct_particle gives a very nice reconstruction.

mpirun --np 5 --oversubscribe relion_refine_mpi --o Refine3D/job001_3D/run --auto_refine --split_random_halves --i allparticles_bin8_3D.star --ref InitialModel/recon_3D.mrc --trust_ref_size --ini_high 30 --dont_combine_weights_via_disc --pool 3 --pad 2 --ctf --particle_diameter 400 --flatten_solvent --zero_mask --solvent_mask masks/fullmask_final.mrc --solvent_correct_fsc --oversampling 1 --healpix_order 4 --auto_local_healpix_order 4 --offset_range 2 --offset_step 2 --sym C1 --low_resol_join_halves 40 --norm --scale --j 4 --gpu "" --pipeline_control Refine3D/job108/

error: `=== RELION MPI setup ===

  • Number of MPI processes = 5
  • Number of threads per MPI process = 4
  • Total number of threads therefore = 20
  • Leader (0) runs on host = gpu265
  • Follower 1 runs on host = gpu265
  • Follower 2 runs on host = gpu265
  • Follower 3 runs on host = gpu265
  • Follower 4 runs on host = gpu265 ========================== uniqueHost gpu265 has 4 ranks. GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on follower 1 mapped to device 0 Thread 1 on follower 1 mapped to device 0 Thread 2 on follower 1 mapped to device 0 Thread 3 on follower 1 mapped to device 0 GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on follower 2 mapped to device 1 Thread 1 on follower 2 mapped to device 1 Thread 2 on follower 2 mapped to device 1 Thread 3 on follower 2 mapped to device 1 GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on follower 3 mapped to device 2 Thread 1 on follower 3 mapped to device 2 Thread 2 on follower 3 mapped to device 2 Thread 3 on follower 3 mapped to device 2 GPU-ids not specified for this rank, threads will automatically be mapped to available devices. Thread 0 on follower 4 mapped to device 3 Thread 1 on follower 4 mapped to device 3 Thread 2 on follower 4 mapped to device 3 Thread 3 on follower 4 mapped to device 3 Running CPU instructions in double precision. WARNING: allparticles_bin8_3D.star seems to be from a previous version of Relion. Attempting conversion... You should make sure metadata in the optics group table after conversion is correct. Estimating initial noise spectra from at most 10 particles 0/ 0 sec ............................................................~~(,,"> Auto-refine: Iteration= 1 Auto-refine: Resolution= 29.4827 (no gain for 0 iter) Auto-refine: Changes in angles= 999 degrees; and in offsets= 999 Angstroms (no gain for 0 iter) Estimating accuracies in the orientational assignment ... 0/ 0 sec ............................................................~~(,,"> Auto-refine: Estimated accuracy angles= 11.35 degrees; offsets= 9.72928 Angstroms Auto-refine: WARNING: Iter = 1 The angular accuracy is worse than 10 degrees, so basically you cannot align your particles (yet)! Auto-refine: WARNING: You probably need not worry if the accuracy improves during the next few iterations. Auto-refine: WARNING: However, if the problem persists it may lead to spurious FSC curves, so be wary of inflated resolution estimates... Auto-refine: WARNING: Sometimes it is better to tune resolution yourself by adjusting T in a 3D-classification with a single class. CurrentResolution= 29.4827 Angstroms, which requires orientationSampling of at least 8.37209 degrees for a particle of diameter 400 Angstroms Oversampling= 0 NrHiddenVariableSamplingPoints= 1085 OrientationalSampling= 3.75 NrOrientations= 155 TranslationalSampling= 22.112 NrTranslations= 7 ============================= Oversampling= 1 NrHiddenVariableSamplingPoints= 69440 OrientationalSampling= 1.875 NrOrientations= 1240 TranslationalSampling= 11.056 NrTranslations= 56 ============================= Expectation iteration 1 2.68/2.68 min ............................................................~~(,,"> Averaging half-reconstructions up to 40 Angstrom resolution to prevent diverging orientations ... Note that only for higher resolutions the FSC-values are according to the gold-standard! Calculating solvent-corrected gold-standard FSC ... WARNING: FSC curve between unmasked maps never drops below 0.8. Using unmasked FSC as FSC_true... WARNING: This message should go away during the later stages of refinement! Maximization... 000/??? sec ~~(,,"> [oo] DIRECT_A1D_ELEM(sigma2, i)= nan in: /nfs/scistore16/itgrp/schloegl/src/relion/src/backprojector.cpp, line 1088 ERROR: BackProjector::reconstruct: ERROR: unexpectedly small, yet non-zero sigma2 value, this should not happen... === Backtrace === relion_refine_mpi(_ZN11RelionErrorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_l+0x6a) [0x559c09221b2a] relion_refine_mpi(+0x75709) [0x559c0918e709] relion_refine_mpi(_ZN14MlOptimiserMpi12maximizationEv+0x11b8) [0x559c0925e2e8] relion_refine_mpi(_ZN14MlOptimiserMpi7iterateEv+0x3a1) [0x559c0925ff71] relion_refine_mpi(main+0x52) [0x559c0920cf42] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x14b0e4e4624a] /usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x14b0e4e46305] relion_refine_mpi(_start+0x21) [0x559c09210951] ================== ERROR: BackProjector::reconstruct: ERROR: unexpectedly small, yet non-zero sigma2 value, this should not happen...

RELION version: 5.0-beta-3-commit-12cf15 exiting with an error ... ERROR: cannot touch file: Refine3D/job108/RELION_JOB_EXIT_FAILURE

MPI_ABORT was invoked on rank 2 in communicator MPI_COMM_WORLD with errorcode 13.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them.

`

meanwhile, 3D classification runs well.

mpirun --np 5 --oversubscribe relion_refine_mpi --o Class3D/job002_bin8_3D/run --i allparticles_bin8_3D.star --ref InitialModel/recon_3D.mrc --firstiter_cc --trust_ref_size --ini_high 60 --dont_combine_weights_via_disc --pool 3 --pad 2 --ctf --iter 25 --tau2_fudge 1 --particle_diameter 400 --fast_subsets --K 1 --flatten_solvent --zero_mask --strict_highres_exp 30 --oversampling 1 --healpix_order 4 --sigma_ang 0.6 --offset_range 5 --offset_step 2 --sym C1 --norm --scale --j 5 --gpu ""

DrJesseHansen avatar Oct 09 '24 09:10 DrJesseHansen