Should have \sum_n W_nk = 1

[bergazin]

Hello,

I'm trying to extract the free energy from a number of yank experiments and some of them are throwing this error

WARNING: Did not converge to within specified tolerance.
max_delta = 2.964487e-03, tol = 1.000000e-12, maximum_iterations = 250, iterations completed = 249
Traceback (most recent call last):
  File "/export/home/bergazin/anaconda3/envs/sampl-2/bin/yank", line 11, in <module>
    load_entry_point('yank==0.23.7', 'console_scripts', 'yank')()
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/cli.py", line 73, in main
    dispatched = getattr(commands, command).dispatch(command_args)
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/commands/analyze.py", line 147, in dispatch
    single_run()
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/mpi.py", line 271, in _wrapper
    return run_single_node(rank, task, *args, **kwargs)
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/mpi.py", line 220, in run_single_node
    result = task(*args, **kwargs)
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/commands/analyze.py", line 142, in single_run
    output = analyze.analyze_directory(args['--store'], **analyzer_kwargs)
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/analyze.py", line 768, in analyze_directory
    analysis_data = auto_experiment_analyzer.auto_analyze()
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/analyze.py", line 172, in make_copy
    return copy.deepcopy(wrap(*args, **kwargs))
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/analyze.py", line 681, in auto_analyze
    _ = self.get_experiment_free_energy_data()
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/analyze.py", line 172, in make_copy
    return copy.deepcopy(wrap(*args, **kwargs))
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/analyze.py", line 541, in get_experiment_free_energy_data
    data[phase_name] = analyzer.analyze_phase()
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/analyze.py", line 150, in analyze_phase
    Deltaf_ij, dDeltaf_ij = self.get_free_energy()
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/multistate/multistateanalyzer.py", line 1898, in get_free_energy
    self._compute_free_energy()
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/yank/multistate/multistateanalyzer.py", line 1855, in _compute_free_energy
    (Deltaf_ij, dDeltaf_ij) = self.mbar.getFreeEnergyDifferences()
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/pymbar/mbar.py", line 542, in getFreeEnergyDifferences
    np.exp(self.Log_W_nk), self.N_k, method=uncertainty_method)
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/pymbar/mbar.py", line 1679, in _computeAsymptoticCovarianceMatrix
    check_w_normalized(W, N_k)
  File "/export/home/bergazin/anaconda3/envs/sampl-2/lib/python3.7/site-packages/pymbar/utils.py", line 360, in check_w_normalized
    (firstbad, column_sums[firstbad], np.sum(badcolumns)))
pymbar.utils.ParameterError: Warning: Should have \sum_n W_nk = 1.  Actual column sum for state 0 was 1.005054. 20 other columns have similar problems

My pymbar version is 3.0.5. Here is the YAML file I used for this calculation, the experiment logfile, and all my env packages in case it helps error_files.zip

[jchodera]

I'd cross-post this in https://github.com/choderalab/yank/issues too just in case!

Usually, the problem underlying this warning is that the different distributions are thermodynamically inconsistent, but I'm not sure why YANK would be doing this unless the simulations are incredibly short or most of the data is getting thrown out by equilibration detection.

[bergazin]

Downgrading to pymbar version 3.0.3 has fixed this

[jchodera]

Tagging @mrshirts: Do you have anyone that can take a look at what went into pymbar 3.0.4 that may have broken things?

[mrshirts]

I can take a look after the Holidays - I don't think there's anyone else who can right now.

[davidlmobley]

We're running into this same issue with GROMACS-based free energy calculations, so it's not specifically yank-related. @hannahbaumann is going to provide input files/scripts. She was able to bypass by downgrading pymbar (as did Yank -- https://github.com/omnia-md/conda-recipes/pull/1027 ) but we probably need to sort this out soon.

I think Hannah's test case will help as it's an analysis-only failure: We have an analysis which seems OK and works with the older pymbar, but fails with the new pymbar (via alchemlyb).

@Lnaden are you still involved with this package? I see some relatively recent merges from you.

[mrshirts]

Happy to look at bugfixes - we are working to get 4.0 out in the next week or so, so would want to get any bugfixes in. Is there a dataset to test?

[hannahbaumann]

I opened a new issue here: https://github.com/choderalab/pymbar/issues/419 and provided some input files! Let me know if you need more information!