cryodrgn `analyze_landscape` fails on `train_vae` run using preprocessed data

[olibclarke]

Hi, running cryodrgn analyze_lanscape fails with the appended error. I ran it after train_vae on EMPIAR-11043, preprocessed with downsampling to 64px. Is it incompatible with train_vae runs using preprocessed data, or am I doing something wrong?

(cryodrgn2) cryodrgn analyze_landscape train_vae_full_64_round2/ 15
2022-10-27 19:48:31.284228     Namespace(workdir='/data/processing/cryodrgn_test/train_vae_full_64_round2', epoch=15, device=None, outdir=None, skip_umap=False, vol_ind=None, sketch_size=1000, Apix=1, flip=False, downsample=128, skip_vol=False, thresh=None, dilate=5, mask=None, linkage='average', M=10, pc_dim=20, plot_dim=5, func=<function main at 0x7ff20d262ca0>)
2022-10-27 19:48:31.284282     Saving results to /data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15
2022-10-27 19:48:31.348248     Sketching distribution...
2022-10-27 19:54:28.548869     Generating volumes...
2022-10-27 19:54:28.554600     Running command:
cryodrgn eval_vol /data/processing/cryodrgn_test/train_vae_full_64_round2/weights.15.pkl --config /data/processing/cryodrgn_test/train_vae_full_64_round2/config.pkl --zfile /data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000/z_values.txt -o /data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000 --Apix 1 -d 128
2022-10-27 19:54:30.548597     Use cuda True
2022-10-27 19:54:30.548629     Namespace(weights='/data/processing/cryodrgn_test/train_vae_full_64_round2/weights.15.pkl', config='/data/processing/cryodrgn_test/train_vae_full_64_round2/config.pkl', o='/data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000', prefix='vol_', verbose=False, z=None, z_start=None, z_end=None, n=10, zfile='/data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000/z_values.txt', Apix=1.0, flip=False, invert=False, downsample=128, norm=None, D=None, qlayers=None, qdim=None, zdim=None, encode_mode=None, players=None, pdim=None, enc_mask=None, pe_type=None, feat_sigma=None, pe_dim=None, domain=None, l_extent=None, activation='relu', func=<function main at 0x7fe2fe6c8430>)
2022-10-27 19:54:30.570726     Loaded configuration:
{'dataset_args': {'ctf': '/data/processing/cryodrgn_test/ank_ctf.pkl',
                  'datadir': None,
                  'do_pose_sgd': False,
                  'ind': '/data/processing/cryodrgn_test/train_vae_round1_keep.pkl',
                  'invert_data': True,
                  'keepreal': False,
                  'norm': [0, 271.7933],
                  'particles': '/data/processing/cryodrgn_test/data/preprocessed/64/particles.ft.txt',
                  'poses': '/data/processing/cryodrgn_test/ank_pose_fixed.pkl',
                  'window': True,
                  'window_r': 0.85},
 'lattice_args': {'D': 65, 'extent': 0.5, 'ignore_DC': True},
 'model_args': {'activation': 'relu',
                'domain': 'fourier',
                'enc_mask': 32,
                'encode_mode': 'resid',
                'feat_sigma': 0.5,
                'pdim': 1024,
                'pe_dim': None,
                'pe_type': 'gaussian',
                'players': 3,
                'qdim': 1024,
                'qlayers': 3,
                'zdim': 8},
 'seed': 3884}
Traceback (most recent call last):
  File "/usr/local/envs/cryodrgn2/bin/cryodrgn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/__main__.py", line 64, in main
    args.func(args)
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/eval_vol.py", line 90, in main
    assert args.downsample <= D - 1, "Must be smaller than original box size"
AssertionError: Must be smaller than original box size
Traceback (most recent call last):
  File "/usr/local/envs/cryodrgn2/bin/cryodrgn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/__main__.py", line 64, in main
    args.func(args)
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/analyze_landscape.py", line 342, in main
    generate_volumes(z, outdir, vg, K)
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/analyze_landscape.py", line 71, in generate_volumes
    vg.gen_volumes(f'{outdir}/kmeans{K}', centers)
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/analyze_landscape.py", line 87, in gen_volumes
    analysis.gen_volumes(self.weights, self.config, zfile, outdir, **self.vol_args)
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/analysis.py", line 424, in gen_volumes
    return subprocess.check_call(cmd, shell=True)
  File "/usr/local/envs/cryodrgn2/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'cryodrgn eval_vol /data/processing/cryodrgn_test/train_vae_full_64_round2/weights.15.pkl --config /data/processing/cryodrgn_test/train_vae_full_64_round2/config.pkl --zfile /data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000/z_values.txt -o /data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000 --Apix 1 -d 128' returned non-zero exit status 1.

[zhonge]

Can you try again with --downsample 64? Good catch, we can adjust the default value for this option. Thanks!

On Thu, Oct 27, 2022, 8:08 PM Oliver Clarke @.***> wrote:

Hi, running cryodrgn analyze_lanscape fails with the appended error. I ran it after train_vae on EMPIAR-11043, preprocessed with downsampling to 64px. Is it incompatible with preprocessed data, or am I doing something wrong?

(cryodrgn2) cryodrgn analyze_landscape train_vae_full_64_round2/ 15 2022-10-27 19:48:31.284228 Namespace(workdir='/data/processing/cryodrgn_test/train_vae_full_64_round2', epoch=15, device=None, outdir=None, skip_umap=False, vol_ind=None, sketch_size=1000, Apix=1, flip=False, downsample=128, skip_vol=False, thresh=None, dilate=5, mask=None, linkage='average', M=10, pc_dim=20, plot_dim=5, func=<function main at 0x7ff20d262ca0>) 2022-10-27 19:48:31.284282 Saving results to /data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15 2022-10-27 19:48:31.348248 Sketching distribution... 2022-10-27 19:54:28.548869 Generating volumes... 2022-10-27 19:54:28.554600 Running command: cryodrgn eval_vol /data/processing/cryodrgn_test/train_vae_full_64_round2/weights.15.pkl --config /data/processing/cryodrgn_test/train_vae_full_64_round2/config.pkl --zfile /data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000/z_values.txt -o /data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000 --Apix 1 -d 128 2022-10-27 19:54:30.548597 Use cuda True 2022-10-27 19:54:30.548629 Namespace(weights='/data/processing/cryodrgn_test/train_vae_full_64_round2/weights.15.pkl', config='/data/processing/cryodrgn_test/train_vae_full_64_round2/config.pkl', o='/data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000', prefix='vol_', verbose=False, z=None, z_start=None, z_end=None, n=10, zfile='/data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000/z_values.txt', Apix=1.0, flip=False, invert=False, downsample=128, norm=None, D=None, qlayers=None, qdim=None, zdim=None, encode_mode=None, players=None, pdim=None, enc_mask=None, pe_type=None, feat_sigma=None, pe_dim=None, domain=None, l_extent=None, activation='relu', func=<function main at 0x7fe2fe6c8430>) 2022-10-27 19:54:30.570726 Loaded configuration: {'dataset_args': {'ctf': '/data/processing/cryodrgn_test/ank_ctf.pkl', 'datadir': None, 'do_pose_sgd': False, 'ind': '/data/processing/cryodrgn_test/train_vae_round1_keep.pkl', 'invert_data': True, 'keepreal': False, 'norm': [0, 271.7933], 'particles': '/data/processing/cryodrgn_test/data/preprocessed/64/particles.ft.txt', 'poses': '/data/processing/cryodrgn_test/ank_pose_fixed.pkl', 'window': True, 'window_r': 0.85}, 'lattice_args': {'D': 65, 'extent': 0.5, 'ignore_DC': True}, 'model_args': {'activation': 'relu', 'domain': 'fourier', 'enc_mask': 32, 'encode_mode': 'resid', 'feat_sigma': 0.5, 'pdim': 1024, 'pe_dim': None, 'pe_type': 'gaussian', 'players': 3, 'qdim': 1024, 'qlayers': 3, 'zdim': 8}, 'seed': 3884} Traceback (most recent call last): File "/usr/local/envs/cryodrgn2/bin/cryodrgn", line 8, in sys.exit(main()) File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/main.py", line 64, in main args.func(args) File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/eval_vol.py", line 90, in main assert args.downsample <= D - 1, "Must be smaller than original box size" AssertionError: Must be smaller than original box size Traceback (most recent call last): File "/usr/local/envs/cryodrgn2/bin/cryodrgn", line 8, in sys.exit(main()) File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/main.py", line 64, in main args.func(args) File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/analyze_landscape.py", line 342, in main generate_volumes(z, outdir, vg, K) File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/analyze_landscape.py", line 71, in generate_volumes vg.gen_volumes(f'{outdir}/kmeans{K}', centers) File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/analyze_landscape.py", line 87, in gen_volumes analysis.gen_volumes(self.weights, self.config, zfile, outdir, **self.vol_args) File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/analysis.py", line 424, in gen_volumes return subprocess.check_call(cmd, shell=True) File "/usr/local/envs/cryodrgn2/lib/python3.9/subprocess.py", line 373, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'cryodrgn eval_vol /data/processing/cryodrgn_test/train_vae_full_64_round2/weights.15.pkl --config /data/processing/cryodrgn_test/train_vae_full_64_round2/config.pkl --zfile /data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000/z_values.txt -o /data/processing/cryodrgn_test/train_vae_full_64_round2/landscape.15/kmeans1000 --Apix 1 -d 128' returned non-zero exit status 1.

— Reply to this email directly, view it on GitHub https://github.com/zhonge/cryodrgn/issues/173, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJVIJHT7LTFUDOD2A7ZFR3WFMJY5ANCNFSM6AAAAAARQSAFAM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[olibclarke]

Ah thanks Ellen that seems to have done the trick, running now!

[olibclarke]

Hmmm... it gets further, but now crashes after generating the initial volumes:

2022-10-27 20:23:27.006881     [ 1.44096768 -1.611202   -0.97092277  1.32117772 -0.29614815  4.0233016
  0.03508829 -2.55882859]
2022-10-27 20:23:27.421523     Finished in 0:05:24.691943
2022-10-27 20:23:28.006183     Copying UMAP from /data/processing/cryodrgn_test/train_vae_full_64_round2/analyze.15/umap.pkl
Traceback (most recent call last):
  File "/usr/local/envs/cryodrgn2/bin/cryodrgn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/__main__.py", line 64, in main
    args.func(args)
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/analyze_landscape.py", line 355, in main
    raise NotImplementedError
NotImplementedError
(cryodrgn2)

[olibclarke]

Ah, but I hadn't run cryodrgn analyze first (or not for that specific iteration) - is that required?

[olibclarke]

Hmmm - running analyze_landscape after cryodrgn analyze gets further, but then I run into another error:

2022-10-27 20:39:15.012817     State 8: 165 particles
2022-10-27 20:39:15.012983     State 9: 1 volumes
2022-10-27 20:39:16.958401     State 9: 205 particles
Traceback (most recent call last):
  File "/usr/local/envs/cryodrgn2/bin/cryodrgn", line 8, in <module>
    sys.exit(main())
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/__main__.py", line 64, in main
    args.func(args)
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/analyze_landscape.py", line 365, in main
    analyze_volumes(outdir, K, args.pc_dim, args.M, args.linkage, vol_ind=args.vol_ind, plot_dim=args.plot_dim, particle_ind_orig=particle_ind, Apix=args.Apix)
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/analyze_landscape.py", line 250, in analyze_volumes
    g = hack_barplot([counts[i] for i in range(M)])
  File "/usr/local/envs/cryodrgn2/lib/python3.9/site-packages/cryodrgn/commands/analyze_landscape.py", line 243, in hack_barplot
    g = sns.barplot(np.arange(M), counts_)
TypeError: barplot() takes from 0 to 1 positional arguments but 2 were given
(cryodrgn2)

[vineetbansal]

@olibclarke - can you verify your seaborn version - python -c "import seaborn; print(seaborn.__version__)" and let us know? I see that our pyproject.toml specifies seaborn<0.12. Is it possible for you to fetch the latest master branch and do a pip install . again? I see from your traceback (the bit about the subprocess call in your original message) that you may not have the latest master. We're actively fixing bugs in the master branch (thanks to active users like you), so it might be better to keep your branch updated.

[olibclarke]

Hi @vineetbansal, here is the output:

(cryodrgn2) python -c "import seaborn; print(seaborn.__version__)"
0.12.1

Ok will install again! I think last time I installed was late last week (from master)

EDIT: After trying with the latest version from master, it completes without errors 😊