[BUG] Slurm execution error: TomoBEAR/for_redistribution_files_only/run_TomoBEAR.sh: Not a directory

[dacolombo]

Describe the bug

Hi all, after using local execution of TomoBEAR for the Ribosome (EMPIAR-10064) tutorial, I am trying to submit the last DynamoAlignment job to a Slurm cluster since it needs more resources. When running the following command:

TomoBEAR slurm ~/ribosome_empiar_10064_dynamo.json ~/defaults.json

The job is correctly submitted to the queue, but it fails with this error as soon as it starts:

line 17: /TomoBEAR-0.2.0/TomoBEAR/for_redistribution_files_only/run_TomoBEAR.sh: Not a directory

Indeed, the directory TomoBEAR/for_redistribution_files_only is not present inside of the TomoBEAR installation directory. I have seen in the code that this directory is accessed during the compilation and initialisation: https://github.com/KudryashevLab/TomoBEAR/blob/8a940e3cd0d5990aa37ea3de763a37573d68ae23/environment/initializeEnvironment.m#L166 and https://github.com/KudryashevLab/TomoBEAR/blob/8a940e3cd0d5990aa37ea3de763a37573d68ae23/utilities/compileTomoBEAR.m#L116

But I can't find how these directory are created and what should be the content of the run_TomoBEAR.sh script inside of it. I have a run_TomoBEAR.sh script in the TomoBEAR installation directory, which also makes use of the script inside of for_redistribution_files_only.

Am I missing something? Is there some procedure to generate this run_TomoBEAR.sh script? What should be the content of this script?

From what I understand, TomoBEAR has been initialised correctly, with the only warning showing being the missing conda environment to be provided for some optional dependencies.

Project JSON file The project json file is the same present in the repository for the Ribosome (EMPIAR-10064) tutorial.

~/ribosome_empiar_10064_dynamo.json

{
    "general": {
        "project_name": "Ribosome",
        "project_description": "Ribosome EMPIAR 10064",
        "data_path": "~/tomobear_test/tests/",
        "processing_path": "~/tomobear_test/output",
        "expected_symmetrie": "C1",
        "apix": 2.62,
        "tilt_angles": [-60.0, -58.0, -56.0, -54.0, -52.0, -50.0, -48.0, -46.0, -44.0, -42.0, -40.0, -38.0, -36.0, -34.0, -32.0, -30.0, -28.0, -26.0, -24.0, -22.0, -20.0, -18.0, -16.0, -14.0, -12.0, -10.0, -8.0, -6.0, -4.0, -2.0, 0.0, 2.0, 4.0, 6.0, 8.0, 10.0, 12.0, 14.0, 16.0, 18.0, 20.0, 22.0, 24.0, 26.0, 28.0, 30.0, 32.0, 34.0, 36.0, 38.0, 40.0, 42.0, 44.0, 46.0, 48.0, 50.0, 52.0, 54.0, 56.0, 58.0, 60.0],
        "rotation_tilt_axis": -5,
        "gold_bead_size_in_nm": 9,
        "template_matching_binning": 8,
        "binnings": [2, 4, 8],
        "reconstruction_thickness": 1400,
        "as_boxes": false
    },
    "MetaData": {
    },
    "CreateStacks": {
    },
    "DynamoTiltSeriesAlignment": {
    },
    "DynamoCleanStacks": {
    },
    "BatchRunTomo": {
        "skip_steps": [4],
        "ending_step": 6
    },
    "StopPipeline": {
    },
    "BatchRunTomo": {
        "starting_step": 8,
        "ending_step": 8
    },
    "GCTFCtfphaseflipCTFCorrection": {
    },
    "BatchRunTomo": {
        "starting_step": 10,
        "ending_step": 13
    },
    "BinStacks": {
    },
    "Reconstruct": {
    },
    "DynamoImportTomograms": {
    },
    "EMDTemplateGeneration": {
        "template_emd_number": "3420",
        "flip_handedness": true
    },
    "DynamoTemplateMatching": {
    },
    "TemplateMatchingPostProcessing": {
        "cc_std": 2.5
    },
    "DynamoAlignmentProject": {
        "iterations": 3,
        "classes": 4,
        "use_symmetrie": false,
        "use_noise_classes": true
    },
    "DynamoAlignmentProject": {
        "iterations": 3,
        "classes": 2,
        "use_noise_classes": true,
        "use_symmetrie": false,
        "selected_classes": [1]
    },
    "DynamoAlignmentProject": {
        "iterations": 3,
        "classes": 4,
        "use_noise_classes": true,
        "use_symmetrie": false,
        "selected_classes": [1]
    },
    "DynamoAlignmentProject": {
        "iterations": 3,
        "classes": 4,
        "use_symmetrie": false,
        "use_noise_classes": true,
        "selected_classes": [1]
    },
    "DynamoAlignmentProject": {
        "iterations": 3,
        "classes": 4,
        "use_noise_classes": true,
        "use_symmetrie": false,
        "selected_classes": [1]
    },
    "DynamoAlignmentProject": {
        "iterations": 3,
        "classes": 4,
        "use_noise_classes": true,
        "use_symmetrie": false,
        "selected_classes": [1]
    },
    "DynamoAlignmentProject": {
        "iterations": 3,
        "classes": 3,
        "use_noise_classes": true,
        "use_symmetrie": false,
        "selected_classes": [1],
        "box_size": 1.10,
        "binning": 4
    },
    "StopPipeline": {
    },
    "DynamoAlignmentProject": {
        "classes": 1,
        "iterations": 1,
        "use_noise_classes": false,
        "swap_particles": false,
        "use_symmetrie": false,
        "selected_classes": [1],
        "binning": 4,
        "threshold":0.8
    },
    "DynamoAlignmentProject": {
        "classes": 1,
        "iterations": 1,
        "use_noise_classes": false,
        "swap_particles": false,
        "use_symmetrie": false,
        "selected_classes": [1,2],
        "binning": 2,
        "dt_crop_in_memory": 0,
        "threshold":0.9
    },
    "BinStacks":{
        "binnings": [1],
        "use_ctf_corrected_aligned_stack": false,
        "run_ctf_phaseflip": true
    },
    "Reconstruct": {
        "reconstruct": "unbinned"
    },
    "DynamoAlignmentProject": {
        "classes": 1,
        "iterations": 1,
        "use_noise_classes": false,
        "swap_particles": false,
        "use_symmetrie": false,
        "selected_classes": [1,2],
        "binning": 1,
        "threshold":1
    }

Error message

line 17: /TomoBEAR-0.2.0/TomoBEAR/for_redistribution_files_only/run_TomoBEAR.sh: Not a directory

To Reproduce Steps to reproduce the behaviour:

1. Run TomoBEAR with Slurm execution (either with the compiled script TomoBEAR or the run_TomoBEAR.sh script).
2. Job is submitted but the job fails with the error above.

Expected behavior The job is started locally in the node allocated through Slurm.

• TomoBEAR Version: 0.2.0
• MotionCor2 Version: 1.6.3
• Gctf Version: 1.06
• Dynamo Version: 1.1.532