looper icon indicating copy to clipboard operation
looper copied to clipboard

`looper.write_sample_yaml` pre-submit function assumes the sample has `sample_name` as an attribute

Open nleroy917 opened this issue 9 months ago • 7 comments
trafficstars

I was trying to submit jobs by leveraging the write_sample_yaml pre-submit function. This was my pipeline_interface.yaml:

pipeline_name: cranger-atac
pipeline_type: sample
pre_submit:
  python_functions:
    - looper.write_sample_yaml
command_template: >
  python {looper.piface_dir}/run.py {looper.output_dir}/submission/{sample.id}_sample.yaml

However, before any submission scripts are written, I get the following stack trace:

Traceback (most recent call last):
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/bin/looper", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/cli_pydantic.py", line 343, in main
    return run_looper(args, parser, test_args=test_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/cli_pydantic.py", line 265, in run_looper
    return run(
           ^^^^
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/looper.py", line 475, in __call__
    curr_pl_fails = cndtr.add_sample(sample, rerun=rerun)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/conductor.py", line 372, in add_sample
    self.submit()
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/conductor.py", line 409, in submit
    script = self.write_script(self._pool, self._curr_size)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/conductor.py", line 633, in write_script
    namespaces = _exec_pre_submit(pl_iface, namespaces)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/conductor.py", line 734, in _exec_pre_submit
    _update_namespaces(namespaces, func(namespaces))
                                   ^^^^^^^^^^^^^^^^
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/plugins.py", line 156, in write_sample_yaml
    sample["sample_yaml_path"] = _get_yaml_path(
                                 ^^^^^^^^^^^^^^^
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/conductor.py", line 74, in _get_yaml_path
    or f"{namespaces['sample'][SAMPLE_NAME_ATTR]}"
          ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/peppy/simple_attr_map.py", line 26, in __getitem__
    return self._mapped_attr[item]
           ~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'sample_name'

You can follow the breadcrumbs and look into this _get_yaml_path function to see that it uses a hard-coded sample name attribute key that comes from peppy.const. I believe this is a bug, correct? I've renamed my sample_table_index column name to id.

I am able to resolve this issue by reverting and making my sample_table_index by sample_name.

nleroy917 avatar Jan 28 '25 19:01 nleroy917