looper
looper copied to clipboard
`looper.write_sample_yaml` pre-submit function assumes the sample has `sample_name` as an attribute
trafficstars
I was trying to submit jobs by leveraging the write_sample_yaml pre-submit function. This was my pipeline_interface.yaml:
pipeline_name: cranger-atac
pipeline_type: sample
pre_submit:
python_functions:
- looper.write_sample_yaml
command_template: >
python {looper.piface_dir}/run.py {looper.output_dir}/submission/{sample.id}_sample.yaml
However, before any submission scripts are written, I get the following stack trace:
Traceback (most recent call last):
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/bin/looper", line 8, in <module>
sys.exit(main())
^^^^^^
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/cli_pydantic.py", line 343, in main
return run_looper(args, parser, test_args=test_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/cli_pydantic.py", line 265, in run_looper
return run(
^^^^
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/looper.py", line 475, in __call__
curr_pl_fails = cndtr.add_sample(sample, rerun=rerun)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/conductor.py", line 372, in add_sample
self.submit()
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/conductor.py", line 409, in submit
script = self.write_script(self._pool, self._curr_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/conductor.py", line 633, in write_script
namespaces = _exec_pre_submit(pl_iface, namespaces)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/conductor.py", line 734, in _exec_pre_submit
_update_namespaces(namespaces, func(namespaces))
^^^^^^^^^^^^^^^^
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/plugins.py", line 156, in write_sample_yaml
sample["sample_yaml_path"] = _get_yaml_path(
^^^^^^^^^^^^^^^
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/looper/conductor.py", line 74, in _get_yaml_path
or f"{namespaces['sample'][SAMPLE_NAME_ATTR]}"
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/sfs/gpfs/tardis/home/xjk5hp/code/scripts/model-training/atacformer-scatlas-base/.venv/lib/python3.11/site-packages/peppy/simple_attr_map.py", line 26, in __getitem__
return self._mapped_attr[item]
~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'sample_name'
You can follow the breadcrumbs and look into this _get_yaml_path function to see that it uses a hard-coded sample name attribute key that comes from peppy.const. I believe this is a bug, correct? I've renamed my sample_table_index column name to id.
I am able to resolve this issue by reverting and making my sample_table_index by sample_name.