planemo uploads to the Tool Shed include unwanted files

[gregvonkuster]

It seems planemo is uploading unwanted files to the Tool Shed with individual tool uploads. In some cases it uploads multiple tools from a demultiplexed git repository into a single tool repository, while in other cases it uploads unwanted files from other tools.

The individual ImageJ2 tools are demultiplexed in here: https://github.com/bgruening/galaxytools/tree/master/tools/image_processing/imagej2 Planemo uploaded each of these tools to the TTS, including every ImageJ2 tool in each individual repo:

Repository 'imagej2_adjust_threshold_binary' revision c05d862ac753 (repository tip)
Contents:
imagej2_adjust_threshold_binary.py
imagej2_adjust_threshold_binary.xml
imagej2_adjust_threshold_binary_jython_script.py
imagej2_analyze_particles_binary.py
imagej2_analyze_particles_binary_jython_script.py
imagej2_analyze_skeleton.py
imagej2_analyze_skeleton_jython_script.py
...

On the MTS, the meme tool repo contains the fimo tool's Python script:

Repository 'meme_meme' revision 487ce3fa1822 (repository tip)
Contents:
all_fasta.loc.sample
fimo_wrapper.py
meme.xml
test-data/
tool_data_table_conf.xml.sample
tool_dependencies.xml

There are other similar examples on both the TTS and MTS.

[jmchilton]

I'm pretty much okay with this as default behavior. Despite my desire to do something about this for years - tools still don't have a way to specify tool files they require - so planemo just grabs pretty much everything that isn't a tool by default since these files don't hurt anything.

I consider the tool shed tarballs pretty much a build artifact and a black box - so there is no need in my mind to customize them extensively as long as they work. I understand that you and say @peterjc may care more about the tarball though - so it is possible to include and exclude files via directives in .shed.yml.

• http://planemo.readthedocs.io/en/latest/standards/docs/best_practices/shed_yml.html#shed-upload-includes-excludes
• https://github.com/peterjc/galaxy_blast/blob/master/tools/blast2go/.shed.yml#L22

A future version of planemo and tools - may require that newer tools (using tool profile versions) require the tools annotate the files they need (this would be wonderful for pulsar) - and the default de-multiplexing behavior may change at that time (but probably only for these newer tools).

[peterjc]

Yeah, I went for explicitly including the files via the .shed.yml manifest as I felt the default was too much magic.

[martenson]

Duplicating the whole test-data folder for each repository for mothur enlarges the suite installation size from ~10MB to ~600MB (for ~120 tools).

Also it makes the repositories themselves more confusing and less standalone - unless you know the planemo magic behind, which shouldn't be the requirement.

so 👍

[bernt-matthias]

May one of you have a look at https://github.com/galaxyproject/planemo/pull/1106?