feat: ensembl reference collections

[mauripops]

feat: collections optional reference parameter(i.e. bacterial references)

This PR builds on the recently pushed (https://github.com/snakemake/snakemake-wrappers/pull/2928) where branches and specific url can be defined as parameters. The previous method was unable to find data from bacteria ensembl due to data in it containing one more folder level. There is also a fix for the spec variable in in ensembl-sequence where the release version would not match for branches other than the default.

QC

• [x] I confirm that:

For all wrappers added by this PR,

• there is a test case which covers any introduced changes,
• input: and output: file paths in the resulting rule can be changed arbitrarily,
• either the wrapper can only use a single core, or the example rule contains a threads: x statement with x being a reasonable default,
• rule names in the test case are in snake_case and somehow tell what the rule is about or match the tools purpose or name (e.g., map_reads for a step that maps reads),
• all environment.yaml specifications follow the respective best practices,
• the environment.yaml pinning has been updated by running snakedeploy pin-conda-envs environment.yaml on a linux machine,
• wherever possible, command line arguments are inferred and set automatically (e.g. based on file extensions in input: or output:),
• all fields of the example rules in the Snakefiles and their entries are explained via comments (input:/output:/params: etc.),
• stderr and/or stdout are logged correctly (log:), depending on the wrapped tool,
• temporary files are either written to a unique hidden folder in the working directory, or (better) stored where the Python function tempfile.gettempdir() points to (see here; this also means that using any Python tempfile default behavior works),
• the meta.yaml contains a link to the documentation of the respective tool or command,
• Snakefiles pass the linting (snakemake --lint),
• Snakefiles are formatted with snakefmt,
• Python wrapper scripts are formatted with black.
• Conda environments use a minimal amount of channels, in recommended ordering. E.g. for bioconda, use (conda-forge, bioconda, nodefaults, as conda-forge should have highest priority and defaults channels are usually not needed because most packages are in conda-forge nowadays).

Summary by CodeRabbit

Release Notes

• New Features

  • Added optional branch and collection parameters for Ensembl annotation and sequence downloads
  • Introduced new rules for retrieving off-branch annotations and genome sequences
  • Enhanced configuration flexibility for FTP server data retrieval

• Improvements

  • Updated logging paths for genome and annotation retrieval rules
  • Improved URL construction to support custom collections and branches

• Testing

  • Added new test cases for off-branch annotation and sequence retrieval

[github-actions[bot]]

This PR was marked as stale because it has been open for 6 months with no activity.

[coderabbitai[bot]]

📝 Walkthrough

Walkthrough

The pull request introduces enhanced configurability for Ensembl annotation and sequence retrieval processes. The changes add two new optional parameters, branch and collection, to the configuration files for both annotation and sequence entities. These modifications allow users to specify different branches and collections when downloading data from ENSEMBL FTP servers. The implementation includes updates to metadata files, Snakefiles, wrapper scripts, and test cases to support these new parameters, providing more flexibility in retrieving genomic data from specific branches and collections.

Changes

File	Change Summary
bio/reference/ensembl-annotation/meta.yaml	Added optional branch and collection parameters with example values
bio/reference/ensembl-annotation/test/Snakefile	Added get_off_branch_annotation rule, updated log path for get_annotation_gz rule
bio/reference/ensembl-annotation/wrapper.py	Added collection variable to modify URL construction
bio/reference/ensembl-sequence/meta.yaml	Added optional branch and collection parameters with example values
bio/reference/ensembl-sequence/test/Snakefile	Added get_off_branch_genome rule, updated log paths for chromosome retrieval rules
bio/reference/ensembl-sequence/wrapper.py	Added collection variable to modify URL construction
test.py	Added two new test functions: test_ensembl_sequence_off_branch and test_ensembl_off_branch_annotation_gtf

Sequence Diagram

sequenceDiagram
    participant User
    participant Snakefile
    participant Wrapper
    participant ENSEMBL FTP

    User->>Snakefile: Specify branch and collection
    Snakefile->>Wrapper: Pass branch and collection parameters
    Wrapper->>ENSEMBL FTP: Construct URL with branch/collection
    ENSEMBL FTP-->>Wrapper: Return requested data
    Wrapper->>User: Download and save data

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[fgvieira]

Ping @mauripops

[github-actions[bot]]

This PR was marked as stale because it has been open for 6 months with no activity.

[fgvieira]

@mauripops can you fix the conflicts?