Add onedata objectstore

[bwalkowi]

Object store plugin for Onedata distributed data management system (https://onedata.org/). Requires Onedata version >= 21.02.5.

How to test the changes?

(Select all options that apply)

• [x] I've included appropriate automated tests.

• [ ] This is a refactoring of components with existing test coverage.

• [x] Instructions for manual testing are as follows:

  1. Add the onedatafilerestclient package to the currently used requirements (can be found in conditional-requirements.txt).

  2. Create object_store_conf.xml file in the config dir, copying the part concerning Onedata from object_store_conf.xml.sample. Fill in the config (access_token, onezone_domain, space, disable_tls_certificate_validation - you may want True - see below).

     You will need an account in a Onedata system instance, however, version 21.02.5 is not officially released yet. The best way to test is to use the "demo mode", which allows easily setting up a dockerized Onedata environment: https://onedata.org/#/home/documentation/topic/stable/demo-mode

  3. Create galaxy.yml file in the config dir with the following content:

  galaxy:
    object_store_config_file: object_store_conf.xml
  

  4. Start the galaxy server, log in.

  5. Upload some data and run some jobs. Open the Onedata UI and navigate to your space to see the stored Galaxy datasets.

License

• [x] I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[mvdbeek]

This looks pretty cool, anything we can do to help ?

[bwalkowi]

Hello, sorry, I accidentally opened it too early. Although this is a working prototype, we are still working on improvements. In a few days, when the final version is ready, I will reopen the pr.

[bwalkowi]

@mvdbeek it's finally ready - I'm reopening the pr.

[bwalkowi]

I specified the Onedata object store dependencies as optional so that they would not be installed by default and that is why the automated tests failed. Is there a specific requirements.txt file used during testing that I can add them to?

[bgruening]

@bwalkowi can you please fix the linting errors and update with the main branch. Thanks!

[bwalkowi]

@bgruening done!

[jmchilton]

The integration test failures look legitimate.

ObjectStore configured to use Onedata, but no OnedataFileRESTClient dependency available. Please install and properly configure Onedata or modify Object Store configuration.

The conditional dependency handling looks right though - I don't know immediately what to do.

[mvdbeek]

Conditional dependencies that we're actually testing need to be added to https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/dependencies/dev-requirements.txt and https://github.com/galaxyproject/galaxy/blob/dev/pyproject.toml#L129

[bwalkowi]

nice. It seems everything but code scanning is green. Do you have any recommendations regarding Clear-text logging of sensitive information ? I do not think what is logged is that sensitive and it seems some other files have similar problems.

[mvdbeek]

You haven't touched these, so no need to change that here.

[github-actions[bot]]

This PR was merged without a "kind/" label, please correct.

[jmchilton]

I made a bunch of changes to this with the PR #18174 . By using a new superclass for caching object stores there was a lot less duplication with other object stores and you should have gotten the bug fixes we put into #18117. We merged that PR but I would re-run through it with your setup and let us know if anything broke and send patches. Thanks so much for the contribution!

[bwalkowi]

nice! sure thing, I will test it and try to send eventual patches within a week.