Add api for async

[FredericBGA]

Async mode is now available thought API

It seems to work, I can access to /api/async/fasta_sequences/<>/<>?STATUS=OK&URL=<>&key=<>.

There is still a lot of work to do or review:

• [ ] Functionality should not be copied and pasted between controllers
• [ ] API additions are more permanent changes to Galaxy than many other potential changes and so a second opinion on API changes should be sought.
• [ ] New API functionality should include functional tests.
• [ ] Changes to reflect modifications to the API should be pushed upstream to the BioBlend project if possible.

[FredericBGA]

@mvdbeek I need a little help to go further. This PR works here for my first use case:

• the first step /async/{tool_id} is used

• then I use the API /api/async/{tool_id}/{data_id}/{data_secret} in order to go through the authentication

I need for another use case to be able to add /api/async/{tool_id} but it fails. It fails I think because when I use the API link (with apikey), the trans.history is not defined here: https://github.com/galaxyproject/galaxy/blob/31945793926a93411ad35d0e1a8e4af3f0deec00/lib/galaxy/webapps/galaxy/api/async.py#L141

It works when I try to use the URL /api/async/{tool_id} using a web browser (where I'm already authenticated). Do I have a way to get the current history when I come from the API?

edit: maybe something like that api/histories/most_recently_used? but from trans.sa_session

thank you.

[mvdbeek]

Hey @FredericBGA, happy new year!

What is the error that you see ? trans.history should only be None for anonymous users, but that said an API endpoint should probably require a history id, and then use that history.

It would also be helpful if you can update the PR description with the motivation for the PR. I remember we discussed something about this on gitter but I forgot and the gitter search is not that helpful.

[FredericBGA]

Happy new year @mvdbeek.

The error I see, with few debug print (with URL /api/async/{tool_id}?key=#######):

galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:45:03,761 [p:30817,w:2,m:0] [uWSGIWorker2Core6] async dataid -> None galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:45:03,762 [p:30817,w:2,m:0] [uWSGIWorker2Core6] FRED HERE no data_id must be parameter submission galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:45:03,762 [p:30817,w:2,m:0] [uWSGIWorker2Core6] FRED HERE primer3plus query ? galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:45:03,768 [p:30817,w:2,m:0] [uWSGIWorker2Core6] FRED HERE <galaxy.model.HistoryDatasetAssociation(None) at 0x7f8d9f5c78d0> galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:45:03,768 [p:30817,w:2,m:0] [uWSGIWorker2Core6] FRED trans.history <galaxy.web.framework.webapp.GalaxyWebTransaction object at 0x7f8d9c779410> None galaxy.web.framework.decorators ERROR 2020-01-07 11:45:03,769 [p:30817,w:2,m:0] [uWSGIWorker2Core6] Uncaught exception in exposed API method: Traceback (most recent call last): File "lib/galaxy/web/framework/decorators.py", line 282, in decorator rval = func(self, trans, *args, **kwargs) File "lib/galaxy/webapps/galaxy/api/async.py", line 153, in index trans.app.security_agent.set_all_dataset_permissions(data.dataset, trans.app.security_agent.history_get_default_permissions(trans.history)) File "lib/galaxy/model/security.py", line 791, in history_get_default_permissions for dhp in history.default_permissions: AttributeError: 'NoneType' object has no attribute 'default_permissions'

If I use the same URL (without the key, but in a browser where I'm already authenticated):

galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:48:47,136 [p:30817,w:2,m:0] [uWSGIWorker2Core1] async dataid -> None galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:48:47,136 [p:30817,w:2,m:0] [uWSGIWorker2Core1] FRED HERE no data_id must be parameter submission galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:48:47,137 [p:30817,w:2,m:0] [uWSGIWorker2Core1] FRED HERE primer3plus query ? galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:48:47,145 [p:30817,w:2,m:0] [uWSGIWorker2Core1] FRED HERE <galaxy.model.HistoryDatasetAssociation(None) at 0x7f8d9c787810> galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:48:47,167 [p:30817,w:2,m:0] [uWSGIWorker2Core1] FRED trans.history <galaxy.web.framework.webapp.GalaxyWebTransaction object at 0x7f8d9c76dd90> <galaxy.model.History(13) at 0x7f8d9c883d90> galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:48:47,215 [p:30817,w:2,m:0] [uWSGIWorker2Core1] FRED HERE after trans.sa_session.add(data) <galaxy.web.framework.webapp.GalaxyWebTransaction object at 0x7f8d9c76dd90> galaxy.webapps.galaxy.api.async DEBUG 2020-01-07 11:48:47,342 [p:30817,w:2,m:0] [uWSGIWorker2Core1] FRED HERE after trans.history.add_dataset

I will update my motivations, but quickly:

• the first use case (that works) was to add a button in a HTML Blast report (in order to grab sequences). I need the API (/api/async/{tool_id}/{data_id}/{data_secret}) because Galaxy is authenticated and the server that creates the fasta file need to contact Galaxy with STATUS=OK&URL=#### to this API URL.
• the second use case (WIP) is to grab primer3plus results from Galaxy (I try to set up https://galaxyproject.org/admin/internals/data-sources/#asynchronous-data-depositing). I need here an API also for /api/async/{tool_id} because I need primer3plus to be able to get in touch with Galaxy.

[FredericBGA]

I managed to do something that works. trans.galaxy_session seems to be always None as I come from an API request. So I can't get user and history from it. I've then manually fetched the most recent history (just like in lib/galaxy/web/framework/webapp.py):

my_history = trans.sa_session.query(trans.app.model.History).filter_by( user=trans.get_user(), deleted=False).order_by(trans.app.model.History.update_time.desc()).first()

[FredericBGA]

Here is another version of this PR. As our Galaxy is authenticated, we need an API endpoint for Asynchronous processes. Otherwise we are always redirected to the login page when the asynchronous server tries to reach Galaxy.

• /api/async/{tool_id} is used for the first connection, when you need to get in touch with Galaxy.
• /api/async/{tool_id}/{data_id}/{data_secret} is used for the second connection, we you want to tell Galaxy where are located the data.

using the PR we managed to launch Primer3Plus (https://github.com/primer3-org/primer3plus) from Galaxy, pick primers and then get the results within the galaxy current history.

[nsoranzo]

@FredericBGA Is this ready from your side? It was marked WIP at a certain point (by @martenson).

[FredericBGA]

@FredericBGA Is this ready from your side? It was marked WIP at a certain point (by @martenson).

Yes. I've tried to follow the guidelines (https://docs.galaxyproject.org/en/master/api/guidelines.html). I used the four bullet points (good pratices) in order to create the task list:

• [ ] Functionality should not be copied and pasted between controllers

I think that this one is OK. There is no other controllers with the same goal.

• [ ] API additions are more permanent changes to Galaxy than many other potential changes and so a second opinion on API changes should be sought.

I was waiting for this second opinion.

• [ ] New API functionality should include functional tests.

This one is not very easy, I don't know how/what to do.

• [ ] Changes to reflect modifications to the API should be pushed upstream to the BioBlend project if possible.

Do you think that Bioblend needs this? The asynchronous data depositing (https://galaxyproject.org/admin/internals/data-sources/#asynchronous-data-depositing) is not widely used I guess...

[mvdbeek]

I think that this one is OK. There is no other controllers with the same goal.

Correct me if I'm wrong, but looks like it is copy-pasted from https://github.com/galaxyproject/galaxy/blob/73b56c5ac1d3ac44114a2ebb6ac1f8f773b6581d/lib/galaxy/webapps/galaxy/controllers/async.py#L23

I don't think the API should return a filled-in mako. I still haven't understood what is wrong with the async controller ? Can you not authenticate the existing web controller route ?

[FredericBGA]

I think that this one is OK. There is no other controllers with the same goal.

Correct me if I'm wrong, but looks like it is copy-pasted from

https://github.com/galaxyproject/galaxy/blob/73b56c5ac1d3ac44114a2ebb6ac1f8f773b6581d/lib/galaxy/webapps/galaxy/controllers/async.py#L23

I don't think the API should return a filled-in mako.

sorry, I try to do my best, so yes you're right. I meant no other controller related to an API entry. It's far for being perfect but it does what I need: launch primer3 cgi and then the results are stored in a file in the user's history. Using the regular controller I didn't understand how to go through authentication.

I still haven't understood what is wrong with the async controller ? Can you not authenticate the existing web controller route ?

sorry again, I've try(https://github.com/galaxyproject/galaxy/pull/8570#issuecomment-573602443) to explain that using the examples found here: https://github.com/hexylena/galaxy-data_source-examples, each time the primer3 server needs to get in touch with Galaxy (return OK, return the URL where the data can be fetch), the URL was redirected to the login page. If there is another way to handle this I will be happy to test and improve the way I do.

It works, I can launch primer3 from Galaxy, use the primer3 interface (with a lot of parameters, tabs, help) and it goes back to Galaxy (with the results in your current history) when the user hits the "Pick Primers" button.

[FredericBGA]

@mvdbeek Hi Marius, we can close this PR if you think that if does not fit the galaxy needs. Or maybe I can try a bit more to understand how to go through my authentication issue. You were talking about authenticate the existing web controller route. Can you help me to dig into this?

Or maybe as the code of api/async.py is the same as controllers/async.py, can we only set the decorators from @web.expose to @expose_api (I see some @web.legacy_expose_api in some controllers). It will avoid this duplication of code. Let me know, it's not a big deal for me to keep this only for our galaxy.