Plugin loading without setup.sh

[abretaud]

Hi, In recent JBrowse version (1.15.x, 1.16), to add a plugin we have to run setup.sh to rebuild all the js stuff. In older version it was possible to add some plugins in a much simpler way (no need to run setup.sh), for example:

   "plugins" : [
      {
         "location" : "https://cdn.rawgit.com/elsiklab/gccontent/5c8b0582ecebf9edf684c76af8075fb3d30ec3fa/",
         "name" : "GCContent"
      },
      {
         "location" : "https://cdn.rawgit.com/TAMU-CPT/bookmarks-jbrowse/5242694120274c86e1ccd5cb0e5e943e78f82393/",
         "name" : "Bookmarks"
      },
      {
         "icon" : "https://galaxyproject.org/images/logos/galaxy-icon-square.png",
         "location" : "https://cdn.rawgit.com/Arabidopsis-Information-Portal/ComboTrackSelector/52403928d5ccbe2e3a86b0fa5eb8e61c0f2e2f57",
         "name" : "ComboTrackSelector"
      },
      {
         "location" : "https://cdn.rawgit.com/TAMU-CPT/blastview/97572a21b7f011c2b4d9a0b5af40e292d694cbef/",
         "name" : "BlastView"
      }
   ],

This was used by the Galaxy JBrowse tool .

I know there are very good reasons to have switched to this new js build mechanism, but I was wondering if it would be possible in future versions of JBrowse to be able to add plugins more dynamically. There is a great catalog of plugins in https://gmod.github.io/jbrowse-registry, it would be really great to be able to add (and remove) them quickly to a deployed instance.

[abretaud]

(ping @erasche as we discussed it in https://github.com/galaxyproject/tools-iuc/pull/1683)

[cmdcolin]

This would be awesome. The issue is basically that webpack wants all of the dependencies available at compile time. There might be some workarounds possible but I dont actually know what approach would be best...I think the plugins would probably also need webpack bundling to interact with the main module

[hexylena]

xref https://github.com/GMOD/jbrowse/issues/981

[cmdcolin]

Possible inspiration sources for doing this

https://github.com/OpenNTF/dojo-webpack-plugin/issues/191 (not sure if this is relevant still) https://github.com/Esri/arcgis-webpack-plugin (I think they are build time but have an advanced dojo-webpack-plugin usage) https://webpack.js.org/plugins/dll-plugin/ (not sure if this is relevant but maybe plugin can build against jbrowse this way?)

[hexylena]

@abretaud https://rawgit.com/ well that super sucks.

[abretaud]

Yep, that's bad... So the plugins are broken in the galaxy tool at the moment... Do you have an idea where we could host it to try to fix the last release of the jbrowse?

[cmdcolin]

Maybe https://raw.githack.com/ works?

[abretaud]

@cmdcolin githack.com did not work, but cdn.jsdelivr.net looks ok Do you have an idea if finding a solution for this issue could be long or not? Not to rush you, I'm just wondering if we'd better wait for it for https://github.com/galaxyproject/tools-iuc/pull/1683, or if we should find a plan b

[cmdcolin]

I do not necessarily see a short term solution. I considered maybe the idea of webpack DLL to allow both plugin to depend on but I don't know exactly the path for that

[cmdcolin]

For the time being it looks like this will simply not be possible. We'll close for now. If anyone wants to experiment they are welcome but it's seems not roadmappable right now

[hexylena]

Even if there isn't a short term solution, shouldn't we leave this open? Since it's of interest to the community? It's still an issue, even if it can't be placed on the project roadmap currently.

Likewise the JBrowse registry becomes more difficult to run if we have to compile in all of the demos, currently it only continues to function because of the older JBrowse in use.

[cmdcolin]

I could re open but this was essentially just a design decision that was made when we chose web pack. I'll reopen for posterity. Again welcome to give it a try

[laceysanderson]

Just pinging this issue. The current process of installing plugins makes it very difficult to add plugins to an existing JBrowse. Specifically, I have a number of JBrowse 1.16.3 instances and would like to add the fullscreen plugin. We need the functionality of the plugin but I do not feel comfortable switching a production JBrowse to -dev and re-running setup.sh.

I think finding a solution is critical otherwise your large repository of plugins becomes essentially useless.

[cmdcolin]

Is there anything that is particularly problematic about running setup.sh with a dev build?

[laceysanderson]

It's more so switching from a production build to a dev build that makes me nervous. I believe I would have to rebuild my JBrowse from scratch just to add a plugin? or is there an easier way?

[cmdcolin]

The current model (since 1.13.0) is that it automatically rebuilds jbrowse from the source code when ./setup.sh if you are using a dev build or a github clone. This is done using webpack behind the scenes. We have tried to make this process as painless as possible but it is technically "rebuilding it from source code" when you do the ./setup.sh

We use things like yarn.lock to make sure the outputted build stable and reproducible when you perform this step.

[laceysanderson]

But if I already have a production site which was not built from a github clone, I would have to create a fresh JBrowse instances from a github clone and reload all my data?

[cmdcolin]

@laceysanderson you would not need to rebuild any data directories. There have not been any changes to jbrowse that affect backwards compatibility so it should not require any reloading or reprocessing of data. I am not sure precisely how you have it setup but the process would likely be that you move your "data" directory from the old jbrowse folder into a the new jbrowse folder e.g. the dev version. Then put the plugins in the new jbrowse folder's plugins directory, and re-run setup.sh. The setup.sh will not affect any of your data directories

[cmdcolin]

Possibly also copy the jbrowse.conf/jbrowse_conf.json if you use those global configs