WES Endpoint Collections

[rishidev]

We would like to keep a record of WES deployments and endpoints but need to clarify where such a list (or possibly lists) would be stored and how they would be used.

Some motivations behind this are:

1. This would give us a list of places to contact with updates if required
2. It would give people wishing to run a workflow (from Dockstore) a list of places to try running a workflow
3. It could possibly give people wishing to set up their own end-point a list of people who can assist with this.

These may not be the same list, and there has also been an expression of interest with putting these into a service-registry

[denis-yuen]

Heads-up @wshands The human-readable list of exceptions can go here too

[jaeddy]

#119 is directly related to this goal, and ga4gh/cloud-interop-testing#52 provides a decent starting point. Some manual work is probably needed to clean up the current list, but should be possible to automate in the future.

[briandoconnor]

Ideas about how this can evolve over time:

• list of WES endpoints and drivers that built them
• a registry so Dockstore and others can show more information about the WES endpoint when that workflow ran successfully
• longer term, having a nice, user facing WES registry with information on how to get an account.

[jaeddy]

Revisiting this issue. We're planning to collect information about WES implementations in this sheet; some details for endpoints used for the Basel testbed were previously captured here.

Here's a rough list of the attributes that we want to collect for each implementation. Note: some fields might apply at multiple levels — e.g., transfer protocol: if the server uses a master node to receive WES requests and deploy workflows via https, but individual jobs are run via a Cromwell instance that only supports gs buckets.

• Server
• Base URL
• Developer/affiliation
• Workflow engine(s) (e.g., Cromwell, Arvados, Toil, Seven Bridges) 
  • Language (e.g., CWL, WDL) - Language version (e.g. 1.0.0, draft-3) 
  • Transfer protocol / filesystem supported (e.g., https, s3, gs, keep) 
  • Cloud platform (e.g., AWS, Google, Azure)
  • WES specification version
• Endpoint version (e.g., if the maintainer is using semvar for their stack)
• Architecture (e.g., is WES a native endpoint in the system's API? are they using a local adapter to translate WES calls to different API(s)? are they using something remote like Lambda/API Gateway?)
• Access/authorization description (e.g., token, API key, encryption, white-listing, oauth, etc.)
• Endpoint status (e.g., health, development stage)

[ruchim]

I feel the sheet linked by @jaeddy earlier gets to the heart of keeping a list of known WES endpoints. @briandoconnor -- if you can look it over, we can resolve this ticket based on whether it needs anything else. If yes, my recommendation is to close this ticket.

[uniqueg]

Given that there's now a GA4GH service registry (API, reference implementation, reference deployment (API root), presentation), this may now be pretty much redundant.

I do feel though that the central registry deployment should be more prominently advertised across the various spec repos etc.

[ruchim]

@jb-adams -- I'm trying to see if there's anything worth left doing here before considering this issue completed?

[jb-adams]

@briandoconnor and I met over hackathon week to get some implementations and services registered for WES (and other cloud APIs), but we haven't finished registering the things we're aware of. So maybe good to leave this ticket open and as "in progress" for tracking purposes

[jb-adams]

@ruchim in the FASP hackathon doc , we have a rough list of WES implementations/services we want to register in the API. It's under ID 7. If there are any more you're aware of feel free to add to make our list complete!

[ruchim]

added some that were mentioned here https://docs.google.com/spreadsheets/d/1h4eSWn3mAZcJGF-YwPKbx7q4AfV6e6q78nuDOBaiIHM/edit#gid=671434367 < though it may be outdated