specification
specification copied to clipboard
SignalK
Webapp packaging
We should define how webapps are packaged to be compatible with the Signal K ecosystem and work on multiple servers.
Node server's current npm based webapp delivery assumes that the installed package contains html files either directly under root or under public directory.
Assuming that a webapp is just a package of the webapp files is too simplistic imho. This means that the package can not contain anything that is not published over http.
Examples of stuff that we will or might need in addition to the HTML5 app files themselves:
- general metadata: name, description, author, version etc (no harm in putting these inside the app though, but they need a well defined location)
- security settings: how should the app or parts thereof be treated in terms of security
- dependencies: SK api version ranges, certain functionality (like subscription support), other webapps etc
- combined webapp and serverside package, containing not only the webapp files but also java/c++/js/kotlin whatnot code to be run
The apps must use only relative urls, making them agnostic to the url where they are mounted. If we take less SK centric view the apps may be installed in a system that just happens to have a SK api, with server root and any fixed prefix like /apps already populated by something else.
The argument that it should be possible to just expand the packages under an http server's static directory is too simplistic in my view and can be circumvented with an index.html that redirects to ./public/.
Widely used web servers like Apache and Nginx can be configured to serve the public directories, not unlike the system where public_html directories under users' home directories were automounted as .../~userid/.
Any app store like functionality will download and expand the packages programmatically. That functionality can choose to expand just the public directories of the packages.
I don't think the simplicity of just expand stuff under static files home outweighs being more future proof by making it possible to include all kinds of functionality, not just HTML5 app files, in the package.
Agree that the app should provide an index.html in the app root. Using a redirect in there is fine too as long as that sets basedir or any other required params.
Configuring a server to look for ./public is always possible, but has no particular advantage over an index.html in root, and is more complex so I dont see the value. In any case so long as the location is fixed, and std for all apps, then it works.
I think that if an app needs more than just static files (eg a server component) then it can be better handled by creating a separate server package. This allows each server to create a package based on its architecture, and be maintained by different teams, rather than bundling everything into a single app.
Along those lines the maven/nexus style repositories (https://www.sonatype.com/products-overview) handle all kinds of resources (java, node, python, docker, etc) and may be more accommodating than npm, which is very node oriented.
Please note that there nothing npm specific in this issue. How apps are distributed is another issue.
Above I tried to argue the value of the package containing data that is not inherently part of the webapp itself. You skipped over all my points but one.
Are you suggesting that a webapp package is just the webapp files with index.html, nothing else? If that is the case then we can pretty much close this issue as there isn't much if anything to define.
I do see value in packaging that allows future extension by defining a non-root fixed location for the assets.
Sorry, I agree with your other points, that a package may need metadata, dependencies, etc. I didnt really consider that as part of the deployed package but it makes sense that its included in the distribution archive.
Those assets will need definition. Would a suitable formatted single json file suffice? Maybe something using a well known format like package.json. In fact if a known file is always present, eg an app.json in the root, then it can contain all the data required for deployment, which provides a lot of flexibility.
The servers install process can then set up the app appropriately
I propose we add the following REST API calls.
- GET /signalk/v1/apps/search?keyword=signalk-webapp searches npmjs.com, returns current npm/node-server json format, basically an array of package.json text.
- GET /signalk/v1/apps/list lists installed apps, returns current npm/node-server json format, basically an array of package.json text.
- GET /signalk/v1/apps/install/{appName}/{appVersion} installs app
- GET /signalk/v1/apps/upgrade/{id}/{appVersion} upgrades app
- GET /signalk/v1/apps/remove/{id}/{appVersion} removes app.
That should allow any client to search, list install etc on any server.
I also found that a simple npm install appname does not work unless you are already in a node/npm project. The install looks for package.json in the current dir or parent, and fails.
We need a way to download a simple archive (tar, zip,etc) which can then be unpacked on non-nodejs installations. I couldnt find one on npm/google
Works here:
tmp$ mkdir foo
tmp$ cd foo
foo$ npm install @signalk/freeboard-sk
npm WARN saveError ENOENT: no such file or directory, open '/private/tmp/foo/package.json'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN enoent ENOENT: no such file or directory, open '/private/tmp/foo/package.json'
npm WARN foo No description
npm WARN foo No repository field.
npm WARN foo No README data
npm WARN foo No license field.
+ @signalk/[email protected]
added 1 package from 1 contributor and audited 1 package in 4.562s
found 0 vulnerabilities
foo$ ls node_modules/@signalk/freeboard-sk/
LICENSE README.md bundle.js css fonts fonts.css img index.html index.js lib package.json
Yeah, bug at my end. Would still be better to download an archive tho, then my server would not be dependent on node/npm
AFAIK packages in npm are just tarballs and you can download the packages as such: https://stackoverflow.com/questions/33530978/download-a-package-from-npm-as-a-tar-not-installing-it-to-a-module/33549885
The reason to use npm was just convenience - if there are other good options for a free registry that allows people to publish their apps and that we can use for retrieving lists of available apps & versions and for download.
Is there a free Sonatype registry available? If not I don't see that as an option at this stage.
Side note: we will need to address abandoned packages, that are left in the registry and the publisher no longer maintains them, not even by deprecating them. This can be achieved for example by changing the keyword used to identify the webapps and updating all active ones.
Mostly that url assumes npm is installed. If not then use:
curl https://registry.npmjs.org/PACKAGE-NAME/ \ | jq '.versions[."dist tags".latest].dist.tarball'
Saving here as its easy to miss
https://www.deps.co/guides/public-maven-repositories/ is a good summary of the java ecosystem but all are java centric which is not much different to npm centric
In java land there is also jitpack.com which builds a release on the fly from any github java project, but it doesnt do npm ☹
If we can get useable tarball archives using just REST calls then npmjs.org is fine. Since we are looking for an html5 webapp solution here npm will also be the most familiar to the devs too.
For server packages we can use the repository of choice as the server is implementation specific anyway.
Ive set up swagger in the artemis server and have swagger definitions for the REST webapp API https://signalk-dev.slack.com/files/U02EUBYB6/FD3HNSPJ7/swagger_definition_for_signalk_webapp_management_api.txt
Ive been able to get the tarballs above without npm, and added support to artemis server for the API above. Also added swagger support so we can have self documenting APIs
Swagger is good, we are moving from schema additions to new rest apis, like authentication.
That said I don’t think server management apis like webapp installation api are very high priority compared to for example authentication. We need to get things done, into the spec, instead of just adding issues.
Like this one: webapp structure sounds like an appendix in the spec. Other ideas?
I did the authentication api late yesterday, bit rough yet, nned to add the cookie/token stuff. Slowly going through and swaggering all the APIs.
See swagger definition at https://github.com/SignalK/signalk-java/blob/artemis/signalk-static/docs/openapi.json When https://github.com/SignalK/signalk-java/tree/artemis is installed the swaggerUI is available at http://localhost:8080/docs/
So how do we add this to the specification? I think Rob's suggestion needs a bit of work - I don't think we are quite ready to add this to the spec yet.
First of all I think this is about webapps, not generic apps. Or if the path is apps there should be a way to consider webapps.
GET /signalk/v1/apps/search?keyword=signalk-webapp searches npmjs.com, returns current npm/node-server json format, basically an array of package.json text.
What are we searching for? I think this should be GET /signalk/v1/webapps/available. The fact that we are searching and it is npmjs (Node server is using now two backends btw) is incidental and I don't think clients need to know that. Otherwise clients could do the search themselves.
The response from current node-server is not package.json text, but like this:
},
{
"name": "@signalk/freeboard-sk",
"version": "1.0.0",
"description": "Openlayers chartplotter implementation for Signal K",
"author": "[email protected] (Signal K team)",
"npmUrl": "https://www.npmjs.com/package/%40signalk%2Ffreeboard-sk",
"isPlugin": false,
"isWebapp": true,
"installedVersion": "1.0.0"
},
{
GET /signalk/v1/apps/list lists installed apps, returns current npm/node-server json format, basically an array of package.json text.
GET /signalk/v1/webapps/installed is more to the point.
GET /signalk/v1/apps/install/{appName}/{appVersion} installs app
This MUST be a POST to prevent CSRF. `GET /signalk/v1/webapps/install/{appName}/{appVersion} appName MUST be urlEncoded, some npm package names contain forward slashes.
Installation is not instantaneous, so it should be handled like an async action so that the client can get notified / it can poll for completion.
GET /signalk/v1/apps/upgrade/{id}/{appVersion} upgrades app
Why not just use install?
GET /signalk/v1/apps/remove/{id}/{appVersion} removes app.
This MUST be POST to prevent CSRF. uninstall would be more in line with install. Async action as well.
ping @sbender9 @panaaj
@rob42 could you add a PR for a swagger definition of the API, taking my comments above into account?
Swagger PR - yes. Also considering a pact.io test of the API contract.
upgrade vs install, yes - I added upgrade thinking there may be a case where some different processing is required for upgrade, but thats probably unfeasible as app specific anyway. We need to note that apps are expected to self-initialize on install or upgrade if required, eg reformat data in local storage.
POST - yes. Brings up the issue of CSRF token handling by apps. Should we have a std header, and define that webapps use it if provided?