automatically generate update manifest for self-hosted updates

[Azeirah]

trafficstars

It would be extremely helpful to have a command to generate an updateManifest.json within this web-ext tool, just like how jpm is able to do it. Right now, I'm using a custom script, but I feel it should be part of web-ext.

Is there any reason why web-ext doesn't automatically generate an updateManifest.json file during the sign task like jpm does it?

[kumar303]

Right now, I'm using a custom script

Why do you need a script for this? Is it helpful because it pre-populates the extension ID? I'd be curious to look at this because I can't think of anything else that is possible to generate automatically.

Just to provide more context for other readers, updates for WebExtensions are different from SDK based add-ons (made with jpm). This article explains how updates work. The first step is to add an update_url to the WebExtension manifest.

[Azeirah]

Why do you need a script for this?

I'm using a custom script because I want to automate the process from git push, on my webextension to automatically deploy the new version on my website if the whole pipeline -- test, build.. and then deploy -- is passed.

I am still working on this. The custom script is indeed based on the article you mentioned.

Am I wrong in assuming that it's possible to automate this process?

1. Add update_url to manifest.json
2. Create initial updateManifest.json file, and upload it to my webserver
3. Whenever I git push, check if version in manifest.json has changed
4. If so, fetch the old updateManifest.json
5. Add the details of the new version to the new updateManifest.json (which should be handled by web-ext, not by my custom script, in my opinion)
6. Upload the new updateManifest.json to my web server

All variables, such as the extension ID, web server URL, will be defined within the pipeline itself.

To clarify, the script takes three variables as input; the old updateManifest.json, the extension ID, and the URL where the newly generated XPI will be placed.

[kumar303]

Ah, thanks for the details. I understand this better.

Yes, I think the build command should produce updateManifest.json if the manifest includes update_url.

I don't think web-ext should try to push updateManifest.json to a remote server though. The details for how to do that are going to vary from server to server so it seems out of scope.

[kumar303]

Note that the sign command calls build automatically

[kumar303]

It may be easier to start this as a standalone command like web-ext update-info (or something)

[Azeirah]

I don't think web-ext should try to push updateManifest.json to a remote server though.

Oh no I completely agree, that'd be out-of-scope.

It may be easier to start this as a standalone command like web-ext update-info (or something)

I agree, because it differs from cfx's and jpm's commands in that it needs the old updateManifest.json file to work. Otherwise, I'd say include it in the build step.

[kumar303]

it needs the old updateManifest.json file to work

This should be straight forward because we can decide on a standard file name like web-ext-update-manifest.json or something. It could be overidden by an option too. I imagine developers would commit this file to their repo. The command could start by looking for the file and appending versions to it if it exists.

[Azeirah]

I imagine developers would commit this file to their repo

Not sure, if I understand the update page correctly, you can use a single web-ext-update-manifest.json file for multiple addons. It'd get complicated if you wanted to use this single file in multiple repositories, where the easiest solution would be a git submodule with only this file inside of it.

An object containing one entry for each add-on to be updated. For each such entry, the name of the property must be the add-on's ID, and the value must be an object describing the add-on and its updates.

I think it's a better idea to just let the user decide where to get this file from, be it that it's stored inside the repo, or downloaded from some webserver. And just pass the path of this file to the command

web-ext generateUpdateManifest [extensionID] [updatedXPIURL] [oldUpdateManifest.json]

[kumar303]

you can use a single web-ext-update-manifest.json file for multiple addons

Crap, so you can. That makes it a bit more complicated. I suppose the command will need to take an option then, like:

web-ext update-manifest --existing-manifest ~/my-shared-update-manifest.json

[Azeirah]

I don't think web-ext should try to push updateManifest.json to a remote server though.

Now I understand where you were coming from. The update_link property in the updateManifest should point to the newly generated XPI file, right?

{
    "addons": {
        "someAddonExtensionId": {
            "updates": [
                {
                    "version": "1.1",
                    "update_link": "https://example.com/plugins/plugin-version-1.1.xpi"
                }
            ]
        }
    }
}

The https://example.com/plugins/ needs to be filled in, no? That's what I meant with updatedXPIURL and URL where the newly generated XPI will be placed. in my earlier comments.

This shouldn't handle uploading, only fill in the url in the updateManifest.

[kumar303]

Yes, so, we would need to specify the update_link URL on the command line in some way like:

web-ext ... --update-link="https://example.com/plugins/${xpiFileName}"

I thought of something else. The script could parse applications.gecko.update_url from the manifest and if it's a valid URL, it could pull the existing update manifest JSON from that URL and append new versions to that.

When I think about this more, it might make sense to only bake this into the sign command since that's the only thing that produces a signed XPI for self-distribution anyway.

[Azeirah]

I thought of something else. The script could parse applications.gecko.update_url from the manifest and if it's a valid URL, it could pull the existing update manifest JSON from that URL and append new versions to that.

Hadn't thought of that. What would the script do if the manifest doesn't have the applications.gecko.update_url property? Throw an error? Or should it tell the user to provide the optional parameter --existing-manifest?

[kumar303]

if applications.gecko.update_url isn't in the manifest, no update manifest needs to be generated. I think it would be ok to expect the developer to add update_url manually when they want to serve updates. We could add documentation about this.

[Azeirah]

I want to let you know I'm working on it, and am making progress.

Here's where I'm currently at.

• [x] src/cmd/sign.js, add function generateNewUpdateManifest(...)
  • [ ] Fetch an updateManifest.json from the location pointed to by manifest.applications.gecko.update_url
  • [ ] Error messages (show the user an error message, and exit with error code)
    • [x] the old, fetched updateManifest.json is invalid
    • [x] unable to fetch updateManifest.json due to connection error
    • [x] argument passed to --update-link flag is invalid (== is missing {xpiFileName})
      Currently, I've issued error messages for the three above errors, but no exit with error code.
      
      
    • [ ] More potentially erroneus states?
  • [ ] Warning messages
    • [x] The user passed the --update-link flag, but manifest.json lacks an applications.gecko.update_url property
    • [x] The fetched updateManifest.json had no previous release of an addon with this extension-id. Warn the user that if this should not be the case to check her extension-id
    • [ ] More potential warnings?
• [x] program.js, add --update-link param to sign .command
• [ ] all code complies to flow
• [ ] all code complies to the linter
• [ ] test cases?
• [ ] documentation
  • [ ] add description of the --update-link param
  • [ ] add explanation on how to auto-generate updateManifest

I think it'd be helpful to do a short code-review after I've finished with the core task, so the code can be shaped to fit the codebase better, if you have the time?

[kumar303]

Great, thanks for getting it started! I can definitely give you an early review, just create a pull request and let me know when it's ready to look at. As for errors, you can just throw either a UsageError or WebExtError and the program will exit non-zero.

[Azeirah]

Great, thanks for getting it started! I can definitely give you an early review, just create a pull request and let me know when it's ready to look at.

Nice, I'll let you know when it's ready enough

As for errors, you can just throw either a UsageError or WebExtError and the program will exit non-zero.

Thanks, replaced my log.error calls.

@kumar303 I see that sign.js's sign function returns an object, and then I see that the sign function is exported in index.js like this

import build from './build';
import lint from './lint';
import run from './run';
import sign from './sign';
export default {build, lint, run, sign};

Does this mean that sign's return value is completely ignored? Is the return value important for anything at all?

[kumar303]

The sign() function from cmd/sign.js always returns a promise that resolves to a signingResult object. It is up to the caller to ignore it or not. Yes, it could be important for anyone who is using web-ext as a library rather than on the command line.

[Azeirah]

Yes, it could be important for anyone who is using web-ext as a library rather than on the command line.

Ah, that makes a lot of sense, because there is only one call to sign in the entire web-ext, and it's implicit in index.js, meaning its return value is ignored.

[Hypnosphi]

it might make sense to only bake this into the sign command since that's the only thing that produces a signed XPI for self-distribution anyway.

But what about private (corporate) extensions? It's recommended to use ESR or unbranded builds for them, but how to generate xpi's in that case?

[kumar303]

Hi @Hypnosphi . This issue is just about generating an update manifest. You should have no trouble running a private corporate extension on a release build using web-ext sign. An unbranded build is not required. If you run into trouble, feel free to file a separate issue.

[MarZab]

This issue looks dead so I've made a script to update the manifest from the xpi's themselves.

https://gist.github.com/MarZab/251cff66797b76b632d59cb88fdcc8a9