Managing common dependancies between the host app and engines

[gitmil]

We have a host app and multiple routable engines they all share some common lib (addon). For any small change to common lib we have to do a release and update host app and all the engines' package.json. So we would have to also create releases of all the engines as well. We are using semantic versioning but The reason of doing it this way is, we have moved tests related to engines into engines and when we run engine's test in dummy app we need the latest version of those common libs in the engines. Is there a better solution to solve this process.

Also it will be nice if someone shares their processes regarding engines. Its been bit difficult to manage engines in terms of dependancies.

[MelSumner]

We are having the same issue, and would benefit from some extra documentation/clarification. (+@dgeb)

[villander]

@dgeb Is there a solution for this? or does a feature need to be created for it?

[villander]

@MelSumner you found any solutions to this?

[Leooo]

We are thinking on our side to use ^ versions of our common addon in the package.json of both the parent app and every engine. That way new builds of the parent app will always automatically get the last common addon version both for the app and the engines.

Problem we have left with that is that we would still need to run every engine's tests together with the parent app tests, to make sure the new common addon version doesn't actually break the engines. Anyone knows how to do that?

[mydea]

This might not work for everyone, but we've recently moved our app + engines to a yarn workspace, roughly following this guide: https://medium.com/square-corner-blog/ember-and-yarn-workspaces-fca69dc5d44a

Now we run the tests for the host app & all engines in CI whenever we make a change to any of the engines.

[deepan83]

So there seems to be some unpredictable behaviour with common addon dependencies in lazy loaded engines with host version modules (of the same addon). We've seen the host version modules being overridden with the engine's version causing bugs. Would the solution be for engines to build its dependencies into it's own namespace? /engine-name/addon-name/module-name for example so they don't conflict with host app modules?

[nmcclay]

I wanted to leave a note that this issue has caused my company a lot of confusion as well. What is happening is that if you have multiple lazy-loading engines, and each engine has a shared dependency on a common addon whichever engine loads first will get its version of the addon defined for every other engine and even the HOST APP if it invokes the same component name. This is an issue if you are using engines to spread domain concerns down to individual teams because they all have to be in lock step with what version of this shared addon they use or else risk errors when component APIs change and a user doesn't load a particular engine first.

Most interestingly the code for all versions of the shared addon does exists within the engine-vendor.js source, however the namespace for its define is in conflict with the other engine's and host app's name so it will never be able to be imported. Personally this is a huge pain to deal with, and I'm currently exploring what @deepan83 suggests in name-spacing the modules somehow that you can manually import the correct version of your addon in each engine.

[deepan83]

@nmcclay happy to help with this as well. i'll do some digging.

[nmcclay]

So I played around with this and I have what seems to be a solution. Basically its a two part solution, first the shared addon needs a namespace configuration in its index.js:

const addonName = require('./package').name;

module.exports = {
  name: addonName,

  included(app, parentAddon) {
    // Quick fix for add-on nesting
    // https://github.com/aexmachina/ember-cli-sass/blob/v5.3.0/index.js#L73-L75
    // see: https://github.com/ember-cli/ember-cli/issues/3718
    while (typeof app.import !== 'function' && (app.app || app.parent)) {
      app = app.app || app.parent;
    }

    // if app.import and parentAddon are blank, we're probably being consumed by an in-repo-addon
    // or engine, for which the "bust through" technique above does not work.
    if (typeof app.import !== 'function' && !parentAddon) {
      if (app.registry && app.registry.app) {
        app = app.registry.app;
      }
    }

    // Per the ember-cli documentation
    // http://ember-cli.com/extending/#broccoli-build-options-for-in-repo-addons
    let target = (parentAddon || app);
    this.options = target.options || {};

    // only allows namespace to be set once per nested addon
    if (!this.namespace && this.options['namespace']) {
      this.namespace = `${this.options['namespace']}-${this.name}`;
      this.name = this.namespace;
    }
    this._super.included.apply(this, arguments);
  }
};

Then you can provide this namespace to your addon from your engine's index.js:

const EngineAddon = require('ember-engines/lib/engine-addon');
module.exports = EngineAddon.extend({
  name: 'engine-alpha',

  lazyLoading: Object.freeze({
    enabled: true
  }),

  namespace: 'alpha',

  isDevelopingAddon() {
    return true;
  }
});

Finally, in your engine's addon directory, you can declare a uniquely named component that will export this namespaced version of the addon declared in your package.json:

// addon/components/alpha-foo-bar
export { default } from 'alpha-addon-test/components/foo-bar';

(edit: cleaned up index.js implementation)

[dgeb]

Thanks for the discussion so far, and apologies for taking so long to get involved myself.

I'd like to share a helpful repo from @codeofficer that explores dependency conflicts between apps, engines, and addons: https://github.com/CodeOfficer/demo-engine-addon-dependency-conflicts The README clearly lays out several types of conflicts that can be reproduced.

Some conflicts are very difficult to solve in a general case, but I believe we need to start with clear warnings as part of the build process. @codeofficer's suggestion to use ember-cli-dependency-lint is a good first step.

I just wanted to check in for now, but hope to be back soon after exploring some possibilities.

[nmcclay]

Checking back on this issue I can confirm the issue we're encountering matches conflict 2. The above solution I posted turns out to be an incomplete solution. Specifically it doesn't resolve nested components, only top level component namespaces. I'm continuing to search for an adequate solution to this problem in order to allow our engines to invoke their version of their shared addon dependency without requiring all my engines to be in lock step dependency wise.

Whats most frustrating is that the code itself is bundled and there, but it will not be invoked because the first time the component is invoked that is the version that is cached in ember and will be invoked in all other engines or the host app that references it.

[nmcclay]

This issue is also referenced in the official docs with the recommendation to avoid including different versions of the same addon. I would like to find a solution to this issue that doesn't involve our teams having to be in lock-step and they can opt into updates when its convenient for them rather than requiring anyone that needs an upgrade to force everyone else to integration test a centralized change.

[dgeb]

@nmcclay Check out https://github.com/dgeb/ember-cli-addon-guard, which is the solution I developed based on ember-cli-dependency-lint. ember-cli-addon-guard will automatically identify any conflicting addons shared among an app and its engines (and other addons) at build time. Furthermore, there is an experimental namespacing feature described in the README that allows multiple versions of the same addon to coexist.

[nmcclay]

@dgeb you are amazing! I tried this out and encountered a few unexpected issues getting it working with my engines, but after some hacking I got it to work for both the top level duplicated addon and any nested sub-components without any additional engine template rewriting. Very cool stuff and exactly what I was looking for!

One thing I noticed is ember-engines 0.7.1 is dependent on "calculate-cache-key-for-tree": "^1.1.0", which fill fail your CacheKeyDependencyChecks.

*edit - problem is unrelated to calculate-cache-key-for-tree dependency, will follow up in ember-cli-addon-guard repo.

[dgeb]

@nmcclay great, glad it's working for you (for the most part)! I'll follow up with any issues you open in ember-cli-addon-guard.

[villander]

@nmcclay ember 0.7.1 already have deduplication as feature flag.

[nmcclay]

@villander I tried out the deduplication feature flag, but it doesn't seem related to conflict 2 from demo-engine-addon-dependency-conflicts repo. Its not a duplication problem as much as a unique namespace problem that doesn't allow you to load different versions of shared addons within engines.

[villander]

right! makes sense. But I think that conflict 2 must be handle on ember-engines codebase, because our deduplication flow already have control about host-app of ember-engines.

what you think? this is scope of ember-engines addon?

cc: @dgeb @rwjblue @2hu12 @stefanpenner

[dgeb]

ember-engines relies on addons' cache keys to uniquely identify each version and should only dedupe addons with matching cache keys. Every addon with a unique version will also have a unique cache key, since the default cache key is calculated by hashing the addon's package.json (which includes version as a field). While ember-engines should be concerned with deduping addons with matching cache keys (to reduce bloat), I don't think it should be its responsibility to dedupe addons with different cache keys. Choosing winning versions, or merging all versions, automatically is only going to cause endless problems that are difficult to diagnose and debug.

I built ember-cli-addon-guard outside of ember-engines because these problems are not unique to engines. Although conflict 2 is a unique symptom for lazy-loaded engines, the underlying problem is the same: ember-cli flattens addons into a single namespace regardless of version.

To address this, ember-cli-addon-guard makes you aware of addon dependency conflicts via a hard build error. You can then address these conflicts in one of two ways:

• remove the duplication by adjusting the dependencies (preferable)
• namespace conflicting addons by cache key, allowing them to coexist (not preferable, experimental)

I think these are the only two reasonable ways to handle these conflicts in the current system in which dependencies are all flattened. Of course, we'll have more options available if we stop flattening dependencies.

[Leooo]

A gotcha we had recently here and that may be of use to some: we are splitting our old shared addon into two bits:

• addon_1 with models / services / test helpers / mirage helpers; that bit doesn't actually need to go into the engines' dependencies, instead it can be provided by the parent app / dummy app through the engines' devDependencies (for models, that's because the parent app provides the store as a service).

• addon_2 with .. just components left (and few initializers), and this one is the only part left that HAS to still be consumed as one of the dependencies of the engine - because components are needed in hbs templates etc.

So at the end of the day, addon_1 doesn't need to be the same version in all engines, as it is only built in the parent app or dummy app, so when updating your addon you just need to update the engine you are working with, and the parent app, and you are done - for all models, services, tests and mirage helpers (supposing of course you are confident that updating your shared addon won't break your other engines' tests..).