mpetuska/npm-publish vs gciatto/kt-npm-publish

[gciatto]

Hello @mpetuska, I'm trying to understand whether two relevant features of mine are currently supported by npm-publish. In case they are not, they could be my first actual contribution to your plugin!

Feature 1: Organizations

I see that npm-publish lets the user specify an organization for the generated JS project, via the following syntax:

npmPublishing {
  organization = "my.org" // (Optional) Used as default scope for all publications
}

What's the actual semantics of this statement? I would expect this makes the generated package.json contain a field of the form "name": "@my.org/<projectName>". Is my intuition correct? Furthermore, to make this coherent with NPM packaging, I would also expect this statement to make any other JS sub-project generate JS code contain statements of the form require("@my.org/<projectName>") instead of require("<projectName>"). Is this the case?

My plugin, kt-npm-publish does not technically provide an organization field, yet it does provide an easy way to alter the generated JS code, in order to inject organizations into it. Would a similar feature be useful in npm-publish?

Feature 2: Automatic conversion of generated dependencies from local to general

Currently, the JS/NPM transpiler generates package.json files where dependencies are in the "local" form:

"dependencies": {
    "kotlin": "file:/path/to/project/build/js/packages_imported/kotlin/KOTLIN_VERSION",
    "kotlin-test-js-runner": "file:/path/to/project/build/js/packages_imported/kotlin-test-js-runner/KOTLIN_VERSION",
    "kotlin-test": "file:/path/to/project/build/js/packages_imported/kotlin-test/KOTLIN_VERSION",
    "some-other-dependency": "file:/path/to/project/build/js/npm_modules/some-other-dependency/DEPENDENCY_VERSION",
  },

whereas NPM publishing requires dependencies to be in "general" form:

"dependencies": {
    "kotlin": "KOTLIN_VERSION",
    "kotlin-test-js-runner": "KOTLIN_VERSION",
    "kotlin-test": "KOTLIN_VERSION",
    "some-other-dependency": "DEPENDENCY_VERSION",
  },

Ideally, any plugin should provide an easy way to automatically switch from local to general form, without having to specify versions or package names explicitly---which would be redundant w.r.t. ad-hoc section of the build.gradle(.kts) files. Is this currently supported by npm-publish?

To the best of my understanding, you currently only support the removal/retaining of dependencies by name, or the hard-coding of a particular dependency, given its name and version---which should therefore be explicitly represented. Is my intuition correct?

[mpetuska]

Not quite correct.

When it comes to organization field, it only generates appropriate package.json#name value (@myorg/package-name). Modifying require statements in generated js code is only useful in legacy backend, because IR backend embeds all kotlin-js dependencies in generated js file already, so given the fact that legacy backend will go away soon, I don't think it's worth adding support for require tweaks in js files at this stage.

As for versions, npm-publish automatically adds all relevant modules and versions from gradle's dependencies { implementation(npm("xxx", "version")) }. The reason it's excluding kotlin dependencies is the same as above, IR backend already embeds them inside generated js. For legacy mode it adds them via bundled dependencies, since most of the time kotlin js dependencies are not available on npm.

To getter understand what's going on, run ./gradlew pack on sandbox module and check what publications are assembled for various kotlin-js use cases (js, mpp, legacy, ir).

[mpetuska]

Just to be clear, this was by no means me saying no to the idea, I'm just clarifying current behaviours and looking to hear your further thoughts if you still think something could work better, given the context.

[gciatto]

Hello @mpetuska, sorry for my late answer. I needed some time to conduct a few experiments, since I realised I was not aware of how the IR backend works.

Modifying require statements in generated js code is only useful in legacy backend, because IR backend embeds all kotlin-js dependencies in generated js file already, so given the fact that legacy backend will go away soon, I don't think it's worth adding support for require tweaks in js files at this stage.

I was astonished by this statement, expecially the "IR backend embeds all kotlin-js dependencies in generated js file already" part, so I experimented a little bit with that.

But first of all, let me explain why I was astonished. Bundling all dependencies into a unique .js file makes no sense to me if one is producing a library to be published on NPM. That woul create A LOT of JS code duplication (I believe the reason behind that is obvious, but feel free to ask for details in case it's not). Such a bundling may make sense if someone is producing a JS script to be loaded by a webpage. But I argue that any plugin supporting publishing on NPM should consider the use case where several inter-dependent Kotlin MPP projects are published on NPM together, possibly under a particular organization ID.

So what's the actual functioning of the IR backend. It does indeed work differently than the legacy backend, but not that much. In particular, if properly configured, the IR beckend generates one NPM project for each Kt projects, and sets their dependency accordingly, despite using the "local" syntax above. However, it may still bundle several projects into a single file for test projects. So if one runs the jsTest task, and only looks at the *-test generated projects, they will be convinced that the IR beckend does bundles all JS code together. On the contrary, by running the assemble task, one may notice that the IR backends still generates distinct Node projects, provided that:

• all projects expose at least one class via the @JsExport annotation (no class is exported by default on the IR backend, so if not JsExport is contained in the code, the corresponding project will be compiled into an empty Node project)
• all projects are configured as kotlin { js { binaries.executable() } }

After a few try-and-error, and some readings, I ended up believing this is the correct setup for writing Node libraries in Kt. Dummy example project to reproduce the experiments here: https://github.com/gciatto-unibo/dummy-ktmpp

So I hope I was able to explain why I believe the feature above are important and why they will still be important for the IR backend.

[mpetuska]

This sounds like a TON of work, but if you're feeling brave I'm happy for you to implement it under some experimental config flag (disabled by default).

The reason I'd like it under flag is that this would be a bit too invasive to user's config to leave as default behaviour, so having them to agree with it explicitly seems sensible. Also it's prone to break with each kotlin release as it would depend quite a lot on kotlin plugin's internals.

[mpetuska]

Also I'm still not convinced it's even possible to disable IR backend "bundling" of kotlin.js dependencies. Your experiment repo doesn't really have any libraries (just executables)

[mpetuska]

Hold on with the implementation a bit if you don't mind, I've asked JB guys to join this discussion to clarify the IR backend behaviours for us.

[gciatto]

This sounds like a TON of work

Well, actually not that much. It's slightly more than a find & replace in JS sources... or at least it works like that in `kt-npm-publish}.

but if you're feeling brave I'm happy for you to implement it under some experimental config flag (disabled by default).

That's the plan if you don't mind.

Also I'm still not convinced it's even possible to disable IR backend "bundling" of kotlin.js dependencies. Your experiment repo doesn't really have any libraries (just executables)

What do you mean by "libraries"? Do you mean dependencies?

Hold on with the implementation a bit if you don't mind, I've asked JB guys to join this discussion to clarify the IR backend behaviours for us.

No problem with that: I'm currently working on my PhD thesis, so the time for coding is pretty low. Apologies for that, BTW.

[mpetuska]

What do you mean by "libraries"? Do you mean dependencies?

Basically binaries.library() vs binaries.executable()

[mpetuska]

Hey, just re-read your comments on reaktive discussion regarding this. Especially

Problem 2 only arises if you need/want to change the default project name generated by the Kotlin compiler. This is far from trivial, especially in multi-projects builds, because it is NOT sufficient to edit the generated package.json files to change the project names.

So for instance: if your root project is named foo and it has two Kotlin MP/JS sub-projects named bar and baz respectively, then the Kotlin compiler will generate the foo-bar and foo-baz projects. Let's assume this is not fine for your, as, for instance, you may be willing to publish both projects under a common organization, say @acme. This implies they should be renamed to @acme/foo-bar and @acme/foo-baz respectively. This implies their all occurrences of foo-* in all package.json files and in all .js files should be updated accordingly...

The matter gets complicated if the two projects are inter-dependent.

So, again, long story short: no automate way to do so. The only semi-automated way to do so I'm aware of is my kt-npm-publish plugin.

I now get what you mean and agree that this is indeed a must for a plugin as most of the people scope the package to publish to GH.

[mpetuska]

Just had a go with reaktive lib setup for this and to be honest I didn't encounter any issues when importing scoped declarations without tinkering with generated js files, so I guess just modifying package.json is enough.

[gciatto]

What did you tried exactly? 

Imho renaming of node project in js sources is hard when multiple interdependent KtJs projects must be renamed consistently to be published togheter. 

So if you have two projects, foo and bar, where foo depends on bar, and you rename foo in @org/foo and bar in @org/bar by changing the package.json files alone, then importing @org/bar should simply work as project bar does not reference itself.

Issues may arise when one tries to import @org/foo as its source code vill contain import statements of the form

require('bar') 

instead of

require('@org/bar')

So, renaming of a module should step through the renaming of the package.json of that module, plus the renaming of that module in any other module of the same project which depends on it.

This is a complex task indeed.

However, if the renaming should just add an @org/ prefix to all modules in a project, the task can be simplified by leveraging on the assumption that all JS modules are named as rooProject-moduleName. So it is sufficient to search&replace the string require('rootProject in all generate js files with require('@org/rootProject

[mpetuska]

Ah, I think I get the idea. Renaming is only needed for kotlin.js consumers of the module, and NOT the module itself, right?

That actually gave me an idea. How about instead of tinkering with js files, we intercept webpack task and just declare kotlin.js dependencies as webpack externals (this is how pm dependencies are currently excluded from bundling). The problem then is that the module would only be able to see @JsExport declarations from that other kotlin.js module, therefore introducing a hard separation between modules.

Finally, if you just rename references, that doesn't achieve anything as bar would still get bundled into foo js output and not declared in package.json.