How/where to publish RapydScript packages?

[feihong]

If I make a RapydScript library, what is the best way to publish it? I know of npm, but it seems like it's mostly for JavaScript packages. Maybe PyPI? But if I use pip to install a package, is there a good way to make it visible to the RapydScript compiler?

[kovidgoyal]

This is not something I have though about much. Personally, I just use like to use git to distribute my packages. Then you can make the RS compiler aware of them by using the --import-path command line option. This works like PYTHONPATH for python packages.

The only thing using systems like pip/npm buys you is the ability to auto-resolve dependencies, but IMO this is more of a mis-feature as it leads to enormous and unneeded dependency chains. You can see this particularly in the npm ecosystem, but python is getting there as well.

That said I am not adamantly opposed to having a package repo for RS, it is just not much a priority for me. I will leave this issue open in case other people would like to comment.

[feihong]

What if I made a command called rapyd_install that basically does what pip does except it also creates a rapyd_modules folder with symlinks to the RapydScript-specific module directories in site-packages?

Maybe you could also do something similar with npm but I'm not as familiar with that package management system.

[kovidgoyal]

Using PyPI as a backend to store uploaded packages? Sounds like a reasonable approach to me. Some things to look out for:

1. System-wide vs. user local virtualenv type installations
2. Creating some mechanism to allow adding data files to rapydscript packages
3. I dont know if the PyPI maintainers would be ok with using their server to host non-python packages

[kovidgoyal]

Actually, thinking about (3) I think we could just use github itself as a package repo. We just need to establish some conventions for repository layout of rapydscript packages, then the package installer could just fetch them using git for github (or any other git host). This will work rather like pathogen works for vim packages.

As for creating a central package index, that can be left for the future, at which time the package manage tool can grow a "publish" sub-command.

[feihong]

I looked a little into using npm since there are packages in there not written in JavaScript. But unless you are willing to compile your code to ES5 when publishing, it doesn't make sense to publish RapydScript packages to npm.

[kovidgoyal]

Yeah, I dont think using a pre-existing repo is going to pan out.

[CodeDoes]

We could have a central repository with a list to the packages hardcoded into the source.

There's also the option of a central repository which contains all of the packages.

[CodeDoes]

we could tequnically have a use npm too

[devxpy]

Creating some mechanism to allow adding data files to rapydscript packages

This can be done using a MANIFEST.in file and enabling the include_package_data in setup.py

I dont know if the PyPI maintainers would be ok with using their server to host non-python packages

Did this with my project, react pages. Everything seems to be working fine :) https://github.com/pycampers/react-pages

Personally I prefer pip over npm any day.

[kovidgoyal]

Interesting, thanks for the heads up.

[icarito]

Thinking about the "batteries included" aspect of Rapydscript-NG - that they are the Javascript batteries not the Python ones, maybe check out a browser side bundler such as this? If we could import jquery, import React and so on, and have SystemJS resolve and provide dependencies for us, this would be really cool during development. SystemJS could be leveraged to bundle dependencies for production too when building from the command line with RapydScript-NG.

[icarito]

Here's another npm-compatible bundler compatible with the browser.

[kovidgoyal]

It's hard for me to get a good sense of what's needed, as the current rs-ng infrastructure fully meets my needs. The most basic elements of a packaging system are:

1. Create packages
2. Publish packages
3. Use packages in other projects

This is complicated in the case of rs-ng by the fact that packages can be in two languages (rs-ng and javascript), with many different runtimes (browser, nodejs, other JS runtimes such as duktape). Further, one would want to be able to both:

1. Compile all used packages into a single JS file
2. Split up packages for dynamic loading in the browser to save on bandwidth/load times
3. Use packages both in the browser or in offline node contexts, either via compilation or dynamic include at runtime

All in all, this seems like a pretty large problem, I dont think I have the bandwidth/motivation available to tackle it, as the RS compiler meets all my personal needs as is.

[icarito]

Thanks Kovid.

Your reasoning is sound. I was describing what "bundlers" do in JavaScript.

Naturally I don't think RapydScript-NG should try to be a bundler.

I'm still an apprentice with regard to JavaScript - I've used WebPack and Browserify.

In my learning process I contributed to a Python loader for Webpack: https://github.com/martim00/python-webpack-loader and implemented a Python module loader for the Nuxt Vue.js framework: https://github.com/nuxt-community/python-module

---

The reason I care about this is I'm polishing Jappy Creator (repo) which basically is a learning IDE with RapydScript-NG as engine and facing a few design questions with regard to how to offer popular modules to end users and to publish user's creations. Jappy is meant to work on the client side only relying on a server for access to reading/writing files.

The current approach would be to add .pyj files to src/lib/, yes? I'll do this for now.

I think a Javascript bundler could be configured without a need for direct support in RapydScript-NG. One would use require for this purpose in WebPack and Browserify, and it should simply work in RapydScript.

I made a RapydScript-NG transform for Browserify a few days ago: https://www.npmjs.com/package/rapydscriptify-ng

It is based on a previous one I made for the original RapydScript: https://www.npmjs.com/package/rapydscriptify

With this, one can use Browserify to bundle Rapydscript files (this covers the points 1-2-3 you listed previously).

I'll think hard how to provide a consistent experience when compiling RapydScript-NG code in the browser. I think RequireJS or SystemJS bundlers could provide solutions for Jappy Creator's requirements (internally it already uses RequireJS).

[kovidgoyal]

If you need any additional API in the embedded compiler to facilitate your use case, let me know and I'll take a look. It's not clear to me what exactly you need, do you want the ability to add rapydscript modules to the embedded compiler when it is generated? Dynamically at runtime? Some combination of the two?

[icarito]

I've just integrated p5 library with Jappy with including a pyj file in lib/src and thus made a way to import p5 into Jappy. Nice! This is sufficient for my use case for now! :-)

It is now possible to do import p5 and it should just work.

With regard to making new .pyj available for importing at runtime for files compiled in the browser, I've overriden the file_data object and this seems to work. This was asked here too by another project. It's a bit clunky, not sure if it's meant to be used or internal. Perhaps a better interface or docs on how to do it cleanly would be super. Thank you in any case, we love RapydScript-NG :+1: