Compatibility with Transcrypt?

[Michael-F-Ellis]

trafficstars

I've had very good experiences this past year with Jacques de Hooge's Transcrypt Python to JS transpiler. http://www.transcrypt.org/ and https://github.com/QQuick/Transcrypt

It handles almost the entire Python language and produces high-quality JS that can seamlessly use existing JS libraries including jQuery, React and Plotly. As a result, I'm now developing control systems with web server HMI's entirely in Python -- no JS, no HTML, no CSS. I'm loving it and so are my clients.

Having Parsimonious available in the browser would be super-useful. As a simple example it would let me create a "Try it" interface for tbon that would do the parsing and midi file creation in a user's browser without requiring server side processing. Which means it would look like a static service and I could serve it from github.io.

Would you, Erik, be willing to take a look and maybe exchange ideas with Jacques? From looking at the Parsimonious source I see it's importing relatively few Python libraries. That's promising in terms of Transcrypt compatibility. The question is probably whether Parsimonious could be forked to not use any libs with C extensions as Transcrypt doesn't handle those.

I'll post an issue in Transcrypt's repo linking to this.

Thanks! Mike

[lucaswiman]

Parsimonious has no imports other than the standard library. What C extensions are you referring to?

[Michael-F-Ellis]

I'm probably speaking a little out of my depth. What I meant is that Transcrypt has taken the approach of supporting the Python language (3.5+) but very little so far of the standard library beyond math, random and part of numpy. The author's approach has been that since so many JS libraries exist it makes more sense to make it easy for transpiled Python to use those.

I filed this issue because wanted to get some discussion going between to the two projects to see what barriers actually exist.

[lucaswiman]

Oh, I forgot about the dependency on six. The imports from the standard library are pretty minimal:

~/opensource/parsimonious $ git grep '\bimport ' parsimonious/*.py | grep -v 'from parsimonious'
parsimonious/exceptions.py:1:from six import text_type, python_2_unicode_compatible
parsimonious/expressions.py:9:from inspect import getargspec
parsimonious/expressions.py:10:import re
parsimonious/expressions.py:12:from six import integer_types, python_2_unicode_compatible
parsimonious/expressions.py:13:from six.moves import range
parsimonious/grammar.py:8:from collections import OrderedDict
parsimonious/grammar.py:9:from inspect import isfunction, ismethod
parsimonious/grammar.py:11:from six import (text_type, itervalues, iteritems, python_2_unicode_compatible, PY2)
parsimonious/nodes.py:9:from inspect import isfunction
parsimonious/nodes.py:10:from sys import version_info, exc_info
parsimonious/nodes.py:12:from six import reraise, python_2_unicode_compatible, with_metaclass, \
parsimonious/utils.py:3:import ast
parsimonious/utils.py:5:from six import python_2_unicode_compatible

It's up to Erik, but I'd vote against changing the source to be compatible with a partially-complete python-to-js transpiler. Once Transcrypt is feature complete (i.e. supports the Python standard library), then maybe some minor changes to support it would be OK. Have you considered looking into PEG libraries in Javascript?

[Michael-F-Ellis]

Here's a summary of the the differences between Transcrypt and CPython.

http://www.transcrypt.org/docs/html/differences_cpython.html

I know there are various PEG libraries for JS and that's certainly an option. On the other hand I've found Parsimonious remarkably easy to work with and would at least like to know whether it's feasible to be able to use it under Transcrypt.

[lucaswiman]

Looks like it's a pretty strange subset of Python, and libraries not specifically designed to work with it are unlikely to work correctly. To support it, you'd have to write very non-idiomatic Python, so it doesn't seem like a good fit for a library geared at being pure-python.

[Michael-F-Ellis]

My experience with Transcrypt has been very different. I've not found it necessary to write non-idiomatic python to make good use of it. But YMMV, of course.

Let me see if I can focus this discussion a little more by considering what would be required to fork Parsimonious and make it function under Transcrypt.

Let's assume first the elimination of six and a decision to support only Python 3.5+

Then, looking at the import list you provided and neglecting six, it can be summarized as:

from inspect import getargspec, isfunction, ismethod
import re
from collections import OrderedDict
import ast
from sys import version_info, exc_info

Now I know that ast is what Transcrypt uses to parse Python source so it seems at least conceivable that it could at the very least precompile the grammar. I'd need for Jacques to weigh in on that as well as OrderedDict and the functions from inspect.

I believe Transcrypt can already handle the imports from sys.

That leaves re. I'm sure there must be a ton of C code underneath that module to achieve good performance. So it would probably come down to creating a wrapper around JS's own regexes that mimics the Python Re syntax.

So to my admittedly naive first glance it doesn't seem impossible. Another avenue, should Jacques and crew see some benefit, would be to incorporate logic and interfaces from Parsimonious into a Transcrypt module.

[lucaswiman]

Something that doesn't natively support regular expressions won't work well, given that parsimonious extensively uses the re module, and subtle differences between python and javascript regular expressions would probably be a major source of bugs. parsimonious also uses method decorators, and this seems like a very weak argument for eliminating their use. Also, this is a big change:

Let's assume first the elimination of six and a decision to support only Python 3.5+

At the moment all of my applications of parsimonious are on 2.7, and I'd bet that's true of many other users of the library.

[Michael-F-Ellis]

Sounds like a fork would be the best approach.

[erikrose]

Thanks for the clueful comments, all. Weighing in here:

• I want to keep 2.7 support for now, as a lot of my own stuff is 2.7 as well.
• I think JS and Python may have similar enough regexes to tell people "Just make sure you write a compatible subset if you're targeting both JS and Python."
• I'm not sure how Transcrypt handles the chunks of the stdlib it does support, but it seems like the cleanest way to make Parsimonious compatible with it is to add support to Transcrypt for the few stdlib things we use. isfunction and ismethod ought to be portable, and getargspec should be technically possible as well (worst case, we have to look at the introspected source of a function). OrderedDict is pretty much just an ES6 Map, and version_info is a constant. exc_info depends on to what degree Transcrypt supports exceptions. The only thing we use ast for is interpreting \n, \t, etc. within Parsimonious strings. So really, the question is how easy it is to add new stdlib routines to Transcrypt.

Though I don't have the time to see this through right now, I encourage anyone else to have a swing at it! Or you're welcome to try going about it the other way, on a fork or a branch. Btw, it is an eventual goal to make Parsimonious' syntax compatible to the degree possible with Michael Foord's original PEG syntax. Then you should be able to pretty much drop grammars back and forth between it and, say, peg.js.

[Michael-F-Ellis]

Just a quick update. I'm punting for now on trying to get Parsimonious to run in the browser. I've now got a not-very-fancy web page to demo my notation language at https://ellisgrant.pythonanywhere.com that sends the notation back to a server for processing.

Should be adequate unless I get thousands of users (LOL).