Support for python3.8

[pbsds]

trafficstars

This seems like this is a very hard transition.

They have changed the signature of types.CodeType, adding an extra positional argument called posonlyargcount. Accounting for this is easy, but they have also made drastic changes to the bytecode format, making it difficult to translate the existing bundled bytecode.

My diff so far accounting for the `posonlyargcount` arg in CodeType

diff --git a/__init__.py b/__init__.py
index 4283df8..099f936 100644
--- a/__init__.py
+++ b/__init__.py
@@ -17,9 +17,14 @@ def load():
             for code in load(fd):
                 eval(code)
     except FileNotFoundError:
+        def _py37to38_CodeType_shim(*args):
+            if len(args) == 15:  # py3.7 and lower
+                args = (args[0], 0, *args[1:])  # insert a posonlyargcount=0
+            return types.CodeType(*args)
+
         try:
             with open(os.path.join(BUNDLE_DIR, PY_TAG + '.dgbundle.py')) as fd:
-                for code in eval(fd.read(), {'C': types.CodeType}):
+                for code in eval(fd.read(), {'C': _py37to38_CodeType_shim}):
                     eval(code)
         except FileNotFoundError:
             raise ImportError('python implementation {!r} not supported'.format(PY_TAG))
diff --git a/core/__init__.dg b/core/__init__.dg
index ea2f5ca..f6edcbc 100644
--- a/core/__init__.dg
+++ b/core/__init__.dg
@@ -55,9 +55,9 @@ save = code tag dirname ->
 
 _repr = x -> if
   x :: types.CodeType =>
-    fields  = ['argcount', 'kwonlyargcount', 'nlocals', 'stacksize', 'flags', 'code', 'consts', 'names']
+    fields  = ['argcount', 'posonlyargcount', 'kwonlyargcount', 'nlocals', 'stacksize', 'flags', 'code', 'consts', 'names']
     fields += ['varnames', 'filename', 'name', 'firstlineno', 'lnotab', 'freevars', 'cellvars']
-    'C({})'.format $ ','.join $ map (f -> _repr $ getattr x $ 'co_' + f) fields
+    'C({})'.format $ ','.join $ map (f -> _repr $ getattr x $ 'co_' + f) $ filter (f -> hasattr x $ 'co_' + f) fields
   x :: list  => '[{}]'.format $ ''.join $ map (i -> _repr i + ',') x
   x :: tuple => '({})'.format $ ''.join $ map (i -> _repr i + ',') x
   otherwise => repr x

I tried my hand at using decompyle3, but in my attempts it failed parsing any dogelang bytecode from python 3.7.

If my understanding of the dg compiler is correct, it compiles directly to bytecode, making the current implementation not work on py3.8+ anyway.

[pyos]

Shouldn't actually be all that hard...looks like there are only two major changes:

• finally handling is now sort of like Java's old jsr opcode - need to insert the new CALL_FINALLY opcode before every outbound jump/return;
• the compiler is now expected to keep track of stack depth (which is already done) and insert an appropriate amount of pops instead of using BREAK_LOOP and CONTINUE_LOOP's automatic stack unwinding

Not sure what's up with END_ASYNC_FOR, and MAP_ADD is not used (it's for dict comprehensions).

A bigger problem is that dg is way behind on features, e.g. string interpolation and parameter/return annotations.

I tried my hand at using decompyle3, but in my attempts it failed parsing any dogelang bytecode from python 3.7.

There's no point in doing that, yeah. Once the compiler supports 3.8, a new bundle for it can be generated by running on 3.7 with -b cpython-38 030800F0 or something like that.

[pyos]

Heh, I looked through the current version of ceval.c and not only CALL_FINALLY is about the same as JVM's jsr, but it'll share jsr's fate as well: https://bugs.python.org/issue33387

[LeaveNhA]

Anyone who wants to implement the desired changes to fix that issue so the language can continue to live? Version restriction is killing it.

If someone interested, please ping me and we can brainstorm and solve it. Thank you.

[pbsds]

I did spend a couple weeks trying to convert dg to be a transpiler in fall 2020, as the python AST representation is not as much of a moving target as the cpython vm is. Since python 3.8 there have been many more major changes to the vm.

https://github.com/pbsds/dg/tree/transpiler

I stopped near close to the end when i realized i had to resort to litter the closures with lots of intermediate variables to ensure correct execution order. This is not transparent to the program, as these hidden variables will show up when calling locals, or when inspecting the resulting code objects.

Since i stopped, the python AST has in preparation of PEP-638 gotten some features that could make the transpilation work better, but i believe a better approach would be to adapt the current tokenizer and parser to work as a frontend to hissp. One such example is hebigo, whose implementation seems to be quite small.

[LeaveNhA]

I have two options here to continue;

• As the original author decided, I can continue to follow his direction and will play the puss-in-the corner with the esoteric CPython VM.
• Grasp your approach and contribute your way to finish at once.

What would you suggest me, @pbsds?

[pbsds]

This is ultimately an exercise you'll most likely be doing, go for the direction you find interesting. The first direction is a great opportunity to learn about the python VM and about stack machines in general. The other direction is an adventure into the python AST bindings, or the land of lisp.

[LeaveNhA]

Thanks for your comment! I will inform you fellas here when I decide.

[pbsds]

Someone on lobste.rs linked to this: https://github.com/rocky/python-xasm, might be viable?

[LeaveNhA]

Hmm, I really wanna do this. But I cannot proceed further because I need to gather my goals. If I took this language and get back to the design phase, I need to write at least 2 paper over it. Dg over semantic perspective and a short comparative analysis with Dg and Python.

I will be decided in next two weeks. I'm distracted enough until then.

[LeaveNhA]

Obviously, I decided not do it.

But I still open to discuss and communicate about the language. Feel free to write to me! 🙋🏻‍♂️