textX + PyEcore

[aranega]

Note: In this issue, are reported discussions and experiments around the association of textX and PyEcore. Progress can be found here ~~and it currently requires the PyEcore version from the develop branch~~. (follows discussion from igordejanovic/textX#41 and igordejanovic/textX#42)

TextX is a meta-language, inspired by Xtext which allows you to easily generate a parser and an metamodel/model-based AST for a textual language. The metamodel based AST created by textX is not yet compatible with the Ecore API. This POC tries to bring PyEcore, so Ecore, to textX in order to be able to handle an Ecore compatible model and metamodel.

This addition could open a way of interoperability between Java-EMF and Python-EMF (PyEcore) through the xmi/ecore file format. Moreover, using pyecoregen, it could enable a full Python T2M (Text to Model), and M2T (Model to Text) experience.

The goal of this POC is to see if PyEcore can be integrated inside textX following three scenarios:

1. from a textX grammar, an Ecore metamodel is automatically generated, and the generated parser + metamodel can be directly used to build the AST.
2. using existing dynamic Ecore metamodel (built in memory) and a textX grammar, textX uses the defined dynamic Ecore metamodel to build the AST.
3. using a first compilation phase: a textX grammar is used for generating the Python Ecore metamodel code (through the textX cli). Then the generated Python code is used by textX to build the AST.

This latter scenario allows the user to easily add behavior to the metaclasses instances.

The scenario 2 had been already validated. This implied minor modifications of how textX create instances and init them (no need to init instances anymore as PyEcore does it), and a biggest modifications on how the cross-references are resolved.

The scenario 1 and 3 relies on the same principle: the Ecore metamodel must be generated on-the-fly from the textX grammar. Consequently, the isses and challenges are not the same as in scenario 2.

Here what's have been validated so far and the future features to add:

• [X] Replace textX metaclasses creation by EClass instances
• [X] Add EAttribute/EReference support for EClass
• [X] Reference eOpposite support for static and dynamic Ecore metamodel (not automatically generated at the moment)
• [X] Deal with EClass inheritance
• [X] Deal with abstract classes
• [X] EPackage creation for a grammar/metamodel
• [X] Detection of rules that define EDataType
• [X] Creation of EDataType
• [X] Detection of rules that define EEnum (currently not bullet-proof)
• [X] Creation/Filling of EEnum
• [X] Basic data types support (should be modified)
• [x] Management of split grammar (defined in many files)
• [x] Management of multiple namespace/EPackage
• [x] Detect 'match' and 'non-match' rule call in the same rule definition as semantic errors
• [x] Better support for multiple grammars relative paths
• [X] Find a common way of dealing with basic EDataType and textX builtin basic types
• [x] Deal with split grammar/metamodel generation (generations and imports)
• [x] Add a cli option to generate the PyEcore metamodel using pyecoregen
• [x] Add special parameter to pass an existing EPackage as user-classes
• [x] Add a kind of "switch" for PyEcore activation in textX
• [x] Add better multiple metamodel generation support (currently, some imports are missing)
• [ ] Add documentation! (obviously)

As a complement of this list, preferably, the PyEcore integration is optional and can be activated using a parameter in textX. Also, another idea would be to add a dedicated keyword to introduce eOpposite from grammar.

The generation of the metamodel Python code using the cli also implies the question of re-generation and perservation of manually inserted modifications. This issue is more general and could be addressed in pyecoregen or pymultigen.

[moltob]

That's absolutely amazing! I was consideruing doing similar work by extending the ANTLR syntax and generating the parse tree listeners on the fly to build the AST. I'll watch this closely... :-)

Regarding your comment about adding custom code to generator output. I am no fan of adding manual content to generated files, as usually this leads to issues. Just think about how to add such content to GIT. You'd be checking in generated content as well in this case.

Therefore, manual code should really always be in a separate file. This is one of the reasons C# came up with the partial keyword, allowing the developer to write code in a separate file extending the class generated from UI wizards.

In Python I'd follow a similar route: Add a generator option that results in mixin classes being used in the generated code. These classes can then be implemented in a user-controlled module.

What do you think?

[aranega]

I hope I will be able to make something usable with acceptable performances. Finally the difficult part is more related on "how to integreate everything without disturbing the original behavior". I don't know if it will be possible.

Regarding the modification of the generated code, I suppose it really depends on your developement process. If your metamodel is stable and never change (e.g: the UML 2.5 metamodel), you can directly modify the generated code as there is many EOperation, and side-effect collections (which must be derived) where the behavior cannot be generated. To sum up, in this case, you generate once and use the generator as a bootstrap.

In the other hand, the C# concepts of partial classes sounds really cool, especially to deal with issues you exposed. I think I see how it could be done using mixin, but I'm not sure. If I got this straight, you still need to generate the mixin classes in a side module (and inheritance), but the code would only be generated using a special option in the cli? So, the EOperation code should be modified in case the option is passed: the current default implementation should call the super() method, and the generated code in the mixin should raise the NotImplementedError?

Perhaps I missed something, but, anyway, this behavior, and the fact that it can be activated through an option sounds amazing!

[igordejanovic]

I agree with @moltob. If the target language supports integration of generated and manually written code it is always the best option. You treat the generated code as the product of compilation (like e.g. .class in Java, or .pyc in Python) and you never check it in the git repo.

Generated mixin (or say abstract) classes can always be regenerated from the meta-model /grammar.

As a convenience, I would also generate stub classes for manual modification that inherit from them. These classes/files should be generated only first time and never regenerated. They should be checked in the git repo.

As a side note, if the target language doesn't allow extensions git can be your friend. Just keep generated code in the git on the separate branch and merge that to your manually written code. Treat the generator as one of the devs that works on a dedicated branch.

[aranega]

I'm not opposed to the idea of introducing a separation between generated and manual code, I'm 100% ok with this. I just would like the ability to choose wether you decide to generate the mixin classes or not. The idea is just to give the choice to the users to decide if they want to modify the generated classes or not. For prototyping and very simple projects, this can still have some interest.

If I'm not mistaken, here is an example of how could look the generated code (and generated mixin method) with EOperation and derived attribute/references:

# [...] PyEcore init stuffs

class A(EObject, AMixin, metaclass=MetaEClass):  # AMixin is added to the metaclass A
    name = EAttribute(eType=EString)
    _qualified = EAttribute('qualified', eType=EBoolean)

    def __init__(self, name=None, *, **kwargs):
        super().__init__(**kwargs)
        # [...] all init generated by pyecoregen

    def operation1(self):  
        super().operation1()  # the operation calls the mixin implementation

And in another .py with the mixin classes (generated only once during the first generation):

class AMixin(object):
    def __init__(self):
        super().__init__()

    # here is the code that should be modified for the 'derived' property
    @property
    def qualified(self):
        return self._qualified

    @qualified.setter
    def qualified(self, value):
        self._qualified = value

    # and here is the code that should be modified for the EOperation implementation
    def operation1(self):
        raise NotImplementedError("EOperation 'operation1' is not implemented!")

EDIT> Another pros of this approach: it becomes possible to differentiate the method generated from EOperation than the one manually written for technical purposes.

[moltob]

That looks like what I had in mind. A few refining suggestions:

Due to Python's MRO, mixins should always be listed before base classes, to make sure their implementations are selected.

Then I would design the command line to allow the user specification of a module name, from where the generated code can import the mixins. And since there already is a metaclass, you could selectively derive from the mixin class, depending on whether it is defined in the imported module by applying a filter to the bases list in __new__. That would allow to use mixin classes for all generated classes, but the user only defines them for the few where he actually needs it. Just a thought.

Let me know if you'd like me to extend the generator in that direction. Currently I think that would be an extension to pyecorgen only, not pymultigen, as it's a specifc extension to the generated class code.

[aranega]

@moltob Indeed, it sounds more like a dedicated solution for pyecoregen, this adaptation of the code generator would have no meaning in pymultigen. By the way, I really prefer this solution over the one I had in mind (AST fusion a little bit like JMerge). Thanks @moltob @igordejanovic for the idea!

Regarding the bases filtering __new__, I must say that it's a little bit fuzzy for me. The final goal sounds perfect, but I have trouble understanding the whole mechanic of the trick. This would also imply a modification on PyEcore?

Anyway, if you could extend the generator that way, that would be terrific! Thanks a lot Mike!

EDIT> Regarding the new option on the command line, could this be possible to have a kind of 'default' name filled from the main EPackage name (in case none is given). Having some case of default value is always great to lower the required actions for testing/experimentations.

[aranega]

Here are some news: I succeeded to deal with split grammar, and multiple namespaces/EPackage, modification was finally quite easy as I made TextXMetaModel class inherits from EPackage. Following this idea, the textX metamodel could be seen as an extension of the ECore metamodel.

However, I hit a wall playing with some rules. At the end, this is not a big issue, but it shows that when I introduce PyEcore, I can recognize a more constrained grammar familiy than pure textX. Is there an example from the 'Rhapsody' example:

Property:
    '-' name=ID '=' (values=Value (';'? !('-'|'}') values=Value)*)? ';'?
;

Value:
     INT | STRING | FLOAT | Object | GUID | ID
;

With PyEcore, the right type for Value is difficult to determine as it is abstract. There is no attribute, so it cannot be an EClass. It cannot be an EEnum, of course. If Value is identified as an EDataType with python object type, the values relation becomes an EAttribute, it works, but we totally miss the 'containment' constraint for Object and GUID instances. If we identifies it as a special EReference towards EObject type, it does not work as INT/STRING... are compatible instance of EDataType, but are not instance of EObject per say. This rule is an issue using PyEcore, but it is not with textX alone as the type checking for model building is not as strict as the one with PyEcore.

To work with this grammar, I had to adapt it:

Property:
    '-' name=ID '=' (values=Value (';'? !('-'|'}') values=Value)*)? ';'?
;

Value:
     ObjectValue | GUIDValue | PrimitiveValue
;

ObjectValue:
    value=Object
;

GUIDValue:
    value=GUID
;

PrimitiveValue:
    value=INT | value=STRING | value=FLOAT | value=ID
;

This time, everything works, and the containment constraints are good. At the end, this is not so much an issue because of Python duck-typing, requests and expressions on the model are not complex to write (with Java, it would have been a nightmare). So, yeah... this is not a big issue, but we have to keep this in mind.

[igordejanovic]

That's great! Yeah, textX is made to be more aligned with Python dynamic nature, thus you can write more natural/compact grammars. Probably some of those cases could be detected and reported as warnings/errors when pyecore is used.

I pulled your branch but experiments don't run successful. Several small errors, mainly usage of cls.__name__ instead of cls.name. Is the code in your fork up to date?

Maybe now it would be a good time to start writing tests?

I'll try to help out in making pyecore integration optional and selectable by parameter.

[aranega]

Thanks! There is still some rough edges, but I'm confident it will nicely work at the end.

Indeed, textX takes advantage of Python "dynamicity". I think I can detect those cases and raise a dedicated error.

Regarding the errors you had, did you use the develop version of PyEcore or the one in pypi? The current experiment requires the develop version as I'm adjusting the PyEcore code. For the experiment_xxx.py files, they are running without issue on my side, but it is possible I forgot some cls.__name__. Also, I didn't modify other files than metamodel/textx/model.py at the moment. I was pretty straight forwards for the experimentation in order to see what was the issues, there is some code I need to rewrite, for example, I was thinking about modifying the _new_class() method with:

if not hasattr(cls, '__name__'):
    cls.__name__ = name

This would add the __name__ attribute on EDataType and EEnum, and limit the cls.__name__ modifications through the code, but I'm not sure this is a good idea.

The last "big" thing, I think, is finding a way of translating basic textX types to Ecore ones in a transparent way. I have some ideas about this, I hope they will work! With this handled, I will rewrite some parts of the code to match the textX convention, try to handle the existing tests (if possible), and add some more. By the way, about tests, how do you launch the existing ones? I tried python setup.py test, but it fails to detect tests. How should I launch them?

Thanks a lot @igordejanovic, it will be great to have help about the PyEcore integration as I'm still not sure about how to properly do it!

[igordejanovic]

Regarding the errors you had, did you use the develop version of PyEcore or the one in pypi? The current experiment requires the develop version as I'm adjusting the PyEcore code.

Yes, you are right. I forgot to switch to develop branch and the execution took a different path running into some of the leftover cls.__name__ in the textX code.

I was thinking about modifying the _new_class method with:

if not hasattr(cls, '__name__'):
    cls.__name__ = name

This would add the __name__ attribute on EDataType and EEnum, and limit the cls.__name__ modifications through the code, but I'm not sure this is a good idea.

I think it is a good idea. __name__ shouldn't hurt EDataType/EEnum/EClass instances I guess.

By the way, about tests, how do you launch the existing ones? I tried python setup.py test, but it fails to detect tests. How should I launch them?

I just run py.test directly giving it the path of where to start. To run all functional tests:

py.test tests/functional/

Thanks a lot @igordejanovic, it will be great to have help about the PyEcore integration as I'm still not sure about how to properly do it!

Thank you for having started this. I'm not sure either what is the proper way to do it but you did a great job so far and the final goal doesn't seems so far away now. ;)

[aranega]

@igordejanovic Playing the tests, I had multiple issues, which is normal. Somes are related to the way they are written as the PyEcore introduction has added some other constnraints, but I have one I'm not sure how to solve it. It is an issue with namespaces and object resolution when a split grammar, is used and rules are "overriden".

As example, here is 3 files a.tx, b.tx, and c.tx:

a.tx

import b

Model:
    'model' name=IDDatatype '{'
    kinds*=Kind
    '}'
;

IDDatatype:
    t=INT
;

b.tx

import c

Kind:
    'kind' name=IDDatatype
;

c.tx

IDDatatype:
    t=ID
;

When I try to recognize: model 45 { kind test_kind }, PyEcore throw me a BadValueError: Expected type INT(int), but got type str with value test_kind instead. I'm not sure, but I think it should make the difference between the IDDatatype from a and the one from c. In the context of b

From the search I made, I saw that the resolution in model.py, in process_node, around line 258 (lines can be different in the original version), this line is problematic:

mclass = metamodel[node.rule_name]

When this line is called, the metaclass searched is always got using the __getitem__ method in metamodel.py which searches first in it's local namespace. As the IDDatatype name is found in the local namespace, it is returned even if it is not the 'good' metaclass. I'm not sure this behavior is the one entended or if I broke something during the experimentation?

Thanks for your help!

EDIT> The current solution I found is replacing:

mclass = metamodel[node.rule_name]

by

mclass = node.rule._tx_class

And some other variable in process_node(...) to get the metaclass from the instance using eClass. But I'm not sure this is the right way to do it.

EDIT2> If I use the original version of textX, and I change the first.tx file from:

Third:
    t=STRING
;

to

Third:
    a=STRING
;

An exception is raised during execution.

EDIT3> I "fixed" the original version by adding a special property for each instance that points to the metaclass. The modifications I made in model.py are:

# .. line 246
# mclass = metamodel[node.rule_name]
mclass = node.rule._tx_class

# .. line 271
inst._tx_metaclass = user_class

# .. line 279
inst._tx_metaclass = mclass

# .. line 346
# cls = metamodel[model_obj.__class__.__name__]
cls = model_obj._tx_metaclass

# .. line 506
# metaclass = metamodel[model_obj.__class__.__name__]
metaclass = model_obj._tx_metaclass

Once again, I'm not sure this is the best thing to do, if the behavior is the one entended or if I misunderstood something.

[igordejanovic]

@aranega I think you got it right! It is a bug. Although metamodel supports lookup by fully qualified class names the base name is used in this place.

Please send me a PR with your change and the test and I'll be happy to merge. Thanks!

EDIT> I like the idea of keeping reference to textX metaclass on object instances anyway. It might be useful to the user of the parsed model, although I haven't exposed/documented metamodel API yet but I probably should. ;)

[igordejanovic]

@aranega Wait a sec. I have to check something. Maybe we could solve this easier.

[aranega]

@igordejanovic No problem, if you have a better solution, it would be perfect!

If, eventually, the solution would not be so easy to implement, I pushed a fix and some tests on a branch: textX/namespace-resolution-issue which is ready to be proposed as PR.

[igordejanovic]

@aranega Thanks! I'm looking into the issue but it seems that it reaches deeper than I thought. Or maybe I got rusty with this code ;)

[igordejanovic]

@aranega I've merged your changes but have changed solution a bit to be more idiomatic for textX. _tx_metaclass instance attribute is replaced by using a standard Python type call, i.e. textX obj metaclass is actually a standard Python object class so no need to keep additional reference.

There are also several reworkings/fixes of the metamodel.py. __repr__ is improved for both classes and instances to give more information so console outputs should be nicer now. _tx_fqn is introduced and used in __repr__ to give the full path of the rule/class. Check it out, it's on the master branch. Thanks for the help!

[aranega]

@igordejanovic That's an awsome news! I updated my fork, and modify the sources so it matches well the new textX code.

Everything's running fine at the moment, but there is some existing tests I need to modify as there is some rules that works well with textX but which needs to be rewritten when PyEcore is used (as stated in a previous message). Also, the typing system for PyEcore is a little bit different as EClass are instance that behave like classes, but they are not classes per say (PyEcore builds an instance level representation for each EClass instance, and it keeps a bound to a Python class that it updates, you can see an example here) . This implies that some assert like:

assert type(obj) is MyMetaclass

Must be changed to:

assert obj.eClass is MyMetaclass 
# or
assert type(obj) is MyMetaclass.python_class

when the metamodel is build in memory by textX. When a static metamodel is used, the first assertion works just fine.

Beside that, everything is good on my side :). I refactored a little bit the code so textX + PyEcore can cohabitate with the original textX version. I will try to make all the tests running fine on the fork, then refactor the code, and try to integrate everything as it should.

[aranega]

So, I'm still on the process of making all the tests turn green. Since yesterday, all the textX examples are running fine (with few modifications)!

This is a good exercice as it forces me to take some distance from the PyEcore code, improve it, and improve the textX integration at the same time. In this context, I added an easier way of manually creating 'static' PyEcore metaclasses (the code generation is still the prefered and recommanded way), you can directly write:

class A(EObject, metaclass=MetaEClass):
    name = EAttribute(eType=EString)

to create a new EClass. This is fine, but still a little but verbose, so I introduced a class decorator that allows you to do this instead (it uses the same mecanisme than the previous example):

@metaclass
class A(object):
    name = EAttribute(eType=EString)

This looks really fine, but I cannot say if this is a good or a bad idea. I know that Mike is not a big fan of decorators for some usage (and for good reasons), but I cannot say in this context. The first mecanism still works, it is just some syntactic sugar.

What do you think?

[igordejanovic]

@aranega great progress! I'm always for cutting down on boilerplate code. Decorator in this case does exactly that and it looks more readable although it is a matter of personal preference. Maybe just to be more specific with the naming (e.g. @EMetaclass)?

[aranega]

@igordejanovic @EMetaclass sounds perfect to me!

[aranega]

So, I was quite busy the last weeks, but I found the time to make all the tests turn green. I had to adapt some so they match PyEcore specificity. They also help me fixing some bugs.

Now that I have two textX version: the original one, and one with PyEcore, it will be easier to see how to properly add a kind of "swtich" to enable or disable PyEcore in textX. It should not be so hard, but here is one modification that is more complex than the others: the cross-ref elements resolution.

In the original version, during the model building, each time a cross-ref towards an other element is found, a CrossRef element is created and added in the model graph. Later, the whole graph is scanned and each time a CrossRef is found, the reference is resolved and the CrossRef is replaced by the resolved element.

With the PyEcore integration, this cannot be done as collections are typed, so the placeholder cannot be added like this. Instead of storing the CrossRef in the model and resolving it later during the second model graph traversal, I store them as parser instance attribute. At the end, the same mecanism for resolving is used, but this time, I only iterate over the stored CrossRef. This cuts the recursive algorithm, but between the first and the second pass, it puts the model in a 'not-complete' state. Is the recursive version without the instance attribute better for textX? Or would the store+iterative version work?

If this modification is not right for the original textX, I will create two different methods for the parser. The new one will be called if the PyEcoree support is enabled.

[igordejanovic]

Really happy to hear that tests are green! From the top of my head I think that there is no special reason why would recursive algorithm be better for textX. Keeping all crossrefs on the parser and resolving later is perfectly fine. It is ok for a model to be in a non-complete state as even with crossrefs in the model it can't be considered complete.

[aranega]

@igordejanovic I performed the modification a feature/iterative-resolve branch. Beside the new "algorithm", it implies only minor modifications of the original code.

With these modifications, I launched the performance tests from tests/perf and I observed a slightly speed-up (performances are already great with textX). The gain was 1s for the big examples.

I can create a PR if you think that this modification can be useful for textX.

[igordejanovic]

@aranega Nice work! Looks good. I've merged your change upstream. Probably will check with some other projects using textX in the next days but so far works ok. It shows a speed increase. Not much but every bit counts! ;)

[aranega]

@igordejanovic Thanks! I'm glad the modification works for you! However, I've just seen that I use a .clear() on the corss-ref collection that will perhaps not fly with Python 2. A del self._crossrefs[:] should be better? Or the collection removal can also be removed (I was concerned about memory)?

[igordejanovic]

It is a one-time operation at the end of resolving so probably won't make much difference. A simple del self._crossrefs would probably be the easiest way.

[aranega]

Got it! I pushed a PR with the fix. Sorry about that, I totally forgot about Python 2.

[igordejanovic]

Thanks, no problem ;)

[aranega]

So, here is some news, I integrated PyEcore inside textX with a "switch" to enable or disable it. The switch is placed in the textx package and work like this:

import textx
from textx.metamodel import metamodel_from_str

textx.enable_pyecore_support()
# to disable it
textx.enable_pyecore_support(enable=False)

I put back all the original tests in the repository, copied the modified tests in a functional_pyecore test directory, and I'm writing new tests to tackle PyEcore specificities. There is still two points that are mandatory to have something clean, but it is almost the end!

I was also thinking about other usage of textX and PyEcore, and I was wondering if it could be interesting to have a textX grammar skeleton generation from an existing ecore file. This could help in the definition of concrete syntax for an existing metamodel. I have some example in mind where it could be very helpful (metamodel is here, but a concrete syntax is missing), but it's possible that they result from some confirmation bias phenomena. Could this be actually useful?

[igordejanovic]

Amazing work! Looks really great. There is an issue with Python 2.7 though:

E     File "/home/igor/repos/textX/textx/metamodel.py", line 72                                                                                                           
E       class TextXMetaModel(*__TextXMetaModel_bases()):                                                                                                                  
E                            ^                                                                                                                                            
E   SyntaxError: invalid syntax                                                                                                                                           

I would like to retain compatibility with Python 2 as there are still users who are for whatever reason stuck with it.

Anyway, when this is solved please send me a PR and I would be really happy to merge upstream.

If you find some time I would love to see a section in the docs dedicated to running textX with PyEcore. I could write it but not sure that I understand all bits and pieces and gotchas.

I was also thinking about other usage of textX and PyEcore, and I was wondering if it could be interesting to have a textX grammar skeleton generation from an existing ecore file. This could help in the definition of concrete syntax for an existing metamodel. I have some example in mind where it could be very helpful (metamodel is here, but a concrete syntax is missing), but it's possible that they result from some confirmation bias phenomena. Could this be actually useful?

Yes, I do think it is useful. This could be a fast track to the textual syntax editing for folks that created their meta-models in the graphical environments. I had a plan to support textX model serialization. With all of this, from an ecore meta-model you would get not only a nice textual syntax (and editor support - the ongoing effort) but also a nice human-readable serialization format. This would further enable using e.g. git for model versioning.