transpyle
transpyle copied to clipboard
mbdevpl
Matlab in Transpyle?
Would it be possible to include Matlab in the Transpyle Repertoire? In my recent reading I have found several promising approaches which may do most of the work for you. I attach my reading notes in case they are useful for you.
- Mc2For : described at http://www.sable.mcgill.ca/mclab/projects/mc2for/ uses a typed intermediate AST approach which might enable simple inclusion in the Transpyle architecture. Available in Java.
- "Matlab to Modelica" described at: http://www.ep.liu.se/ecp/096/137/ecp14096137.pdf also creates a typed intermediate AST
- Matisse/Lara described at: https://www.isi.edu/people/pedro/publications/matisse_matlab_compiler and https://web.fe.up.pt/~specs/projects/lara/doku.php?id=lara:usage#matisse An online demo is at:http://specs.fe.up.pt/tools/matisse/
- Rose compiler.There is evidence of some work on a matlab frontend for Rose at: https://github.com/rose-compiler/matlab-frontend
- Unparser for Python might also be helpful. See: https://mbdevpl.github.io/posts/typed-astunparse/
- MIX10: compiling MATLAB to X10 for high performance Kumar, Vineet & Hendren, Laurie. (2014). MIX10: compiling MATLAB to X10 for high performance. Proceedings of the Conference on Object-Oriented Programming Systems, Languages, and Applications, OOPSLA. 617-636. 10.1145/2660193.2660218.
- https://github.com/jonathf/matlab2cpp & https://ieeexplore.ieee.org/document/7542966 8.https://stackoverflow.com/questions/9845292/a-tool-to-convert-matlab-code-to-python Has a very useful listing of relevant software.
- https://github.com/ebranlard/matlab2fortran
Hi! Thanks for the interest in Transpyle!
And thanks for a long list of resources - from a quick read I think they would be definitely very useful for adding Matlab support into the framework.
Short comment about 5. - it's actually something that also I developed, and then I extended it in various ways to be used in Transpyle project for unparsing Python, Fortran and C++. I completely agree that this approach simplified unparser implementation greatly, and I suspect that adding only unparsing Python AST to Matlab code might be be actually quite relaxing coding time, since the unparsing machinery is quite robust and you have multiple existing extensions as examples to guide you.
And about 4. - ROSE AST is a very relevant concept for my work too, and one of the things I'd be very interested in looking at is actually what challenges ROSE team has faced when trying to add Matlab support. Did they abort due to some technical difficulty? Or no interest in the community? Or is the effort going on?
The same with LLVM IR since we're talking about widely-adopted formats - are you aware of any Matlab -> LLVM IR translators?
On the other hand, I have to admit that even if adding Matlab to Transpyle would be not only theoretically possible, but also seems feasible, currently there is little chance of such functionality to enter my development roadmap. I'm focusing on performance-oriented functionalities, and mostly work to improve transformations performed on typed Python AST (such as inlining, loop transformations, etc.), as well as unparsing Python->Fortran and Python->C++. In the future, I'm possibly gonna work on OpenCL (which is C-based, so earlier C++ efforts will come in handy), as well as possibly Cython which is a very very close cousin of Python and as such might be easiest of all to add - however the performance of Cython is in general worse than pure C++ or Fortran, so Cython might not happen anytime soon.
When speaking about Matlab in terms of performance, I can totally see the point of translating it into Fortran. And I can see that transpilation like: Matlab -> Python AST -> Fortran would probably be made possible with one small addition of some AST translator that takes output of one of the existing Matlab parsers and translates it into Python AST. I rely on this idea already for parsing Fortran and parsing C++ headers, and in Transpyle there is even "XmlAstGeneralizer" class (in https://github.com/mbdevpl/transpyle/blob/master/transpyle/general/ast_generalizer.py) that helps a lot to implement translation from foreign XML-based ASTs into Python AST.
Adding Matlab support should not conflict in any way with what I'm doing - it's just that I already have a lot of things on my to-do list, so I can't say that I can devote too much attention for it. So if you'd be willing to lead the work on it, I would welcome and help as much as I can. And as author of most of the code of the framework, I would probably be able to immediately suggest a solution to almost any implementation problem.
And finally, I'm very interested - what motivated you to consider adding Matlab support here?
Sorry for my slow response to your email dated 26 April. I have been travelling and dealing with other issues which distracted me from hobby interests.
I also wanted to have a play with Transpyle before responding. It is downloading as I write. I’ll respond more fully to your email in a day or two.
I had some difficulty getting the Docker download to work because of a problem specifying the tag. “latest” does not work. (MacOs High Sierra)
Here is my terminal output:
Record of TRANSPYLE Docker Install
xxx$ docker search transpyle NAME DESCRIPTION STARS OFFICIAL AUTOMATED mbdevpl/transpyle Ready-to-use transpiler. 0
xxx$ docker pull mbdevpl/transpyle Using default tag: latest Error response from daemon: manifest for mbdevpl/transpyle:latest not found
xxx$ docker pull mbdevpl/transpyle:taginfo Error response from daemon: manifest for mbdevpl/transpyle:taginfo not found
xxx$ sudo docker pull mbdev/transpyle Password: Using default tag: latest Error response from daemon: pull access denied for mbdev/transpyle, repository does not exist or may require 'docker login'
xxx$ docker login Authenticating with existing credentials... Login Succeeded
xxx$ docker pull mbdevpl/transpyle:taginfo Error response from daemon: manifest for mbdevpl/transpyle:taginfo not found
FIX After reading this: https://github.com/elastic/elasticsearch-docker/issues/215 and in particular the comment by xeraa I went to: https://hub.docker.com/r/mbdevpl/transpyle/tags got the latest number and added a numerical tag and it all worked. I think a note on your docker hub page would be helpful. It would have saved an hour of messing around.
xxx$ docker pull mbdevpl/transpyle:0.7.1 0.7.1: Pulling from mbdevpl/transpyle f476d66f5408: Pull complete etc …... Status: Downloaded newer image for mbdevpl/transpyle:0.7.1
The run command did not work as written. The tag had to be added: docker run -h transmachine -it "mbdevpl/transpyle"
To get it to run I had to add the tag as follows: xxx$ docker run -h transmachine -it "mbdevpl/transpyle:0.7.1”
Incidentally on: https://pypi.org/project/transpyle/#requirements there is a significant spelling mistake. It should read notebook not notebok. python3 -m jupyter notebok --ip="$(hostname -i)" --port=8080 I’m still fiddling with the jupyter notebook interface & learning how to set up ports to Jupyter. The command line works though.
Regards Rob
Oh, thanks for trying it out! Indeed the:
- typo in readme, and
- missing "latest" tag on the Docker Hub
are my mistakes. I fixed those (and "latest" tag now corresponds to master branch), so hopefully at least future users won't run into the same trouble. Sorry for the trouble and thanks again for reporting the issues!
If you run into anything else unexpected, please let me know.
I am a long retired electrical engineer with an interest in image processing. I learnt Fortran in 1962 and try to keep my hand by learning Python and experimenting with algorithms like your dehazer and the recently published deblurring using extreme priors.
Regards Rob
On 26/04/2019, at 14:19, Mateusz Bysiek [email protected] wrote:
Hi! Thanks for the interest in Transpyle!
And thanks for a long list of resources - from a quick read I think they would be definitely very useful for adding Matlab support into the framework.
Short comment about 5. - it's actually something that also I developed, and then I extended it in various ways to be used in Transpyle project for unparsing Python, Fortran and C++. I completely agree that this approach simplified unparser implementation greatly, and I suspect that adding only unparsing Python AST to Matlab code might be be actually quite relaxing coding time, since the unparsing machinery is quite robust and you have multiple existing extensions as examples to guide you.
And about 4. - ROSE AST is a very relevant concept for my work too, and one of the things I'd be very interested in looking at is actually what challenges ROSE team has faced when trying to add Matlab support. Did they abort due to some technical difficulty? Or no interest in the community? Or is the effort going on?
The same with LLVM IR since we're talking about widely-adopted formats - are you aware of any Matlab -> LLVM IR translators?
On the other hand, I have to admit that even if adding Matlab to Transpyle would be not only theoretically possible, but also seems feasible, currently there is little chance of such functionality to enter my development roadmap. I'm focusing on performance-oriented functionalities, and mostly work to improve transformations performed on typed Python AST (such as inlining, loop transformations, etc.), as well as unparsing Python->Fortran and Python->C++. In the future, I'm possibly gonna work on OpenCL (which is C-based, so earlier C++ efforts will come in handy), as well as possibly Cython which is a very very close cousin of Python and as such might be easiest of all to add - however the performance of Cython is in general worse than pure C++ or Fortran, so Cython might not happen anytime soon.
When speaking about Matlab in terms of performance, I can totally see the point of translating it into Fortran. And I can see that transpilation like: Matlab -> Python AST -> Fortran would probably be made possible with one small addition of some AST translator that takes output of one of the existing Matlab parsers and translates it into Python AST. I rely on this idea already for parsing Fortran and parsing C++ headers, and in Transpyle there is even "XmlAstGeneralizer" class (in https://github.com/mbdevpl/transpyle/blob/master/transpyle/general/ast_generalizer.py https://github.com/mbdevpl/transpyle/blob/master/transpyle/general/ast_generalizer.py) that helps a lot to implement translation from foreign XML-based ASTs into Python AST.
Adding Matlab support should not conflict in any way with what I'm doing - it's just that I already have a lot of things on my to-do list, so I can't say that I can devote too much attention for it. So if you'd be willing to lead the work on it, I would welcome and help as much as I can. And as author of most of the code of the framework, I would probably be able to immediately suggest a solution to almost any implementation problem.
And finally, I'm very interested - what motivated you to consider adding Matlab support here?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/mbdevpl/transpyle/issues/7#issuecomment-486901639, or mute the thread https://github.com/notifications/unsubscribe-auth/ABRNQJQD2PKFU5MTDNO5W73PSJRDZANCNFSM4HIJMFFA.
Sorry for the long delay in responding to your questions. Life got in the way! Before I address your questions, a bit about myself. I am a long retired electrical engineer aged 79+ with a hobby interest in image processing and restoration.. I learnt to program in 1962 at University, but only used Fortran & Basic in my work for about 10 years before moving into management. In another life, I would have gone into academia and happily become a researcher. Instead I ended up managing engineers and funding research projects as a sideline.
I am now long retired, but keep my brain active by learning Python and experimenting with image processing algorithms in my spare time.
Now to your main question:
And finally, I'm very interested - what motivated you to consider adding Matlab support here?
About 10 years ago I realised that a lot of valuable research work is being done using Matlab a tool which is inaccessible for many individuals due to high cost. The problem for a sole practitioner or a retiree like me is that Matlab is very expensive to licence and its internals are quite opaque.
For many image processing problems it is also quite slow, which is even more true of Octave.
I decided that my retirement contribution to the world of open source could be to encourage and incrementally improve bridging tools (transpilers) which would make research results more rapidly transferable to free and open languages like Python where there are many ways of accelerating performance. Along the way I would translate any Matlab code that was useful to me into Python and then work to speed it up as necessary. That was the dream!
In reality, my progress has been very slow. For some time I’ve contributed bugs & suggestions to the development of “Smop a Small Matlab to Python compiler” ( https://github.com/victorlei/smop https://github.com/victorlei/smop ) and on Stack Overflow. However recent progress with Smop has been slow and the original author seems to have lost interest after updating it to Python3. Smop is a very good tool but, could it be better or easier to use? There are hopeful signs eg https://github.com/PatrickFURI/smop https://github.com/PatrickFURI/smop which is good when its good and bad when I don’t understand what I’m doing.
That was when I stumbled over Transpyle and wrote to you. I saw that making transpiling from Matlab directly to Python straightforward removes barriers of cost and performance and that all the benefits of performance enhancement work in Python will flow to many more people. I had been reading up on various ways of doing this and that was the source of the list of references I sent you in April and of the notes I attach below.
I saw your project as a potential focus point for transpiler work involving Python, Matlab & Fortran as major languages used in the science and engineering world. At present the scene is quite incoherent and would benefit from a focal point for discussion.
I answer your other questions by adding * text to the copy below.
Regards Rob
On 26/04/2019, at 14:19, Mateusz Bysiek [email protected] wrote:
Hi! Thanks for the interest in Transpyle!
And thanks for a long list of resources - from a quick read I think they would be definitely very useful for adding Matlab support into the framework.
Short comment about 5. - it's actually something that also I developed, and then I extended it in various ways to be used in Transpyle project for unparsing Python, Fortran and C++. I completely agree that this approach simplified unparser implementation greatly, and I suspect that adding only unparsing Python AST to Matlab code might be be actually quite relaxing coding time, since the unparsing machinery is quite robust and you have multiple existing extensions as examples to guide you.
And about 4. - ROSE AST is a very relevant concept for my work too, and one of the things I'd be very interested in looking at is actually what challenges ROSE team has faced when trying to add Matlab support. Did they abort due to some technical difficulty? Or no interest in the community? Or is the effort going on?
- I have searched diligently for more information about Rose and found nothing helpful.
The same with LLVM IR since we're talking about widely-adopted formats - are you aware of any Matlab -> LLVM IR translators?
- See my attached notes on Matlab>LLVM using Sable
On the other hand, I have to admit that even if adding Matlab to Transpyle would be not only theoretically possible, but also seems feasible, currently there is little chance of such functionality to enter my development roadmap. I'm focusing on performance-oriented functionalities, and mostly work to improve transformations performed on typed Python AST (such as inlining, loop transformations, etc.), as well as unparsing Python->Fortran and Python->C++. In the future, I'm possibly gonna work on OpenCL (which is C-based, so earlier C++ efforts will come in handy), as well as possibly Cython which is a very very close cousin of Python and as such might be easiest of all to add - however the performance of Cython is in general worse than pure C++ or Fortran, so Cython might not happen anytime soon.
When speaking about Matlab in terms of performance, I can totally see the point of translating it into Fortran. And I can see that transpilation like: Matlab -> Python AST -> Fortran would probably be made possible with one small addition of some AST translator that takes output of one of the existing Matlab parsers and translates it into Python AST.
- One of the things that has hindered development of transpilers for Matlab was lack of an official grammar. This has been resolved recently with https://github.com/mathworks/MATLAB-Language-grammar/tree/master/Matlab.tmbundle/Syntaxes https://github.com/mathworks/MATLAB-Language-grammar/tree/master/Matlab.tmbundle/Syntaxes (4 months ago)
This grammar has been put to use in Atom code editor via their new parser generator tool Tree-Sitter. Tree-Sitter can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited. It has an impressive list of languages it can parse which may be interesting to you. It is now integrated into Atom (as of the last month) https://tree-sitter.github.io/tree-sitter/ https://tree-sitter.github.io/tree-sitter/ and looks as if it can deal effectively with some of the trickier problems of languages like Matlab (see Tree-Sitter video and http://www.rath.org/matlab-is-a-terrible-programming-language.html http://www.rath.org/matlab-is-a-terrible-programming-language.html). There is also a Beta Syntax tree viewer (I haven’t tried it).
- I don’t know if Type information is handled by Tree-Sitter but I think Sable (Mc2For) & Matisse can generate type information from Matlab code.
I rely on this idea already for parsing Fortran and parsing C++ headers, and in Transpyle there is even "XmlAstGeneralizer" class (in https://github.com/mbdevpl/transpyle/blob/master/transpyle/general/ast_generalizer.py https://github.com/mbdevpl/transpyle/blob/master/transpyle/general/ast_generalizer.py) that helps a lot to implement translation from foreign XML-based ASTs into Python AST.Adding Matlab support should not conflict in any way with what I'm doing - it's just that I already have a lot of things on my to-do list, so I can't say that I can devote too much attention for it.
- I have found other sites for transpilers covering a range of languages but yours is the most current and potentially comprehensive. It has the bonus of focussing on performance enhancement as well which is often a motivation for switching languages in the first place. I’ll attach a list of other transpilers I’ve found. (Transpilers - reading notes.pdf)
So if you'd be willing to lead the work on it, I would welcome and help as much as I can. And as author of most of the code of the framework, I would probably be able to immediately suggest a solution to almost any implementation problem.
- If I were younger I would like to do something like this but it is well beyond my current skill level. Also my age is beginning to tell on my ability to complete such work. I do however believe that there is a need for a focal point to pull together all the information about similar such projects (an Awesome list perhaps) and prod the whole field of transpilation work toward a coherent and cooperative attack. I thought the Rose Project was such a thing but was disappointed by what I found. Look what the NTIRE challenges do for the Image Processing field.
And finally, I'm very interested - what motivated you to consider adding Matlab support here?
- See my answer in the beginning.
Are you aware of mlabpy at: https://github.com/led02/mlabpy ? It claims to produce a python AST from Matlab. It is a development based on smop. I have not tested it.
There is a new and open source tool for parsing and lexing Matlab at: https://misshit.org Github repository at https://github.com/florianschanda/miss_hit It is described as follows:
"There have been other attempts at this, but MISS_HIT tries to do things differently (and I hope better). Specifically: • It is completely stand-alone. It does not require a working MATLAB or Octave setup. • It is not just a bag of reg-ex, or a wrapper arounds some undocumented MATLAB internals. It is based on a feature complete lexer and parser, which supports every single weird feature of the MATLAB language. • It contains features specifically for huge industrial code-bases (i.e. millions of lines): multi-threaded analysis and a sophisticated configuration and justification mechanism. • Designed for both interactive use during development, and integration into modern CI environments."_
It will undoubtedly be a good resource for extending the Matlab capabilities of transpyle.