transplant
transplant copied to clipboard
bastibe
Octave support
wondering whether it would be hard to support octave, given that oct2py does not seem to be developed anymore (last release was ~2 years ago)
Most of the core interaction in transplant should work without issue in Octave. The problem is the communication channel. Transplant uses zeromq as its socket implementation, and calls into libzmq using loadlibrary in Matlab. Octave does not support loadlibrary. The interface wouldn't be hard to implement, though, probably as an oct-file.
If you'd like to take a stab at this and re-implement ZMQ.m for Octave, I'd be grateful for a pull request! As long as the signature of ZMQ.m is not changed, I suspect the rest of Transplant would work with few modifications.
But I am not a user of Octave, and probably won't find the time to do it myself.
I checked and it seemed that the main issue against implementing loadlibrary in Octave is not there anymore, so I opened a Matlab compatibility issue in Octave to see if an official implementation of the function could be possible (https://savannah.gnu.org/bugs/index.php?56176).
Are there other things that need to be adapted in addition to loadlibrary?
loadlibrary is a significant low-level function, so it's unlikely to make it into a released Octave any time soon.
But there's an Octave Forge zeromq package that provides ZeroMQ bindings for Octave using the oct-file linkage mechanism. Maybe you could write a ZMQ.m for Octave that uses that?
(BTW, Transplant looks like a cool library. I've added it to my public list of interesting FLOSS Matlab projects.)
Oh, cool! The zeromq package on Octave Forge should do the trick! It would probably be easiest to make ZMQ.m Octave-aware, and have a special code path in Octave that uses the zeromq package.
I installed the zeromq package.
It would probably be easiest to make ZMQ.m Octave-aware
Sounds like the way to go... I never used zeromq before, but will try...
have a special code path in Octave that uses the zeromq package
what do you mean by special code path in Octave? (sorry if this is obvious)
My idea is to have all Octave-specific code confined to ZMQ.m if possible. So we would have to modify ZMQ.m to have two code-paths, one using loadlibrary for Matlab, and one using the zeromq package for Octave. Both code paths (essentially ifs) should be able to co-exist in the same file.
Would you like to take a stab at this? I'd be really grateful for a pull request!
Are you aware of this: https://github.com/esromneb/zeromq-octave https://github.com/esromneb/zeromq-octave ? and this: https://stackoverflow.com/questions/18877353/is-there-an-implementation-of-zmq-in-octave https://stackoverflow.com/questions/18877353/is-there-an-implementation-of-zmq-in-octave
May be helpful
Regards Rob
On 21/04/2019, at 23:11, Bastian Bechtold [email protected] wrote:
My idea is to have all Octave-specific code confined to ZMQ.m if possible. So we would have to modify ZMQ.m to have two code-paths, one using loadlibrary for Matlab, and one using the zeromq package for Octave. Both code paths (essentially ifs) should be able to co-exist in the same file.
Would you like to take a stab at this? I'd be really grateful for a pull request!
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bastibe/transplant/issues/74#issuecomment-485243378, or mute the thread https://github.com/notifications/unsubscribe-auth/ABRNQJXICC7O55GHDSGE2FTPRRDWDANCNFSM4HGUATNQ.
Would you like to take a stab at this? I'd be really grateful for a pull request!
Sure
I started with it, but found out that zeromq package does not support zmq_ctx_new and zmq_ctx_term. I think it may be ok, but wanted to check it out, as I am a zmq newbie :)
Beyond that, there are issues in the arguments to pass to Matlab vs. Octave. Octave does not support -nojvm and -nodesktop (there is --no-window-system and --no-gui, though), and -r functionality is accomplished with --eval.
I have a way to distinguish matlab from octave in m-code (in the context of ZMQ.m), but we will need of sth similar in python (in the context of transplant_master.py). What do you think would be a good way to add this?I guess we can add a transplant input argument to specify matlab vs. octave...
Mmm, I'm stacked with this:
msg = libstruct('zmq_msg_t', struct('hidden', zeros(1, 64, 'uint8')));
calllib('libzmq', 'zmq_msg_init', msg); % always returns 0
msglen = calllib('libzmq', 'zmq_msg_recv', msg, obj.socket, 0);
assert(msglen >= 0, obj.errortext('zmq_msg_recv'));
msgptr = calllib('libzmq', 'zmq_msg_data', msg);
if not(msgptr.isNull)
setdatatype(msgptr, 'uint8Ptr', 1, msglen);
str = uint8(msgptr.Value);
else
str = uint8([]);
end
err = calllib('libzmq', 'zmq_msg_close', msg);
assert(err == 0, obj.errortext('zmq_msg_close'));
Octave's zero_mq does not support zmq_msg_* functions, only zmq_recv... :/
EDIT: not a problem, octave forge package simplifies all this complexity (context and msg functions are internal)
I have a way to distinguish matlab from octave in m-code (in the context of ZMQ.m), but we will need of sth similar in python (in the context of transplant_master.py). What do you think would be a good way to add this?I guess we can add a transplant input argument to specify matlab vs. octave...
True, I hadn't thought of that. We'll have to start a different binary, too. I think the simplest way to do this would be a subclass of Matlab or TransplantMaster that calls octave instead of matlab, with the appropriate arguments.
True, I hadn't thought of that. We'll have to start a different binary, too. I think the simplest way to do this would be a subclass of Matlab or TransplantMaster that calls octave instead of matlab, with the appropriate arguments.
I implemented this the easy way, simply adding an argument called backend to the Matlab class, which defaults to "matlab" (vs. "octave"). And then a simple conditional sets the proper arguments. I hope it is ok. This is sth that can be improved later if you are not convinced about it.
We'll have a look at the API once it actually works. For now, this is perfectly fine. (I am a strong believer in "tracer bullets": prototype implementations that solve the central problem, but none of the edge cases. Once a prototype is known to work, I like to iterate on the surrounding API. But I find it wasted effort to focus too much on the API while still working on the core algorithms; Too many things still change at that stage, to concern oneself with such details.)
I get the following errors in Octave in comparison to Matlab:
Matlab:
libzmq loaded
socket done
connection done
receiving msg:
{"name":"eval","type":"get_global"}
msg received.
sending msg:
{"type":"value","value":["__function__","eval"]}
msg sent.
receiving msg:
{"name":"eval","args":["0;"],"nargout":-1,"type":"call"}
msg received.
sending msg:
{"type":"value","value":0}
Octave:
zeromq loaded
socket done
connection done
receiving msg:
{"name":"eval","type":"get_global"}
msg received.
sending msg:
{"identifier":"Octave:undefined-function","message":"'idx' undefined near line 236 column 13","stack":[{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson>false","line":236,"column":11,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson>object","line":143,"column":9,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson>value","line":50,"column":20,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/parsejson.m","name":"parsejson","line":27,"column":16,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/transplant_remote.m","name":"transplant_remote>receive_msg","line":200,"column":21,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/transplant_remote.m","name":"transplant_remote","line":71,"column":17,"scope":null,"context":0}],"type":"error"}
msg sent.
receiving msg:
{"name":"what","type":"get_global"}
msg received.
sending msg:
{"type":"value","value":["__function__","what"]}
msg sent.
receiving msg:
{"name":"what","args":["eval"],"nargout":-1,"type":"call"}
msg received.
sending msg:
{"identifier":null,"message":"what: could not find the directory eval","stack":[{"file":"\/app\/share\/octave\/5.1.0\/m\/miscellaneous\/what.m","name":"what","line":85,"column":9,"scope":null,"context":0},{"file":"\/home\/miniconda\/envs\/cenv1\/lib\/python3.7\/site-packages\/transplant\/transplant_remote.m","name":"transplant_remote","line":129,"column":38,"scope":null,"context":0}],"type":"error"}
msg sent.
receiving msg:
And then Octave gets stuck forever in there....
There are errors while sending info coming from transplant_remote.m when calling the "what" and "idx". The first exists in Octave but there should be some sort of compatibility issue. Not sure what is going on with "idx".
This is getting trickier as the mcode does not seem octave compatible...
EDIT: I confused "message" with "idx", fixed.
I'm a bit lost with those errors, if you know how to deal with that, please let me know...
I guess the original issue is that Octave is not able to send this message:
{"type":"value","value":["__function__","eval"]}
I think it first gets lost in parsejson.m. Can you try and see if parsejson works in octave at all?
Wow, you have a whole JSON decoder in there!
You might not need that. Matlab R2016b and newer have jsonencode() and jsondecode() functions now. Octave doesn't but I've written a JSON package for it that provides Matlab-compatible jsonencode and jsonencode implementations: https://github.com/apjanke/octave-jsonstuff
I think it first gets lost in parsejson.m. Can you try and see if parsejson works in octave at all?
In addition to parsejson, it seems there is sth else. When using msgpack, the messages look the same, however Matlab properly ends communication whereas Octave gets stuck:
Matlab:
libzmq loaded
socket done
connection done
receiving msg:
Columns 1 through 13
130 164 110 97 109 101 164 101 118 97 108 164 116
Columns 14 through 26
121 112 101 170 103 101 116 95 103 108 111 98 97
Column 27
108
msg received.
sending msg:
Columns 1 through 13
130 164 116 121 112 101 165 118 97 108 117 101 165
Columns 14 through 26
118 97 108 117 101 146 172 95 95 102 117 110 99
Columns 27 through 37
116 105 111 110 95 95 164 101 118 97 108
msg sent.
receiving msg:
Columns 1 through 13
132 164 110 97 109 101 164 101 118 97 108 164 97
Columns 14 through 26
114 103 115 145 162 48 59 167 110 97 114 103 111
Columns 27 through 39
117 116 255 164 116 121 112 101 164 99 97 108 108
msg received.
sending msg:
Columns 1 through 13
130 164 116 121 112 101 165 118 97 108 117 101 165
Columns 14 through 26
118 97 108 117 101 203 0 0 0 0 0 0 0
Column 27
0
msg sent.
Octave:
zeromq loaded
socket done
connection done
receiving msg:
Columns 1 through 16:
130 164 110 97 109 101 164 101 118 97 108 164 116 121 112 101
Columns 17 through 27:
170 103 101 116 95 103 108 111 98 97 108
msg received.
sending msg:
Columns 1 through 16:
130 164 116 121 112 101 165 118 97 108 117 101 165 118 97 108
Columns 17 through 32:
117 101 146 172 95 95 102 117 110 99 116 105 111 110 95 95
Columns 33 through 37:
164 101 118 97 108
msg sent.
receiving msg:
Columns 1 through 16:
132 164 110 97 109 101 164 101 118 97 108 164 97 114 103 115
Columns 17 through 32:
145 162 48 59 167 110 97 114 103 111 117 116 255 164 116 121
Columns 33 through 39:
112 101 164 99 97 108 108
msg received.
sending msg:
Columns 1 through 16:
130 164 116 121 112 101 165 118 97 108 117 101 165 118 97 108
Columns 17 through 27:
117 101 203 0 0 0 0 0 0 0 0
msg sent.
receiving msg:
You might not need that. Matlab R2016b and newer have
jsonencode()andjsondecode()functions now.
It does indeed, but the official implementation is slower for long strings than my implementation, and uses structs instead of containers.Map for JSON objects, which breaks JSON objects with non-variable-name keys.
@kupiqu, could you print the decoded messages instead of the binary blobs?
At least it is heartening to see that my msgpack parser seems to work fine in Octave.
Unfortunately Octave crashes when I try to do so, that's why I displayed binary data:
zeromq loaded
socket done
connection done
receiving msg:
msg received.
Exception in thread Thread-1:
Traceback (most recent call last):
File "/home/miniconda/envs/cenv1/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/home/miniconda/envs/cenv1/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/home/miniconda/envs/cenv1/lib/python3.7/site-packages/transplant/transplant_master.py", line 127, in reader
print(line.decode(), end='')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 0: invalid start byte
Actually, not sure it's octave that crashes here or just python :/
Unfortunately I don't have much time at the moment to continue digging, and even if I had some I am not sure how to proceed further.
I attach the two files I changed (ZMQ.m and transplant_master.py), with the hope that you, or somebody else, can fix the remaining issues.
Thank you very much for your efforts!
If anyone else wants to take a stab at this, I would be grateful!
The message "UnicodeDecodeError: 'utf-8' codec can't decode byte 0x.." reminded me of something I encountered about 2 years ago in relation to a similar error in SMOP:
[victorlei/smop] UnicodeDecodeError (#155)
It is a generic problem with non unicode characters in a file.
See here: https://wiki.python.org/moin/UnicodeDecodeError and
"Processing Text Files in Python 3 — Nick Coghlan's Python Notes":
http://python-notes.curiousefficiency.org/en/latest/python3/text_file_processing.html
These notes point out that it is a problem arising from the Python2/3 transition and how to resolve it.
Transplant is great and adding Octave would be a big plus. Unfortunately I'm too old and unskilled to have ago at fixing the software.
Just found some old notes on the UnicodeDecodeError: 'utf-8' issue.
Here's the gotcha to look out for: https://www.gsqi.com/marketing-blog/utf-8-bom-robots-txt/ concerning errors from invisible characters.
I encounterd it when trying out a variant of SMOP: https://github.com/PatrickFURI/smop which stopped when processing some m-files containing non utf-8 and invisibles .
The following links were informative too:
https://realpython.com/python-encodings-guide/
http://utf8everywhere.org
It's possible that the affected python code has been fixed since kupiqu ran his tests in 2019 and that there is nothing to repair in Transplant. One can always hope!