simpleflow.execute: Not all errrors are returned base64 encoded

[ybastide]

An OOM for instance won't be caught by try ... except. The wrapper then returns a bogus EOFError.

Testcase without OOM :innocent::

# zz.py
import os
from simpleflow import execute

@execute.python()
def stupid():
    pid = os.getpid()
    os.kill(pid, 9)

python
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import zz
>>> zz.stupid()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/simpleflow/execute.py", line 157, in execute
    base64.b64decode(excline.rstrip()))
EOFError
>>> 

[ggreg]

Thanks @ybastide for reporting the bug! It occurs because the process exits without encoding the traceback from stderr in base64.

The calling process may fallback to reading the string from stderr (most of the time an ImportError or a MemoryError) or to provide a consistent behavior it would use a intermediary process that only reads from its child stderr and reports its child status.

[ybastide]

@jbbarth: we could add

import warnings
warnings.simplefilter('ignore', RuntimeWarning)

to https://github.com/botify-labs/simpleflow/blob/master/simpleflow/execute.py in order to suppress subprocess32's RuntimeWarning: The _posixsubprocess module is not being used. Child process reliability may suffer if your program uses threads. (half tongue in cheek)

[jbbarth]

Great! But we could hide useful error messages with this, no?

Is it specific to our use of pypy, or generic error? If generic, yes, +1 on that idea, at the top of the file. If it's something specific to pypy maybe we could apply it but only if the interpreter is pypy (maybe hard to detect? we don't know how users may call the binary after all..).

Anyway fixing this problem would be great, even if it implies keeping a dirty secret among us three forever :-p

[ybastide]

Well... TBH any output by the callee will mess python.execute's workflow. Both for exception and the JSON result, which has bitten us before. Another possibility would be to use a specific fd. `python.execute would call

python -m simpleflow.execute --com-fd=3

and __main__ would output on --com-fd if specified, stdout otherwise.

[jbbarth]

I like the second idea, but that's not trivial to implement I guess? Let's keep it simple for now and apply the first idea as soon as you have the time. Thanks a lot !!