Thanks

[jjpe]

Unfortunately github doesn't allow for communication other than via bugs, so here we are. I simply wanted to say thanks, I'm likely to use this a bit later in my master thesis.

[skeeto]

Glad to hear it's useful for someone in some capacity! I'm curious, what is your master thesis about? What makes my project relevant?

[jjpe]

I'm going to integrate a language workbench (aka a compiler compiler or metacompiler, a toolset to create new programming languages, both DSLs and Turing Complete ones) into Emacs, and it will likely be with ØMQ (this is not quite certain yet but I currently think it will be hard to beat due to simplicity, speed and sheer flexibility). However, for all its language bindings, ELisp is not (yet!) one of them. If things continue to go in the direction they are, I intend to change that using your FFI to the C version of ØMQ.

[jjpe]

So I'm trying to get it to work on OS X (on my Linux box make && make test completes successfully), and I've gotten it to compile. But running make test segfaults with this output:

rand()

echo -n 'k0w0Cp0w4MrandSco' | ./ffi-glue invalid op: - invalid op: n /bin/sh: line 1: 34463 Done echo -n 'k0w0Cp0w4MrandSco' 34464 Segmentation fault: 11 | ./ffi-glue make: *** [test] Error 139

In order to get it to work, I've added -I/usr/local/Cellar/libffi/3.0.13/lib/libffi-3.0.13/include to the CXXFLAGS var of the Makefile. This path refers to libffi as installed with Homebrew for OS X, which is keg-only.

For good measure, I've also tried it with the native OS X ffi header, /usr/include/ffi/ffi.h: `

rand()

echo -n 'k0w0Cp0w4MrandSco' | ./ffi-glue invalid op: - invalid op: n /bin/sh: line 1: 35388 Done echo -n 'k0w0Cp0w4MrandSco' 35389 Segmentation fault: 11 | ./ffi-glue make: *** [test] Error 139

As you can see, pretty much identical output. And when I try to reproduce the steps of make test manually, after the emacs -batch ... line I get this output:

Running 2 tests (2015-04-29 12:57:25+0200) passed 1/2 ffi-cif

After which it hangs, though I can still kill it with ^C, and if there's any other useful info I can provide, please let me know.

Why does this happen? And what can be done about it?

[skeeto]

Looks like you've got some weird, broken version of the echo command. Maybe this is an OS X thing? It's not interpreting the -n as an argument to echo ("don't print the newline") and instead passing it through as input to the FFI program.

[jjpe]

I see what you mean. The interesting thing is that when I try echo -n 'k0w0Cp0w4MrandSco' | cat on the very same OS X shell (I use zsh on both Linux and OS X), it gives the expected output:

k0w0Cp0w4MrandSco%

The % denotes the End-Of-Line (EOL) here.

Also potentially useful is that if I remove the -n switch from the echo lines in the Makefile, on Linux make clean && make && make test still passes:

rm -f *.o ffi-glue g++ -std=c++11 -Wall -O2 ffi-glue.cc -ldl -lffi -lstdc++ -o ffi-glue

rand()

echo 'k0w0Cp0w4MrandSco' | ./ffi-glue 1804289383 $

cos(1.2)

echo 'd1.2d0d0w1Cp0w3McosSco' | ./ffi-glue 0.362357754476674 $ emacs -batch -Q -L . -l ffi-tests.el -f ert-run-tests-batch Running 2 tests (2015-05-01 14:45:45+0200) passed 1/2 ffi-cif passed 2/2 ffi-cos

Ran 2 tests, 2 results as expected (2015-05-01 14:45:45+0200)

In contrast, on OS X it hangs on the ffi-cos test, after which I need ^C to kill it:

rm -f *.o ffi-glue c++ -std=c++11 -Wall -O -I/usr/local/Cellar/libffi/3.0.13/lib/libffi-3.0.13/include ffi-glue.cc -ldl -lffi -lstdc++ -o ffi-glue

rand()

echo 'k0w0Cp0w4MrandSco' | xargs ./ffi-glue

cos(1.2)

echo 'd1.2d0d0w1Cp0w3McosSco' | xargs ./ffi-glue

emacs -batch -Q -L . -l ffi-tests.el -f ert-run-tests-batch Running 2 tests (2015-05-01 14:49:55+0200) passed 1/2 ffi-cif ^Cmake: *** [test] Error 2

Also notice that on OS X the echo lines don't yield any results whereas they do on Linux (e.g. 0.362357754476674 for the cos(1.2) test).

In the meantime I've started reading ffi-glue.cc, but haven't chewed through all of it just yet since C++ is not a programming language I speak natively. Is it possible there is a stdin-handling bug in the compiled ffi-glue binary?

UPDATE: I've read the ffi-glue.cc code and it seems correct to me. I've also checked which echo command I'm using, it's a built-in. That leaves me wondering where the issue comes from though. Any ideas?

[jjpe]

After more digging around, I get the code to execute farther by changing 2 things:

1. This is a hack, but it turns out to be a necessary one; I changed the Machine.dlsym() method to:

void dlsym() {
    const char *name = static_cast<const char *>(stack.pop().value.ptr);
    std::cout << "dlsym: Popped symbol '" << name << "'\n";
    void *handle = stack.pop().value.ptr;
    std::cout << "dlsym: Popped handle ptr from stack (" << handle << ")\n";
    handle = ::dlopen(0,RTLD_LAZY|RTLD_GLOBAL);
    std::cout << "dlsym: handle overwritten by ::dlopen (" << handle << ")\n";
    char* err_str;
    err_str = ::dlerror();
    if (err_str) {
        std::cout << "dlsym: pre error: " << err_str << "\n";
    }
    void *ptr = ::dlsym(handle, name);
    err_str = ::dlerror();
    if (err_str) {
        std::cout << "dlsym: post error: " << err_str << "\n";
    } else {
        std::cout << "dlsym: got ::dlsym ptr\n";
    }
    stack.push(ptr);
    std::cout << "dlsym: pushed 'fn ptr' (" << ptr << ") to the stack\n";
  }

Just ignore the std::cout lines, they are for debugging purposes. The main change is the handle = ::dlopen(0,RTLD_LAZY|RTLD_GLOBAL); line, without which the ffi-glue binary segfaults when it tries to execute void *ptr = ::dlsym(handle, name);.

1. the Reader.read_double() and Reader.read_float() impl to

void read_float() {
      float f = ::read_double(in_);
      // float f;         \  original impl
      // in_ >> f;        /
      vm_.stack.push(f);
  }

  void read_double() {
      double d = ::read_double(in_);
      // double d;        \  original impl
      // in_ >> d;        /
      vm_.stack.push(d);
  }

with as global fn's

bool is_float_char(char c) {
    return ('0' <= c && c <= '9') || (c == '.');
}

double read_double(std::istream& in_stream) {
    char dest[1024];
    int i = 0;
    char c = static_cast<char>(in_stream.peek());
    while ( is_float_char(c) ) {
        in_stream.get();
        dest[i] = c;
        i++;
        c = static_cast<char>(in_stream.peek());
    }
    dest[i] = '\0';
    return atof(dest);
}

What this accomplished for me is that the compiled ./ffi-glue binary actually parses the d (and f) directives now.

As an aside: when I run echo -n d1.2d0d0w1Cp0w3McosSco|./ffi-glue directly on my shell, I get the (approximately) correct value of 0.362357754476674. Same for echo -n k0w0Cp0w4MrandSco|./ffi-glue, which yields 16807. So it seems that either this use of C++'s istreams isn't as portable we thought, or perhaps it simply interacts poorly with make on OS X, or both.

[mentalisttraceur]

Re: "weird, broken version of the echo command": it's not broken, echo behavior is just unportable.

This is why the standard advice is to always use printf '%s\n' ... instead of echo ... and printf '%s' ... instead of echo -n ....

Also, re:

when I try echo -n [...] {on the very same OS X shell, directly in my shell}

make and your shell are not necessarily using the same echo. I don't remember if make calls the echo program (/bin/echo, iirc) or if it calls /bin/sh (which might have its own built-in), but either way, zsh's built-in is separate from both.

P.S. +1 thanks for this package