miasm
miasm copied to clipboard
cea-sec
Fix MacOS 10.14 build
Hello Team,
Here is a patch that fixes the build on MacOS 10.14. When checkouting the code and building, I came accross this error:
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -pipe -Os -isysroot/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c miasm2/jitter/vm_mngr_py.c -o build/temp.macosx-10.14-x86_64-2.7/miasm2/jitter/vm_mngr_py.o
miasm2/jitter/vm_mngr_py.c:553:24: error: use of undeclared identifier '__BIG_ENDIAN'
self->vm_mngr.sex = __BIG_ENDIAN;
^
miasm2/jitter/vm_mngr_py.c:561:24: error: use of undeclared identifier '__LITTLE_ENDIAN'
self->vm_mngr.sex = __LITTLE_ENDIAN;
^
miasm2/jitter/vm_mngr_py.c:570:27: error: use of undeclared identifier '__BIG_ENDIAN'
if (self->vm_mngr.sex == __BIG_ENDIAN) {
This patch fixes this. Feel free to come back to me if you want me to change anything. Many thanks for this great project and your work :).
Hi!
Thanks for the patch :smiley: Just a question: I don't have a mac to test this, so does this patch fixes completely the compilation on mac, or are there any more fixes to do?
Hi,
This fixes the build on Mac OS Mojave 10.14 without anything else to do. However I just realized I haven't ran the tests on Mac OS. I can do it this evening (timezone Paris, UTC+1) and come back to you with the report.
Hi,
Sorry for the delay. I tried to run the tests on MacOS but it appears they are stuck at x86/unit/mn_strings.py llvm:
TEST/ARCH x86/arch.py
DONE x86/arch.py 16s
TEST/ARCH x86/sem.py gcc
DONE x86/sem.py gcc 0s
TEST/ARCH x86/unit/mn_strings.py llvm
I'll see what I can do to fix this
Strange.
Can you try it after deleting your /tmp/miasm_cache ? (just to be sure there everything is generated by your current version)
Maybe you can see in a top which script exactly takes time?
Hi again,
I checked, but couldn't find the miasm_cache in the /tmp directory.
I used pstree to see which script was stuck, top wasn't giving the file path.
Here is the filtered output:
| | \-+= 00486 nicolasrouviere -zsh
| | \-+= 01803 nicolasrouviere python test_all.py -m
| | \-+- 01807 nicolasrouviere python test_all.py -m
| | \--- 01833 nicolasrouviere /Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/bin/python x86/unit/mn_strings.py llvm
I used the -m option to make it more easy to debug. I'll continue investigating, see what I can find.
Ok. So maybe we have the guilty here. Just another (private?) question: are you on a Mac with powerPC (I mean little endian) or x86?
It's an x86-64 version:
$ sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
Hello,
Sorry for the delay. The issue seems to be with communicate in testset.py line 61:
testpy = subprocess.Popen(([executable] +
init_args + test.command_line),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
cwd=test.base_dir)
outputs = testpy.communicate()
I tried again a few days ago but since then, I've been unable to reproduce the issue again. The only thing I did is, remove the stdout=subprocess.PIPE and stderr=subprocess.PIPE, run the tests (which passed, this time), put the parameters back and run the tests again. Since then, no more deadlock.
The results are the following:
Result: 376/471 pass
Many tests fail with the following stack trace:
ERROR x86/unit/mn_int.py gcc
ld: can't link with bundle (MH_BUNDLE) only dylibs (MH_DYLIB) file '/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/miasm2/jitter/VmMngr.so' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Traceback (most recent call last):
File "x86/unit/mn_int.py", line 39, in <module>
[test(*sys.argv[1:])() for test in [Test_INT]]
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/miasm/test/arch/x86/unit/asm_test.py", line 32, in __call__
self.run()
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/miasm/test/arch/x86/unit/asm_test.py", line 38, in run
self.myjit.continue_run()
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/miasm2/jitter/jitload.py", line 387, in continue_run
return self.run_iterator.next()
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/miasm2/jitter/jitload.py", line 358, in runiter_once
self.pc = self.run_at(self.pc)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/miasm2/jitter/jitload.py", line 315, in run_at
set(self.breakpoints_handler.callbacks.keys())
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/miasm2/jitter/jitcore.py", line 186, in run_at
cur_block = self.disasm_and_jit_block(offset, cpu.vmmngr)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/miasm2/jitter/jitcore.py", line 163, in disasm_and_jit_block
self.add_block(cur_block)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/miasm2/jitter/jitcore_gcc.py", line 87, in add_block
check_call(args)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 190, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cc', '-O3', '-shared', '-fPIC', '/var/folders/9l/2p7mz59x49n6dfbw7q7vsdl00000gn/T/tmpjUdF3r.c', '-o', '/var/folders/9l/2p7mz59x49n6dfbw7q7vsdl00000gn/T/tmp5k0gpt.so', '-I/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/miasm2/jitter', '-I/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7', '/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/miasm2/jitter/VmMngr.so', '/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/miasm2/jitter/arch/JitCore_x86.so']' returned non-zero exit status 1
I'm not sure but I reckon this is linked to gcc not installed. I tried to install it and run the tests again, but with the same results.
Feel free to comment the issue if you need anything else.
Hi!
I don't know and understand the problem. I don't have a Mac, so I am afraid I won't help a lot here.
Hi,
Did you take a look at #499? Maybe there are some fix in this pending PR (unfortunately not updated for a while) that might help you.
Hi,
I am using Miasm on Mac as well (10.14 on x86_64), and I get same errors as mentioned by @nirouviere. If you see the error, there is '.so' files in /miasm2/jitter/VmMngr.so, which ld is not able to link, as obviously a dylib is needed on MacOS. If during the build we can get dylibs created, this issue can possibly be fixed.
I will try to look into how to fix it, meanwhile, any hints are welcome. Thanks.
Hi @su-vikas, Thank you for the hint, maybe it will help in debugging the build on MacOS!
I looked into this a bit, as in this answer - https://stackoverflow.com/a/32765319/769407, I changed the setup.py file. After this I am not getting the linker error anymore, instead I get the following error now:
ERROR x86/unit/mn_daa.py gcc
Traceback (most recent call last):
File "x86/unit/mn_daa.py", line 78, in <module>
[test(*sys.argv[1:])() for test in [Test_DAA]]
File "/Users/lostboy/toolbox/tools/symbolic_execution/miasm/test/arch/x86/unit/asm_test.py", line 32, in __call__
self.run()
File "/Users/lostboy/toolbox/tools/symbolic_execution/miasm/test/arch/x86/unit/asm_test.py", line 38, in run
self.myjit.continue_run()
File "/usr/local/lib/python2.7/site-packages/miasm2/jitter/jitload.py", line 387, in continue_run
return self.run_iterator.next()
File "/usr/local/lib/python2.7/site-packages/miasm2/jitter/jitload.py", line 358, in runiter_once
self.pc = self.run_at(self.pc)
File "/usr/local/lib/python2.7/site-packages/miasm2/jitter/jitload.py", line 315, in run_at
set(self.breakpoints_handler.callbacks.keys())
File "/usr/local/lib/python2.7/site-packages/miasm2/jitter/jitcore.py", line 186, in run_at
cur_block = self.disasm_and_jit_block(offset, cpu.vmmngr)
File "/usr/local/lib/python2.7/site-packages/miasm2/jitter/jitcore.py", line 163, in disasm_and_jit_block
self.add_block(cur_block)
File "/usr/local/lib/python2.7/site-packages/miasm2/jitter/jitcore_gcc.py", line 105, in add_block
self.load_code(block.loc_key, fname_out)
File "/usr/local/lib/python2.7/site-packages/miasm2/jitter/jitcore_gcc.py", line 36, in load_code
lib = ctypes.cdll.LoadLibrary(fname_so)
File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 444, in LoadLibrary
return self._dlltype(name)
File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 366, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(/var/folders/0z/r4gr98p14wj653x6qbwb0cch0000gn/T/miasm_cache/f19b7fafb451b92bcff4d7ad6ad8ed47.so, 6): Library not loaded: build/lib.macosx-10.12-x86_64-2.7/miasm2/jitter/VmMngr.so
Referenced from: /var/folders/0z/r4gr98p14wj653x6qbwb0cch0000gn/T/miasm_cache/f19b7fafb451b92bcff4d7ad6ad8ed47.so
Reason: image not found
Although at the path I can see the *.so is there.
Another point, these *.so files produced now are Mach-O binaries, but still they are named as *.so, which I am not sure why.
I will look into it once I get time again.
Ok. As it seems it is a common mechanism, maybe we should look into other python project which include C as well.
I've ran into the same problem and I'm getting the same OS error. Inspecting the .so with otool -L print shared libraries used gives me the me this output:
(miasm) ➜ miasm git:(master) ✗ otool -L /var/folders/hn/6tlwjdvd44v_d06dbffmxmrm0000gn/T/miasm_cache/29f3ae4002d0acfda0621e565cb4a4b4.so
/var/folders/hn/6tlwjdvd44v_d06dbffmxmrm0000gn/T/miasm_cache/29f3ae4002d0acfda0621e565cb4a4b4.so:
/var/folders/hn/6tlwjdvd44v_d06dbffmxmrm0000gn/T/tmp1x41_z.so (compatibility version 0.0.0, current version 0.0.0)
build/lib.macosx-10.14-x86_64-2.7/miasm2/jitter/VmMngr.so (compatibility version 0.0.0, current version 0.0.0)
build/lib.macosx-10.14-x86_64-2.7/miasm2/jitter/arch/JitCore_x86.so (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
I can't say for sure what the problem is exactly, but my intuition tells me that the libraries whose path begin with "build..." are not loaded due to path being relative.
Hello all,
First happy new year. I've digged in the issue again. I applied the changes mentioned by @su-vikas and I exported the path to the dynamic library using the DYLD_LIBRARY_PATH environment variable:
xport DYLD_LIBRARY_PATH="/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/miasm/build/lib.macosx-10.14-x86_64-2.7/miasm2/jitter:/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/miasm/build/lib.macosx-10.14-x86_64-2.7/miasm2/jitter/arch"
It fixed the dynamic library issue. However, using an environment idea might not be the best idea. The fix raised the number of success to 466/471.
I had 3 errors with thedse_crackme.py file:
Trying to launch the binary without Miasm
/var/folders/9l/2p7mz59x49n6dfbw7q7vsdl00000gn/T/tmpcylBHz
Traceback (most recent call last):
File "dse_crackme.py", line 310, in <module>
stderr=subprocess.PIPE)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 8] Exec format error
Checking the code, I added a check for the Mac OS case and the number of success rose to 469/471.
The only two failure remaining are the following:
ERROR smt2.py
Traceback (most recent call last):
File "smt2.py", line 39, in <module>
s.add(e_z3 != smt2_z3)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/z3/z3.py", line 882, in __ne__
a, b = _coerce_exprs(self, other)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/z3/z3.py", line 1029, in _coerce_exprs
b = s.cast(b)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/z3/z3.py", line 1319, in cast
_z3_assert(is_expr(val), "True, False or Z3 Boolean expression expected. Received %s" % val)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/z3/z3.py", line 91, in _z3_assert
raise Z3Exception(msg)
z3.z3types.Z3Exception: True, False or Z3 Boolean expression expected. Received [If(And(Extract(7,
0,
Extract(15, 0, Extract(31, 0, a) + b)*c) <<
(LShR(d, 5 & 8 - 1) | d << 8 - (5 & 8 - 1)) !=
0,
True),
a + 100,
22) ==
If(And(e +
(1 ^
Extract(0, 0, Extract(7, 0, a)) ^
Extract(1, 1, Extract(7, 0, a)) ^
Extract(2, 2, Extract(7, 0, a)) ^
Extract(3, 3, Extract(7, 0, a)) ^
Extract(4, 4, Extract(7, 0, a)) ^
Extract(5, 5, Extract(7, 0, a)) ^
Extract(6, 6, Extract(7, 0, a)) ^
Extract(7, 7, Extract(7, 0, a))) !=
0,
True),
Concat(mem64[a*a + 7],
Concat(mem64[a*a + 6],
Concat(mem64[a*a + 5],
Concat(mem64[a*a + 4],
Concat(mem64[a*a + 3],
Concat(mem64[a*a + 2],
Concat(mem64[a*a + 1],
mem64[a*a]))))))),
Concat(mem64[a + 7],
Concat(mem64[a + 6],
Concat(mem64[a + 5],
Concat(mem64[a + 4],
Concat(mem64[a + 3],
Concat(mem64[a + 2],
Concat(mem64[a + 1],
mem64[a]))))))))]
ERROR z3_ir.py
Traceback (most recent call last):
File "z3_ir.py", line 102, in <module>
[(0xdeadbeef, 2), (0xdeadbeef + 3, 0)])
File "z3_ir.py", line 26, in check_interp
l = interp.as_list()
AttributeError: QuantifierRef instance has no attribute 'as_list'
As it happens, It seems related to z3 and not to the OS. Maybe I got the wrong version, I have z3 version 4.8.0.0:
$ (miasm_venv) > pip freeze
[...]
z3-solver==4.8.0.0
Hi, Hey, interesting: the jitter seems to run on Mac :smiley: So If I recap, the fix for running the jitter engine under Mac is:
- fix the
byte_orderstuff - fix the
setup.pywith the 'dynamiclib' stuff to have correct libs, and fix the path to make them load
I think this can make a great standalone patch. I agree with you @nirouviere for the z3 bug: I think this is definitely a different problem.
Wow, very nice! I also agree for the dse_crackme thing, this is completely normal (and is related to the way the final test is performed). We also excluded it on Windows. For DYLD_LIBRARY_PATH, did you try a setup install (inside a virtual env maybe)? It could possibly place the libraries in a path already in the env var.
Hello!
I installed miasm using pyenv with virtualenv and the libraries weren't in the path.
On Wed, Jan 2, 2019, 12:47 Camille Mougey [email protected] wrote:
Wow, very nice! I also agree for the dse_crackme thing, this is completely normal (and is related to the way the final test is performed). We also excluded it on Windows. For DYLD_LIBRARY_PATH, did you try a setup install (inside a virtual env maybe)? It could possibly place the libraries in a path already in the env var.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cea-sec/miasm/pull/875#issuecomment-450833489, or mute the thread https://github.com/notifications/unsubscribe-auth/AGhhguLO6nEc8IuLFUIqs_hXUBJbNmhsks5u_I4jgaJpZM4X8F9K .
-- Sincerely, Denis Nutiu
Hello team,
Sorry for being idle for so long. Here is the last fixes. I passed build options for the MacOS platform to change the LC_ID_DYLIB command within the library. Long story short, under MacOS, any binary linking against a shared library will look for this command to find the library. And this was set to lib-macosx-xxxxxx/
I also added an extra check for the dse_crackme test to not run it if we're under MacOS.
I'll see what travis says about this patch and fix potential issues. If you see anything that could be improved feel free to reach me.
The only tests failing now are the two following Z3 tests which are, like stated above, out of scope.
Result: 469/471 pass
ERROR smt2.py
Traceback (most recent call last):
File "smt2.py", line 39, in <module>
s.add(e_z3 != smt2_z3)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/z3/z3.py", line 882, in __ne__
a, b = _coerce_exprs(self, other)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/z3/z3.py", line 1029, in _coerce_exprs
b = s.cast(b)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/z3/z3.py", line 1319, in cast
_z3_assert(is_expr(val), "True, False or Z3 Boolean expression expected. Received %s" % val)
File "/Users/nicolasrouviere/Challenges/Challenges/root-me/Cracking/ELF-MIPS/miasm/lib/python2.7/site-packages/z3/z3.py", line 91, in _z3_assert
raise Z3Exception(msg)
z3.z3types.Z3Exception: True, False or Z3 Boolean expression expected. Received [If(And(Extract(7,
0,
Extract(15, 0, Extract(31, 0, a) + b)*c) <<
(LShR(d, 5 & 8 - 1) | d << 8 - (5 & 8 - 1)) !=
0,
True),
a + 100,
22) ==
If(And(e +
(1 ^
Extract(0, 0, Extract(7, 0, a)) ^
Extract(1, 1, Extract(7, 0, a)) ^
Extract(2, 2, Extract(7, 0, a)) ^
Extract(3, 3, Extract(7, 0, a)) ^
Extract(4, 4, Extract(7, 0, a)) ^
Extract(5, 5, Extract(7, 0, a)) ^
Extract(6, 6, Extract(7, 0, a)) ^
Extract(7, 7, Extract(7, 0, a))) !=
0,
True),
Concat(mem64[a*a + 7],
Concat(mem64[a*a + 6],
Concat(mem64[a*a + 5],
Concat(mem64[a*a + 4],
Concat(mem64[a*a + 3],
Concat(mem64[a*a + 2],
Concat(mem64[a*a + 1],
mem64[a*a]))))))),
Concat(mem64[a + 7],
Concat(mem64[a + 6],
Concat(mem64[a + 5],
Concat(mem64[a + 4],
Concat(mem64[a + 3],
Concat(mem64[a + 2],
Concat(mem64[a + 1],
mem64[a]))))))))]
ERROR z3_ir.py
Traceback (most recent call last):
File "z3_ir.py", line 102, in <module>
[(0xdeadbeef, 2), (0xdeadbeef + 3, 0)])
File "z3_ir.py", line 26, in check_interp
l = interp.as_list()
AttributeError: QuantifierRef instance has no attribute 'as_list'
Sounds great :smiley: Thank you @dnutiu and @nirouviere to improve it again for MacOS. It seems you are getting closer !
@nirouviere you can see the results of your build https://travis-ci.org/cea-sec/miasm/jobs/479689855
Hi everyone, by any chance do you have an update on this issue? it looks like I have a similar error with macOS 10.15.7.
[...]
OSError: dlopen(/var/folders/ts/2nwblqrj59172nzsb1bgmvlh0000gn/T/miasm_cache/b8838e62e19183ee7465864851bbaa87.cpython-38-darwin.so, 6): Library not loaded: /Users/john/venv/lib/python3.8/site-packages/miasm/jitter/VmMngr.so
Referenced from: /private/var/folders/ts/2nwblqrj59172nzsb1bgmvlh0000gn/T/miasm_cache/b8838e62e19183ee7465864851bbaa87.cpython-38-darwin.so
Reason: image not found
