Packaging

[dguido]

Let's find a way to avoid having every user build DeepState from scratch

[khorben]

As a first step I am trying to package it into pkgsrc (https://www.pkgsrc.org/). Although primarily targeting NetBSD, it also supports Linux and MacOS X (https://pkgsrc.joyent.com/install-on-osx/) and has a community generating packages for most platforms for every release (quarterly). I just packaged Manticore 0.1.7 there, even though it does not fully work for me there yet unfortunately.

I am trying to get Angr there as well. For DeepState, it may help packagers to have a first release out (0.0.0?) but it is not a must for pkgsrc.

[khorben]

Making progress: both Angr and DeepState are now in pkgsrc's WIP repository (http://pkgsrc.org/wip/). No binaries are officially built from this repository, but it is a great place for getting reviews and staging integration to the main repository.

[pgoodman]

Alright. I think you should prioritize packaging for homebrew and aptitude, and possibly pip if it lets headers be installed, as macOS and Ubuntu are more important for us than the various BSD distributions.

[khorben]

I managed to complete packaging and import both Manticore and Angr into pkgsrc, just in time for the coming 2018Q1 release (about 2-4 weeks from now). I believe they both work in the current state there. Unfortunately I may not be able to add DeepState for this release anymore. pkgsrc is not only for BSD but also works great on Linux (most if not all distributions) and macOS. Binary packages for both are provided at http://pkgsrc.joyent.com/ together with complete instructions. With this done, I can arrange access to a macOS system and package for homebrew next thing.

[agroce]

For MacOS is there a plan to deal with angr/manticore wanting linux binaries, or Dr. Fuzz wanting 32 bit exexcutables? Being able to install with homebrew, but then it not working for much at all might confuse folks, right? I guess the EVM stuff would work on MacOS with Manticore?

[khorben]

There is a bounty on adding support for macOS in Manticore: https://github.com/trailofbits/manticore/issues/110. Angr also has a related entry: https://github.com/angr/angr/issues/518.

[agroce]

Right, but for now it'd be nice if we had better error messages from trying the examples than:

~/deepstate/build/examples$ deepstate-angr  OneOf
WARNING | 2018-03-28 11:25:09,854 | angr.state_plugins.unicorn_engine | failed loading "angr_native.dylib", unicorn support disabled (dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/angr-7.7.9.21-py2.7.egg/angr/lib/angr_native.dylib, 6): image not found: dlopen(/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/angr-7.7.9.21-py2.7.egg/angr/lib/angr_native.dylib, 6): image not found)
WARNING | 2018-03-28 11:25:10,940 | cle.backends.macho | The Mach-O backend is not well-supported. Good luck!
CRITICAL | 2018-03-28 11:25:10,957 | deepstate.angr | Cannot find symbol `DeepState_Setup` in binary `OneOf`
~/deepstate/build/examples$ deepstate-manticore OneOf 
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/bin/deepstate-manticore", line 11, in <module>
    load_entry_point('deepstate==0.1', 'console_scripts', 'deepstate-manticore')()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources/__init__.py", line 572, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2754, in load_entry_point
    raise ImportError("Entry point %r not found" % ((group, name),))
ImportError: Entry point ('console_scripts', 'deepstate-manticore') not found
~/deepstate/build/examples$ deepstate OneOf 
CRITICAL:deepstate.mcore:Cannot create Manticore instance on binary OneOf: Binary OneOf not supported.

(by the way, it looks like deepstate-manticore is not working there, but I haven't confirmed if this is generally true or something wrong on my machine)

[dguido]

ManticoreEVM works fine on macOS, but anything else is a long ways off. The only platform that angr and manticore can reliably work from is Linux.

[agroce]

Dan, if you manage to get an ELF binary on MacOS, does manticore probably work then? Or are there still OS dependencies that are a problem?

[dguido]

As far as I understand, no, that will not work.

[pgoodman]

My guess is that you can take a Linux ELF binary and run it on Manticore on macOS. The catch is that you likely also need ELFs for all dependent .so files, and so to support a dynamic ELF on macOS, Manticore would need to provide an overlay file system abstraction. If you have a statically linked ELF, then it should work on macOS Manticore just fine.

[agroce]

Hmm, cross-compiling to a statically linked ELF in Rust is trivial, I know. Can clang just do this for the examples on MacOS?

[pgoodman]

Well you could always compile on Linux, then ldd the executable and any other dependencies (example), and copy the exectuable and the discovered dependencies onto the macOS system, put them all in a single directory, and then set up the LD_LIBRARY_PATH environment variable to this directory, and run Manticore that way.

[agroce]

True, but I was thinking if there's a way to get the examples working for people who grab a Mac package. Having them hop over to a VM isn't as useful for that as being able to do a cross-compile via clang somehow during the build on Mac.

[khorben]

More progress: DeepState is now packaged in two parts in pkgsrc, wip/deepstate and wip/py27-deepstate. It finally looks like it should be working from there. I will look at further packaging methods after some more testing.