Raspberry Pi support for mitmproxy.py (lxml)

[esvanoe]

trafficstars

Raspberry Pi install crashes while attempting install.sh (disregarding ntp issues discussed earlier)

Raspberry Pi version B and B+

Install will crash during installation returning the following:

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/tm /pip_build_root/lxml/src/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-armv6l-2.7/src/lxml/lxml.etree.o -w

In file included from src/lxml/lxml.etree.c:232:0:

/tmp/pip_build_root/lxml/src/lxml/includes/etree_defs.h:14:31: fatal error: libxml/xmlversion.h: No such file or directory

compilation terminated.

error: command 'gcc' failed with exit status 1

Cleaning up... Command /usr/bin/python -c "import setuptools, tokenize;file='/tmp/pip_build_root /lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), >file, 'exec'))" install --record /tmp/pip-ErT_gY-record/install-record.txt --single-version- externally-managed --compile failed with error code 1 in /tmp/pip_build_root/lxml Traceback (most recent call last): File "/usr/local/bin/pip", line 9, in load_entry_point('pip==1.5.6', 'console_scripts', 'pip')() File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/init.py", line 185, in main return command.main(cmd_args) File "/usr/local/lib/python2.7/dist-packages/pip-1.5.6-py2.7.egg/pip/basecommand.py", line 161, in >main text = '\n'.join(complete_log) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 62: ordinal not in range(128)

Same error noted on Ubuntu, but fixed via "apt-get install libffi-dev"

Attempting to manually install libffi-dev, libxml2-dev, libxslt-dev on the pi have not resolved this issue.

pip install lxml==3.4.2 - same result as above, issue is with lxml per se, not snoopy-ng.

Working around the issue, while losing some functionality.

go into install.sh and comment out

pip install mitimproxy.py

in directory /plugins/ rename mitimproxy.py > mitmproxy.bkp

[maximcherny]

Is this on a Kali image? Or Raspbian? Or something else? Which version?

I can confirm that this worked for me on both Kali 1.0.6 and Wheezy (from Feb 2014) images on a RPi Model B.

Having said that, when I did this circa July / August I ended up with some outdated version of mitmproxy that I had to subsequently upgrade directly from their git repo (mitmproxy 0.11 wasn't around back then).

In all cases I where I got the same error I was able to proceed after:

apt-get install libxml2-dev libxslt1-dev libffi-dev

[esvanoe]

This was on newest Raspbian, as wel as Kali 1.05-6-9

I have tried the things you mentioned, with the exception of maybe using an older version of mitmproxy, which I am going to mess with today.

Thanks for the idea.

[esvanoe]

So, minor update.

"pip install --upgrade six"

allows for mitmproxy install. unfortunately, snoopy then hangs on the variable "cacert" in mitmproxy.py

If anyone has run this to ground yet, would be glad to hear about it. more to follow.

[7109node]

Any progress with this fix?

[glennzw]

I don't have a RPi at hand to test this, but will try get someone on it. I gave training recently and the students got it running. If anyone figures out a fix please post here.

[stevenhorner]

@esvanoe Did you get any further with the "cacert" variable. I am running Kali 1.1.0 on a regular laptop and I get the same error.

The cacert variable contains a path to the mitmproxy CA Certificate, in my case the location it points to doesn't exist "~/.mitmproxy/mitmproxy-ca.pem". I tried creating a certificate manually following the instructions on the mitmproxy website and changing this variable to point to it but I still get the same error.

Not sure where to go from here I was only testing to see what works and what doesn't in the last update. For my uses mitm isn't really needed but doing some testing of snoopy-ng.

[esvanoe]

I was able to create a cacert, and/or copy one onto the RPI and that seemed to satisfy the variable. I never got much into testing it though as another project took me off topic with this one and I never quite circled back around.

If I get back into this though I will post any updates as it goes.

[NoobieDog]

Run mitmdump on its own and it will generate a pem file

Stu

On Tuesday, March 10, 2015, Steven Horner [email protected] wrote:

@esvanoe https://github.com/esvanoe Did you get any further with the "cacert" variable. I am running Kali 1.1.0 on a regular laptop and I get the same error.

The cacert variable contains a path to the mitmproxy CA Certificate, in my case the location it points to doesn't exist "~/.mitmproxy/mitmproxy-ca.pem". I tried creating a certificate manually following the instructions on the mitmproxy website and changing this variable to point to it but I still get the same error.

Not sure where to go from here I was only testing to see what works and what doesn't in the last update. For my uses mitm isn't really needed but doing some testing of snoopy-ng.

— Reply to this email directly or view it on GitHub https://github.com/sensepost/snoopy-ng/issues/24#issuecomment-78041278.

[stevenhorner]

@NoobieDog When I run mitmdump it just seams to hang if I give it no options?

[maximcherny]

I had mitmproxy 0.10 working at some point in one of my branches as part of some controlled setting experiments. The reality is in the context of smartphones the majority of traffic will be generated by the apps, not necessarily the browser and most of the most popular apps will have SSL pinning in place.

I am not sure what sort of data you'd be expecting to mitm, but without at least getting the mitm certs on the user device (i.e. via a captive portal or iOS config profile) you will end up seeing a ton of advertising cookies and very few actual goodies.

[ahgray]

I can confirm that running fully updated raspian as of 20150902 on a pi b2 that it starts to work after running apt-get install libxml2-dev libxslt1-dev libffi-dev as per @maximcherny suggestion.