Make Pwnagotchi image buildable

[llamasoft]

Fixes #1115 and #1073

Description

This PR fixes the many small things (and some big things) required to get the pwnagotchi image buildable again.

⚠️ This PR is best reviewed one commit at a time.

I tried to keep each commit's changes as small as possible while also including a commit message that fully explains the reasoning or requirement for the changes. The full diff may appear large, but the individual commits tell a story that makes the final result easier to understand.

Also, there are a few intermediate changes that are made redundant by later commits. I did this primarily to make reviewing the commits easier but also so you don't have to accept the whole PR if you don't want to. If you disagree with any of the commits or feel that the changes go too far, you can decide which commits to keep and discard everything after it.

Summary of changes:

(See individual commit messages for a more detailed explanation.)

• 93b6104 to 4748cb6: The absolute minimum amount of changes to get the image to build (similar to #1108).
  The image boots but is unstable. The build process is still likely to break at any time.
• 6634015 to f910f04: Fixes image instability caused by the previous set of commits.
  • 6634015: (Optional) Ensures that the correct CPU type is emulated during the build process.
    This ensures that all packages from apt and pip support armv6 and removes the need to handle opencv and tensorflow separately.
  • e88301e and f910f04: Fixes issues related to the updated base image.
• de8559a to 3d28df6: (Optional) Fixes pwnagotchi's python packaging issues.
  • 3e0fdc4 to f17f784: A sustainable (and hopefully permanent) fix to requirements.txt version pinning using pip-compile.
    This gives pwnagotchi a requirements.txt file that pins everything to the exact versions without having to worry about indirect dependencies causing build issues at a later date.
    The inclusion of a requirements.in file ensures that a working requirements.txt file can be built even if the base image or python version is updated.
  • 3d28df6: Installs pwnagotchi's python and system files from the local branch.
    Previously, the build process would install system files from the local builder/data directory but overwrite them when installing pwnagotchi's python files from the remote master branch.
    This made development for this PR a constant struggle. Each build attempt required me to push changes to my forked repo, update the URLs in the build scripts, build the image, then revert the changes.
    This should also help if you ever wanted to implement automated image builds 😉😉.

Motivation and Context

• [x] I have raised an issue to propose this change (#1115)

I wanted to build my own pwnagotchi but could only obtain a Waveshare v3 display. I saw that support for the Waveshare v3 was added to the repo but no image had been released since then. I tried to build the image myself but quickly ran into issues. While using the available image and copying the updated files would be the easy approach, I felt that getting pwnagotchi images buildable again would be the best for everyone.

How Has This Been Tested?

I've built the image multiple times and use it on my personal RPi0W pwnagotchi. One commenter merged the commits into their test branch and said it was working for them as well.

I don't have a way to confirm that this adds RPi0W2 support, some of the changes in this PR are similar to those in #1108.

Types of changes

• [X] Bug fix (non-breaking change which fixes an issue)
• [X] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to change)

This PR is focuses primarily on bug fixes.
There are a few incidental new features (like being able to install pwnagotchi from an sdist) but they are the result of fixes to the build process and should not affect the functionality of the pwnagotchi application itself.

Checklist:

• [X] My code follows the code style of this project.
• [X] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.
• [X] I've read the CONTRIBUTION guide
• [X] I have signed-off my commits with git commit -s

I was not able to find the documentation source in order to update it. 🤐

The only documentation update is that the image build prerequisites no longer require go or kpartx:

• go is no longer required because Packer and the ARM image plugin are downloaded as binary releases.
• kpartx is no longer required because Packer now uses losetup as of v0.2.2.

[llamasoft]

Bah, I just spotted an issue with this build. Going to be pushing an update in a bit.

[llamasoft]

There we go, all fixed up and ready for review!

Changes since the last push:

• Downgrade Tensorflow to <1.14 due to breaking API changes in preparation of Tensorflow 2.X
• Add libgtk-3-0 apt package for opencv.
• Fix pip install command so that the pwnagotchi executable ends up in the correct place.

[llamasoft]

Somewhat related, but my PR for packer-plugin-arm-image was merged and included in their latest release. 🎉

What that means for this PR: because the Makefile downloads the correct build of packer and packer-plugin-arm-image for the current CPU architecture, it's now possible to build a Pwnagotchi image from a Raspberry Pi! (It takes around 90 minutes on my RasPi 4, but it works!)

[acracer2008]

I just wanted to say thanks for this. I was able to get a buildable image after making the changes. After that the only issues I had where from python not having the correct libraries installed and it being my first time building an image on Linux

[user171]

Thanks for keeping this project alive. It looks abandoned to me.
I tried to build your fork on Arch Linux with all dependencies installed but got this error:

==> pwnagotchi: E: Repository 'http://raspbian.raspberrypi.org/raspbian buster InRelease' changed its 'Suite' value from 'stable' to 'oldstable'
==> pwnagotchi: E: Repository 'http://archive.raspberrypi.org/debian buster InRelease' changed its 'Suite' value from 'testing' to 'oldstable'
==> pwnagotchi: deregistering packer-plugin-arm-image-3521742004 with binfmt_misc
==> pwnagotchi: fuser -k /tmp/armimg-457800909
Build 'pwnagotchi' errored after 37 seconds 827 milliseconds: Script exited with non-zero exit status: 100. Allowed exit codes are: [0]

Did I miss something?

EDIT:

yes, I think I missed the switch to the branch build-fix.

RE-EDIT: Yeah! I could successfully build your image and it works great! Thanks a lot for your work. You should be the maintainer.

[D337z]

One of the biggest issues that I'm seeing with your code is that it isn't future proof. You're specifying a version of the programs to download instead of having it always try to build with the most recent versions of the programs. This means that anyone building this three years from now will be using outdated software to do so. Personally, I would suggest always downloading the latest version when possible (which doesn't include in apt or curl since they're outdated as well).

[user171]

@D337z you say :

One of the biggest issues that I'm seeing with your code is that it isn't future proof.

No code is future proof. See https://en.wikipedia.org/wiki/Software_rot.

You're specifying a version of the programs to download instead of having it always try to build with the most recent versions of the programs. This means that anyone building this three years from now will be using outdated software to do so.

Pinning versions is actually good practice. Not updating code for three years would be the problem in your description. See https://cloud.google.com/blog/topics/developers-practitioners/best-practices-dependency-management.

[llamasoft]

You're specifying a version of the programs to download instead of having it always try to build with the most recent versions of the programs.

This was a deliberate decision, although I tried to keep it as flexible as possible.

Unfortunately, many of the Python packages that pwnagotchi depends on either A) don't follow semantic versioning or B) are still in their 0.X release cycle. The best compromise I was able to come up with is using PEP-440 "compatible release" syntax and using pip-compile to get point-in-time values that are guaranteed to work. I documented a portion of this in the commit messages: 1, 2.

As for system packages, most of them are being installed at the latest version. The system image itself can't default to the latest version because there's no guarantee that its supported by Kali-Pi. Even a seemingly benign system image update within the same major release ended up requiring some manual fixing.

I'm not sure about the ld.so.preload method, but it seems to work.

Yeah... it's a bit hackish, but unfortunately it (or something like it) is required because of how packer works. Once packer mounts the image and chroots into it, the ld.so.prefetch is going to be seen as if it's the host system's ld.so.preload. This can result it in loading libraries of the mounted image's CPU architecture before qemu has a chance to kick in.

The code shouldn't be putting packer into the /usr/bin of the system and should, instead, reference it directly after compiled so that any newer binaries won't be replaced.

It's not doing that any more. It now downloads a static build of packer instead of building it from source (so it no longer needs golang installed on the host) and stores it in /tmp/pwnagotchi/packer. The packer install command downloads the arm image plugin and stores it under the root user's $HOME/.packer/ directory. While this permanent modification is unfortunate, the plugin binaries are versioned and won't conflict with any other version of the plugin that the user has installed.
If at a later date the packer config is updated to HCL, plugin installation and management can be handled directly via packer configs.

As an added bonus, attempting to compile the setup.py script doesn't result it in automatically clobbering the build system's files.

Like I said, it's not perfect but I tried to A) keep the changes as minimal as possible and B) not change the overall architecture of the project.

[D337z]

I am wondering, how are the files being put into the image via packer? All I'm seeing in the json file is that they're being tossing into the src directory instead of the usr/bin of the image. I ran into that small snag. Also, the {{user pwnagotchi_version}} line does not work properly.

[llamasoft]

Most of the files are bundled into the python package. Packer copies the archive into the src directory, then the pip install places them in the correct locations inside the image (via setup.py). Putting the system files in the python archive is required because of how pwnagotchi's update mechanism works.

On Sat, Apr 15, 2023, 2:39 PM D337z @.***> wrote:

I am wondering, how are the files being put into the image via packer? All I'm seeing in the json file is that they're being tossing into the src directory instead of the usr/bin of the image. I ran into that small snag.

— Reply to this email directly, view it on GitHub https://github.com/evilsocket/pwnagotchi/pull/1120#issuecomment-1509924308, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDK7ATPC3FQS4G3EMSP7SDXBLTO5ANCNFSM6AAAAAAQ4DJAQI . You are receiving this because you authored the thread.Message ID: @.***>

[pepperbob]

So after numpy problems with the latest official release (see all the numpy issues, AI not starting) I came by this PR, ran make, flashed the image on a RPI0W and it still works like a charm! Kudos @llamasoft