Embedded Filecoin: Enable cross-compiling

[eigendude]

Description of the Change

Because what would be cooler than filecoin for embedded systems?

Porting something as complex as filecoin and all dependencies to embedded is not a small task, but I was able to make progress and even runtime test some features. This PR outlines the work I've done across various repos and the remaining steps to achieve embedded filecoin.

Structure of the PR

The PR is organized as follows:

1. Unmerged dependent PRs (in this case there's a simple git conflict)
2. Fixes needed for cross-compiling
3. Changes needed for cross-compiling
4. Additions needed for cross-compiling
5. Changes needed for linking filecoin from another process

Next, links to commits needed for cross-compiling libp2p. Depending on feedback, I can break these out and PR to the libp2p repo.

Finally, links to commits in the repos of two different embedded build systems, one of which I runtime tested.

Dependent PRs

Commit:

• [x] [PR 110] CMake: Remove duplicate dependency

The PR diff is adjacent to a later change, so it's more convenient to rebase the PR out when it's merged to avoid the small git conflict.

Fixes for cross-compiling

Commits:

• [x] CMake: Fix error including generated protobuf headers
• [x] CMake: Fix error locating protobuf without Hunter
• [x] CMake: Fix BLS signature library dependence on $HOME
• [x] Fix tinycbor include directory

These commits come first, because they address what could be considered defects. Inclusion would be valuable even if the rest of the PR is omitted. Further description is in the commit message.

Changes for cross-compiling

Commits:

• [x] CMake: Expose CMake options to includes
• [x] CMake: Exclude test-related depends when building tests
• [x] CMake: Allow excluding git submodule depends

These changes enable more control over the build process when embedded in another build system.

CMake is controlled by user variables; the first one we need is -DHUNTER_ENABLED=OFF so that we can have control over dependencies.

Another useful variable is -DTESTING=OFF, because embedding test infra drastically increases the up-front effort. The first two commits above allow this variable to exclude test infra from the dependency discovery in addition to the build.

Finally, I introduced a new variable: -DBUILD_INTERNAL_DEPS=OFF. This causes the in-tree dependencies to be skipped. I needed this for a build system that doesn't support git submodules. Instead, the dependencies are provided by the build sytsem.

Additions for cross-compiling

Commits:

• [ ] CMake: Add Findcppcodec.cmake for building without Hunter
• [ ] CMake: Add Findleveldb.cmake module for building without Hunter
• [ ] [WIP] CMake: Fix cross compiling (needs FindMicrosoft.GSL.cmake?)

When building without Hunter, CMake needs some help discovering dependencies that aren't known to CMake. I took the standard approach of including CMake modules for these missing dependencies.

Findleveldb.cmake is definitely needed for the nested CMakeLists.txt file that links to leveldb. Not sure about Findcppcodec.cmake or FindMicrosoft.GSL.cmake...

Changes for linking from another process

Commits:

• [ ] CMake: Prefix libraries to avoid name clashes
• [ ] CMake: Add install step for libraries
• [ ] CMake: Add install step for headers (WIP)

At this point, make succeeds. The next step of a build system is to run make install so that the other process can link to the necessary libraries and headers. Here, we get the following error:

make: *** No rule to make target 'install'.  Stop.

The second commit above adds this missing step. However, unlike libp2p where the library names are prefixed, this results in an explosion of generically-named libraries polluting the system root. To solve this, I preceded the commit with a functional change that prefixes libraries and test cases to mirror libp2p's approach.

Finally, headers need to be installed too. The third commit takes an approach similar to typical cmake projects, where headers are in the source tree and they are referenced from the CMakeLists.txt file. This was less work for me to get to the runtime testing, as I didn't have to touch code, but I don't think it's the right approach here. Instead, we should mirror libp2p and move headers to a top-level include/ folder. All sources will need to be adjusted to include files from inside a filecoin/ top level folder, similar to how libp2p headers are included, e.g.:

#include <libp2p/protocol/echo/echo.hpp>

This is a requirement to avoid clashes when mixing headers from a generic directory like common/ with external code.

Fixes for libp2p

I made the following fixes for libp2p. Not all are probably necessary, so extra testing is needed. I'll break these out, do the extra work and upstream to libp2p depending on feedback.

Commits:

• [x] CMake: Fix error locating Boost when building without Hunter
• [x] CMake: Fix error locating protobuf without Hunter
• [x] CMake: Fix error locating Boost DI without Hunter
• [x] [WIP] CMake: Fix error locating protoc without Hunter
• [x] CMake: Exclude test dependencies when building tests
• [x] CMake: Allow building without examples
• [x] Fix build error due to missing include
• [x] Fix build error due to mismatched types
• [x] Fix build errors due to missing include

I also got link errors about spdlog so I needed to comment a bunch of logging, not sure why but I'm not including the commit here.

Additions for libp2p

I also authored an addition to libp2p:

• [ ] Add blake2b-256 multicodec enum value

This is because filecoin uses blake2b-256 for CIDs, which shows up as "unknown" in libp2p's CID stringification.

Before:

identity - cidv1 - dag-cbor - unknown-256-d36a2619a672494604e11bb447cbcf5231e9f2ba25c2169177edc941bd50ad6c

After:

identity - cidv1 - dag-cbor - blake2b-256-256-d36a2619a672494604e11bb447cbcf5231e9f2ba25c2169177edc941bd50ad6c

Additions for OpenEmbedded (meta-cryptocurrency layer)

With these changes in place, I'm able to compile and link filecoin and dependencies using two build systems: OpenEmbedded, and Kodi's depends system. I almost succeeded in getting OpenEmbedded to cross-compile the Rust code, but ended up stubbing BLS signatures and proofs for Kodi's build system so that I could get to some runtime testing.

Here's my work on these two build systems. Commits are structured so that filecoin inclusion can be upstreamed.

Commits:

• libp2p: Import C++ implementation of libp2p
• fuhon: Import C++ implementation of Filecoin

This first commits enables libp2p support in the entire OpenEmbedded ecosystem. The second commit joins it with filecoin, but cargo fails a long way into the build, so there's a little work left to do.

Additions for Kodi

Commits for C++17 support:

• [docs] Update Linux readmes for C++17 requirement
• [depends] Check for C++17 support
• Change to C++17 for core

Kodi currently uses C++14. These 3 commits can be PR'd to enable C++17 support, needed for libp2p and filecoin. Merging is probably a long way off, as Kodi supports a long tail of devices with embedded build systems that often use old compilers.

Commits for the depends build system:

• tools/depends: Update OpenSSL to v1.1.1
• tools/depends: Add libp2p and dependencies
• tools/depends: Add Filecoin and dependencies

With the commits in this PR, in addition to stubbing proofs and BLS signatures, I was able to cross-compile filecoin for x86_64 Linux and macOS for runtime testing. Next is to test compilation for Raspberry Pi and Android NDK.

Commits for runtime testing:

• testing filecoin: ipfs data store
• testing libp2p: echo, kademlia and gossip

I didn't spend a lot of time on kademlia, but the ipfs data store, echo and gossip test code produces promising output. Tested on Linux x86_64 and macOS.

Benefits

IoT is the obvious goal, but Kodi opens up another little-known area of embedded: OTT (over-the-top), stretching across both set-top boxes and almost all mobile

Possible Drawbacks

When I do PRs like this people sometimes freak out. Sorry if it causes any negative feelings, but I figured what the hell, might as well share the work and see what happens.

[eigendude]

I see two 🚀! I have experience with big PRs like this and think I can get it merged. I'll break out commits into separate PRs roughly as they're grouped above to facilitate review. It's gonna take a lot of work but I think I'll be able to find the time over the next few weeks to see embedded filecoin happen.

[igor-egorov]

Hi @eigendude
First of all - thank you for your efforts! We much appreciate your contribution.

The most debatable change for us is the headers' disclosure. Initially, Fuhon was not supposed to be a library.

Here is the question (1) - what are the benefits for you to have a library rather than an executable?

If we decide to make it as a library - what should be an interface then (2)? The spec itself is roughly defined, and it seems hard to design a reliable interface now.

Anyway, it seems that most of your changes are going to be merged 🎉 The thing that is not going to be merged - filecoin_add_headers. The approach with a common /include directory for headers is more preferable. The project comes more structured and less error-prone - a developer could forget to expose a particular header in CMakeLists.txt.

Feel free to join our newly created community chat for faster communications https://t.me/fuhon_filecoin :)

[eigendude]

I've upstreamed the libp2p fixes: https://github.com/soramitsu/libp2p/pull/59

Build fixes broken out into new PR: https://github.com/filecoin-project/cpp-filecoin/pull/128

[eigendude]

Two years later, we've merged https://github.com/filecoin-project/cpp-filecoin/pull/128! And I split out the rest of the CMake changes to https://github.com/filecoin-project/cpp-filecoin/pull/601.

My last question is about installing libraries in addition to the built executables (fuhon-node and fuhon-miner). When embedding Fuhon into a single process (needed for e.g. Android TV), the executables aren't needed, but the compiled libraries are.

I would propose we add two CMake options:

option(INSTALL_EXECUTABLES "Enable installing Fuhon executables" ON)
option(INSTALL_LIBRARIES "Enable installing Fuhon's libraries and headers" OFF)

The first option is standard for any kind of cross-compiling. You can't run executables compiled for another system. But there's no harm in installing files that aren't shipped.

If one or both of these options are desired, I'll open a PR with the changes and close this one.