rust bindings fail to build

[brayniac]

Describe the bug Rust bindings fail to build.

To Reproduce Steps to reproduce the behavior:

1. Change into cachelib/rust within this repo (crate root)
2. Run cargo build
3. See error

Expected behavior Expected the crate to build

Output

CacheLib/cachelib/rust$ cargo build --release
    Updating crates.io index
  Downloaded cxx-build v1.0.57
  Downloaded scratch v1.0.0
  Downloaded cxxbridge-macro v1.0.57
  Downloaded anyhow v1.0.51
  Downloaded cc v1.0.72
  Downloaded thiserror-impl v1.0.30
  Downloaded thiserror v1.0.30
  Downloaded once_cell v1.8.0
  Downloaded bytes v1.1.0
  Downloaded serde v1.0.130
  Downloaded cxxbridge-flags v1.0.57
  Downloaded link-cplusplus v1.0.6
  Downloaded proc-macro2 v1.0.33
  Downloaded syn v1.0.82
  Downloaded libc v0.2.109
  Downloaded cxx v1.0.57
  Downloaded codespan-reporting v0.11.1
  Downloaded abomonation v0.7.3
  Downloaded 18 crates (1.5 MB) in 0.70s
   Compiling proc-macro2 v1.0.33
   Compiling unicode-xid v0.2.2
   Compiling cc v1.0.72
   Compiling syn v1.0.82
   Compiling scratch v1.0.0
   Compiling termcolor v1.1.2
   Compiling cxxbridge-flags v1.0.57
   Compiling unicode-width v0.1.9
   Compiling lazy_static v1.4.0
   Compiling serde v1.0.130
   Compiling libc v0.2.109
   Compiling anyhow v1.0.51
   Compiling abomonation v0.7.3
   Compiling once_cell v1.8.0
   Compiling codespan-reporting v0.11.1
   Compiling link-cplusplus v1.0.6
   Compiling cxx v1.0.57
   Compiling quote v1.0.10
   Compiling bytes v1.1.0
   Compiling cxx-build v1.0.57
   Compiling thiserror-impl v1.0.30
   Compiling cxxbridge-macro v1.0.57
   Compiling thiserror v1.0.30
   Compiling cachelib v0.1.0 (/home/brian/CacheLib/cachelib/rust)
The following warnings were emitted during compilation:

warning: /home/brian/CacheLib/cachelib/rust/target/release/build/cachelib-bbeb85d31435509b/out/cxxbridge/sources/cachelib/src/lib.rs.cc:1:10: fatal error: cachelib/rust/src/cachelib.h: No such file or directory
warning:  #include "cachelib/rust/src/cachelib.h"
warning:           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
warning: compilation terminated.

error: failed to run custom build command for `cachelib v0.1.0 (/home/brian/CacheLib/cachelib/rust)`

Desktop (please complete the following information):

• OS: Debian 9 / 10 / 11
• Rust: 1.56.1 (current stable)

Additional context Rust crate is currently not tested in CI. Adding to CI would help make sure it builds and continues to build as changes are made.

[leozzx]

Hi @brayniac, I'm not familiar with rust, but it is complaining cachelib/rust/src/cachelib.h not found. The file does exist, so can you try to build in the root directory (CacheLib/) instead of rust directory?

[brayniac]

@leozzx - unfortunately that won't work because the repo root is not a crate or workspace root.

CacheLib$ cargo build --release
error: could not find `Cargo.toml` in `/home/brian/CacheLib` or any parent directory

[farnz]

Can you try and teach the cc::Build returned by the build.rs that /home/brian/CacheLib is a root for includes? I think this would involve constructing a Path using the result of std::env::current_dir and passing it to the include method on the Build object.

[therealgymmy]

cc @agordon for advice.

[brayniac]

@therealgymmy @agordon @farnz - any updates on this?

[journaux]

"Can you try and teach the [cc::Build](https://docs.rs/cc/1.0.72/cc/struct.Build.html) returned by the build.rs that /home/brian/CacheLib is a root for includes? I think this would involve constructing a Path using the result of [std::env::current_dir](https://doc.rust-lang.org/std/env/fn.current_dir.html) and passing it to the include method on the Build object."

@farnz resolves the first issue, then in debian/ubuntu missing dependency headers forms new errors

  cargo:warning=/CacheLib/cachelib/rust/../../cachelib/rust/src/cachelib.h:24:10: fatal error: cachelib/facebook/admin/CacheAdmin.h: No such file or directory
  cargo:warning=   24 | #include "cachelib/facebook/admin/CacheAdmin.h"
  cargo:warning=      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  cargo:warning=compilation terminated.

when removing the admin (a private facebook header file), the build fails on lack of rust bindings for folly string type.

i removed the references to std::chrono::timestamp & folly::String types, replacing them with primitives i64 & std::string respectively in order to achieve a successful compilation

by example

all in all => need to have the folly rust bindings in order to compile cachelib - otherwise, the community can not implement Cxx Extern Type for ffi.

more than happy to help if a contributor wants to hop on a video call so that we can consume cachelib in rust/c.

thank u for a great library. the only high quality, multi threaded caching library for ssd we've found. twitter pelikan single threaded & does not support 10TB ssd drives.

[therealgymmy]

@vitruvvius: thanks for figuring out a fix! Could you send out a PR?

[journaux]

i'd rather see folly & std::chrono::timestamp resolved w/ FFI bindings than the strategy provided

[journaux]

someone should close this. no urgency. write your own c bindings.

[journaux]

what are we doing about this, if at all? happy to contribute if the facebook team ok w/ transitioning off folly/chrono dependency on the ffi, albeit i doubt that they want to fork + maintain an oss mirrior.

[journaux]

one thing that might be interesting is to move away from hand-writing the FFI & integrating autocxx from google https://github.com/google/autocxx

[nickamorim]

Even with the change proposed by @journaux, the Rust bindings still fail to build.

  cargo:warning=In file included from /Users/nickamorim/src/github.com/Shopify/CacheLib/cachelib/rust/src/cachelib.h:22:0,

  cargo:warning=                 from /Users/nickamorim/src/github.com/Shopify/CacheLib/cachelib/rust/target/debug/build/cachelib-7bbe50bf6a183fab/out/cxxbridge/sources/cachelib/src/lib.rs.cc:1:

  cargo:warning=/Users/nickamorim/src/github.com/Shopify/CacheLib/cachelib/allocator/CacheAllocator.h:19:10: fatal error: folly/CPortability.h: No such file or directory

  cargo:warning= #include <folly/CPortability.h>

  cargo:warning=          ^~~~~~~~~~~~~~~~~~~~~~

  cargo:warning=compilation terminated.

  exit status: 1

  --- stderr

  CXX include path:
    /Users/nickamorim/src/github.com/Shopify/CacheLib/cachelib/rust/target/debug/build/cachelib-7bbe50bf6a183fab/out/cxxbridge/include
    /Users/nickamorim/src/github.com/Shopify/CacheLib/cachelib/rust/target/debug/build/cachelib-7bbe50bf6a183fab/out/cxxbridge/crate


  error occurred: Command "c++" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-gdwarf-4" "-fno-omit-frame-pointer" "-m64" "-I" "/Users/nickamorim/src/github.com/Shopify/CacheLib/cachelib/rust/target/debug/build/cachelib-7bbe50bf6a183fab/out/cxxbridge/include" "-I" "/Users/nickamorim/src/github.com/Shopify/CacheLib/cachelib/rust/target/debug/build/cachelib-7bbe50bf6a183fab/out/cxxbridge/crate" "-I" "/Users/nickamorim/src/github.com/Shopify/CacheLib" "-Wall" "-Wextra" "-std=c++11" "-o" "/Users/nickamorim/src/github.com/Shopify/CacheLib/cachelib/rust/target/debug/build/cachelib-7bbe50bf6a183fab/out/2198180abe54d98b-lib.rs.o" "-c" "/Users/nickamorim/src/github.com/Shopify/CacheLib/cachelib/rust/target/debug/build/cachelib-7bbe50bf6a183fab/out/cxxbridge/sources/cachelib/src/lib.rs.cc" with args "c++" did not execute successfully (status code exit status: 1).

[journaux]

@nickamorim that seems a folly version / include path bug, nonetheless a lot of adjustments needed circa 8 months ago to compile properly (watch out for stack smashing in rust :: c++ ffi). anecdotally, somebody informed me facebook never released their internal cache eviction policy for cachelib, implying the public implementation does not match the private deployment.