Add submodule for C bindings, static compilation

[m-fila]

Adding C bindings and ahead of time compilation to shared library with PackageCompiler.jl or juliac

C components have jetreconstruction_(or JETRECONSTRUCTION_) prefix

For now the C API includes only:

• jet_reconstruct
• exclusive_jets, inclusive_jets
• enums and structs used in their signatures. When needed new structs with C-like layout are introduced (for instance replacing Vector{T} field with Ptr{T} and Csize_t fields for C-facing structs)

The C header file has to be written manually and keep in sync with Julia code. Technically the whole compile could be a separate julia package possibly even living in a separate repo, but I think it'd be harder to keep in sync this way

---

To do:

• [x] write to an output pointer from inclusive_jets and exclusive_jets
• [x] add error handling
• [x] add missing docstrings
• [x] add precompilation statements
• [x] cache all combinations of strategy and algorithm in precompilation
• [x] have actual test
• [x] install JetReconstructionConfig.cmake
• [x] compilation with juliac
• [x] add GenKt
• [x] add recombination schemas
• [ ] support Lorentz vectors
• [ ] support EEJet

[codecov[bot]]

Codecov Report

:x: Patch coverage is 65.55556% with 31 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 79.14%. Comparing base (d3c97f3) to head (7db31ad).

Files with missing lines	Patch %	Lines
src/C_JetReconstruction/C_JetReconstruction.jl	65.55%	31 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #88      +/-   ##
==========================================
- Coverage   80.00%   79.14%   -0.86%     
==========================================
  Files          21       22       +1     
  Lines        1315     1405      +90     
==========================================
+ Hits         1052     1112      +60     
- Misses        263      293      +30     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[m-fila]

The CI for Julia 1.9 fails as documenting@ccallable functions was fixed only in 1.11 :cry:

I couldn't think of a good way of error handling. What I propose here is to:

• use status codes from enum with:
  • value for success,
  • values for each standard Julia exception,
  • single value for the remaining cases.
• print the exception and backtrace to stderr so it's not lost

I also couldn't find a reasonable way to translate Julia exceptions to C++ exceptions without going through C-bottleneck

[Moelf]

We don't need to support 1.9 I guess, but I hope we can backport https://github.com/JuliaLang/julia/pull/51587 to 1.10.

[graeme-a-stewart]

Yeah, I would agree we can drop support for 1.9 now, but the LTS release (1.10) we should try to support

[graeme-a-stewart]

Just report this crashes on OS X with

[ Info: PackageCompiler: Executing /Users/graemes/.julia/dev/JetReconstruction/compile/precompile_execution.jl => /var/folders/kq/jgwv_wcj3ll2n6qwj06pp6jr0000gn/T/jl_packagecompiler_0hx01x/jl_dsyOUt
julia(84816,0x1f264cf40) malloc: *** error for object 0x134add910: pointer being freed was not allocated
julia(84816,0x1f264cf40) malloc: *** set a breakpoint in malloc_error_break to debug

[84816] signal 6: Abort trap: 6
in expression starting at /Users/graemes/.julia/dev/JetReconstruction/compile/precompile_execution.jl:33

I'm investigating!

[m-fila]

Thanks for checking. I think that was because the results_ptrpointee was managed by Julia and got garbage collected. Should be fixed now

[graeme-a-stewart]

Yes, now it works - thanks for the quick fix @m-fila!

[m-fila]

Added compiling with juliac from Julia nightlies. Unfortunately LoopVectorization isn't compatible with juliac and would have to be removed altogether from the project. Also any kind of trimming produces a lot of errors and aborts the compilation.

With the nightlies I get a lot of warnings like this. They might be false positives though:

WARNING: Detected access to binding `JetReconstruction.##meta#64` in a world prior to its definition world.
  Julia 1.12 has introduced more strict world age semantics for global bindings.
  !!! This code may malfunction under Revise.
  !!! This code will error in future versions of Julia.
Hint: Add an appropriate `invokelatest` around the access to this binding.

Taking all into account some results

	PackageCompiler	juliac
Compilation time (s)	~730	~90
Lib size (M)	283	247
Deps size (M)	279	0

with juliac the explicit initialization and shutdown of julia runtime is gone. The pre-compilation is also gone (I didn't manage to reduce completely the overheads of first execution with PackageCompiler either with precompilation-statements, precompilations-execution or precompilation-execution on the test suite). Also now the compiled object doesn't bring any exclusive dependencies that should be kept around it.

[Moelf]

sounds good!

Added compiling with juliac from Julia nightlies. Unfortunately LoopVectorization isn't compatible with juliac and would have to be removed altogether from the project.

for this, I guess time to retry https://github.com/JuliaHEP/JetReconstruction.jl/pull/84

[m-fila]

for this, I guess time to retry #84

Thankfully SIMD and its deps seem to be working fine with juliac

[Moelf]

I guess makes sense, it's just injecting LLVM IRs

[m-fila]

PackageCompiler with LoopVectorization compiles and runs but somehow is much slower ( more than x5) even than than a simple loop with no extra optimization

[Moelf]

https://discourse.julialang.org/t/guide-using-sum-types-to-define-dynamic-c-apis-for-julia-1-12-trim/132111

seems applicable

[m-fila]

I think we don't have anything like this here. From what I understood their function was type-unstable from the begging as returned type was decided by enum which isn't very nice even without juliac involved.

We use enums to select which function (strategy, algorithm) but it's stable as in the end we get the same type, if a function can select return type (jet selection) we do so by passing Type explicitly no enums involved. In the C-interface here we only have single jet type PseudoJet.

[Moelf]

interesting, I thought for sure CInt -> which jet clustering algo is gonna make Julia function not "grounded"

[m-fila]

For us the biggest problems with trimming are:

• no catching exceptions - this is the real elephant in the room here. If any Julia exception reaches C-layer then it's terminate, yet --trim=safe can't catch exceptions. In that example they too throw error if
• no logging allowed
• no cpu-features quering (no LoopVectorization)

From what I understood support for exceptions and logging in static analysis might be added in future

[m-fila]

Also to keep track, the build organization can be rewritten with https://github.com/JuliaLang/JuliaC.jl