recipes: Introduce `RustCompiledComponentsRecipe`, add `pydantic-core` and update `cryptography`

[T-Dynamos]

Closes #2661 #2955 Surpasses #2957

CC: @AndreMiras @misl6 Is approach used here is fine?

[T-Dynamos]

@Julian-O Hi, can you please triggre CI?

[Julian-O]

Sorry, I don't have permissions for that. @misl6?

[AndreMiras]

Interesting approach, thanks for looking this up. The CI actually tried running, but failed because of a syntax error:

[Invalid workflow file: .github/workflows/push.yml#L232](https://github.com/kivy/python-for-android/actions/runs/7658579349/workflow)
The workflow is not valid. .github/workflows/push.yml (Line: 232, Col: 13): Unexpected symbol: '|'. Located at position 23 within expression: (
  github.event_path | contains('pydantic-core')
  # github.event_path | contains('dir2') ||
  # github.event_path | contains('dir3')
)

See here https://github.com/kivy/python-for-android/actions/runs/7658579349 This would be our first Rust recipe I think. It's probably OK that we don't generalize anything yet, but we may want as we get more of them. I don't have much knowledge about Rust, so I don't know if the current method is future proof for dealing with different compiler versions plus dependencies, but I'm fine with it as it's a first, as long as the CI is happy about it. misl6 should have the last word as I haven't been very active in p4a for a long time

[T-Dynamos]

Build works, please somebody cancel all the previous actions triggred except the latest one.

[misl6]

@T-Dynamos FYI, Unit tests are failing.

Regarding the approach, there's something that will be repetitive along all the upcoming recipes for rust-backed libraries? If yes, how about moving these things into a RustCompiledComponentsRecipe ?

[T-Dynamos]

RustCompiledComponentsRecipe

It's good idea. As cryptography is also built on rust.

[misl6]

RustCompiledComponentsRecipe

It's good idea. As cryptography is also built on rust.

Do not take it as a "we should do it like that", feel free to improve the thinking.

Please also remind to add the new dependency into prerequisites.py, and in docs.

[T-Dynamos]

@misl6 Done!, After review I will document all.

[misl6]

Hi @T-Dynamos ,

While we will do our best to fix "Unit tests & build apps / Pytest" tests (not your fault), "Unit tests & build apps / Test updated recipes" are failing due to the rust-related changes.

Can you please take care of it?

[T-Dynamos]

@misl6 Done!

[FilipeMarch]

Cool I tested it here and got pydantic 2 compiled with rust in my Android app! That's very impressive work @T-Dynamos

[T-Dynamos]

@misl6 Happens only on Apple M1, something wrong with linking?

    File "/Users/admin/.python-for-android/build/other_builds/cryptography/arm64-v8a__ndk_target_21/cryptography/src/rust/cryptography-cffi/../../_cffi_src/utils.py", line 61, in build_ffi
      ffi = FFI()
            ^^^^^
    File "/Users/admin/.python-for-android/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/cffi/api.py", line 48, in __init__
      import _cffi_backend as backend
  ImportError: dlopen(/Users/admin/.python-for-android/build/other_builds/hostpython3/desktop/hostpython3/native-build/Lib/site-packages/_cffi_backend.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_ffi_prep_closure'

On my linux x86_64 machine:

❯ nm -gD libffi.so | grep ffi_prep_closure
0000000000006e10 T ffi_prep_closure@@LIBFFI_CLOSURE_8.0
0000000000006da0 T ffi_prep_closure_loc@@LIBFFI_CLOSURE_8.0

Lib file is located at .buildozer/android/platform/build-armeabi-v7a/build/libs_collections/myapp/armeabi-v7a/

Edit: Actually I think this issue presisted in previous runs also but github didn't reported it. All Apple M1 actions ran for 0 seconds :).

[T-Dynamos]

@misl6 A friendly ping. Any progress on review?

[misl6]

Apple M1 runner had some issues, let's wait until it picks up the jobs.

Apart from that, I can't test it promptly ATM as I'm majorly focused on kivy/kivy, but code LGTM and I do not see anything that can hurt the current packaging, so looks like just an improvement.

If the CI is happy, I will be happy too.

Just take into consideration to help out other users with docs/additional PRs if any issue arises.

[misl6]

@T-Dynamos can you please rebase on top of develop as CI pipeline for apple-silicon-m1 looks to fail on an issue that has already been fixed via https://github.com/kivy/python-for-android/pull/2842

[T-Dynamos]

@misl6 CI looks happy now.

[misl6]

Looks like we have a failure on 3 jobs with almost the same issue:

2024-03-17T08:12:12.1384103Z     sources = self.generate_sources(sources, (lib_name, build_info))
2024-03-17T08:12:12.1384888Z               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-17T08:12:12.1386664Z   File "/home/user/.local/share/python-for-android/build/other_builds/numpy/arm64-v8a__ndk_target_21/numpy/numpy/distutils/command/build_src.py", line 378, in generate_sources
2024-03-17T08:12:12.1388177Z     source = func(extension, build_dir)
2024-03-17T08:12:12.1388716Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-17T08:12:12.1390355Z   File "/home/user/.local/share/python-for-android/build/other_builds/numpy/arm64-v8a__ndk_target_21/numpy/numpy/core/setup.py", line 702, in get_mathlib_info
2024-03-17T08:12:12.1391864Z     config_cmd.compiler = bk_c.cxx_compiler()
2024-03-17T08:12:12.1392413Z                           ^^^^^^^^^^^^^^^^^
2024-03-17T08:12:12.1393495Z AttributeError: 'UnixCCompiler' object has no attribute 'cxx_compiler'

Can you make sure you have rebased on top of latest develop branch?

I'm wondering if the setuptools version bump may be the issue?

[T-Dynamos]

Looks like we have a failure on 3 jobs with almost the same issue:

2024-03-17T08:12:12.1384103Z     sources = self.generate_sources(sources, (lib_name, build_info))
2024-03-17T08:12:12.1384888Z               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-17T08:12:12.1386664Z   File "/home/user/.local/share/python-for-android/build/other_builds/numpy/arm64-v8a__ndk_target_21/numpy/numpy/distutils/command/build_src.py", line 378, in generate_sources
2024-03-17T08:12:12.1388177Z     source = func(extension, build_dir)
2024-03-17T08:12:12.1388716Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-03-17T08:12:12.1390355Z   File "/home/user/.local/share/python-for-android/build/other_builds/numpy/arm64-v8a__ndk_target_21/numpy/numpy/core/setup.py", line 702, in get_mathlib_info
2024-03-17T08:12:12.1391864Z     config_cmd.compiler = bk_c.cxx_compiler()
2024-03-17T08:12:12.1392413Z                           ^^^^^^^^^^^^^^^^^
2024-03-17T08:12:12.1393495Z AttributeError: 'UnixCCompiler' object has no attribute 'cxx_compiler'

Can you make sure you have rebased on top of latest develop branch?

I'm wondering if the setuptools version bump may be the issue?

Yes setuptools version bump is the issue, I think this is the time to update numpy to latest version. I will do this ASAP.

[T-Dynamos]

@misl6 Done! :fire: .

[T-Dynamos]

@misl6 Any update on this?

[T-Dynamos]

@misl6 Fixed.

[T-Dynamos]

Sorry had to do one last change, please cancel other actions.