gchq
Rust compaction
THIS IS AN EXPERIMENTAL PR. NOT READY FOR MERGE.
Issue
Resolves 1359
Tests
Rust compaction units tests not present yet.
It takes 20 minutes to to build this PR (compared to around 4 without the Rust build) and it takes 20 minutes every time. Is there any way to avoid recompiling the Rust code if nothing has changed?
We will need both unit tests in the Rust code, and the Java unit tests to test both the Java and Rust implementations.
The actual compaction code doesn't seem to be sending many logs to CloudWatch - it's harder to see that the job is processing than it is with the Java implementation - there are no logs of read 1000000 records, read 2000000 records, ...:
[main] compaction.jobexecution.RustBridge DEBUG - Attempting to load native library from JAR path natives/release
[main] compaction.jobexecution.RustBridge DEBUG - Attempting to load native library from JAR path natives/x86_64-unknown-linux-gnu/release
[main] compaction.jobexecution.RustBridge DEBUG - Extracted file is at /tmp/nativelib-loader_14764728764528488113/libcompaction.so
2023-09-26T12:11:32 [INFO] - load_region; provider=EnvironmentVariableRegionProvider { env: Env(Real) }
[Thread-0] common.action.ChangeMessageVisibilityTimeoutAction INFO - Compaction job...
More comments to come.
I've split out a separate issue to improve test coverage for these changes, so that we can hopefully get this PR merged close to as it is now:
- https://github.com/gchq/sleeper/issues/2850
