hash
hash copied to clipboard
hashintel
Implement fallback for shared memory allocation if resizing fails
🌟 What is the purpose of this PR?
If shared memory resizing fails, we currently error. An alloc-copy-dealloc fallback will have another chance to reallocate shared memory. This should fix the resizing limitation on macOS!
🔗 Related links
- Asana task (internal)
- hashdeps/shared_memory-rs#38
🚫 Blocked by
- [x] Needs to be tested on macOS
- [ ] Investigate memory leak (internal)
- [ ] Ensure that Python will unmap the memory (internal)
🔍 What does this change?
- Add method to create
MemoryIdfrom the shared memory os id - Add an alloc-copy-dealloc fallback for
Segment::resize - Remove workaround for macOS on
Segment-creation
📜 Does this require a change to the docs?
No: The workaround was not mentioned in the README (but should be added if this workaround does not work)
❓ How to test this?
Run a simulation with these changes, e.g.
RUST_LOG=warn cargo run -p cli -- -p tests/examples/air_defense_system single-run -n 15 --log-format full
📹 Demo
0.192897128s WARN memory::shared_memory::segment: Could not resize memory, using fallback old_size=968 new_size=2048 id="shm_6d35f9b18209467ba642c6c69800a7f7_37706"
0.257926919s WARN memory::shared_memory::segment: Could not resize memory, using fallback old_size=2048 new_size=8192 id="shm_6d35f9b18209467ba642c6c69800a7f7_13375"
0.316792384s WARN memory::shared_memory::segment: Could not resize memory, using fallback old_size=1024 new_size=2048 id="shm_6d35f9b18209467ba642c6c69800a7f7_13885"
0.372096694s WARN memory::shared_memory::segment: Could not resize memory, using fallback old_size=2048 new_size=2432 id="shm_6d35f9b18209467ba642c6c69800a7f7_57370"
- To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1202482070497363
Not going to merge this right now because
- We found a memory leak with is most probably related to shared memory
- Python probably needs unmapping of the re-allocated segment, this needs further investigation
Added links to the OP
To note: https://github.com/hashintel/hash/pull/985 fixes a memory leak caused by resources not being properly cleaned up at the end of experiments, however memory leak investigation remains ongoing.
