bloomberg
Ability to write to a pipe from `memray.Tracker`
Is there an existing proposal for this?
- [X] I have searched the existing proposals
Is your feature request related to a problem?
I want to write to a fifo instead of a file but memray complains if the file I point it to already exists. The end problem I am trying to solve is that I want to stream the result to a distributed file system instead of place it on the disk.
Describe the solution you'd like
I'd like to write to a pre-existing fifo pipe.
Alternatives you considered
If memray could just give me the bytes in an iterator that would be fine too.
As things stand today, this isn't simple. Memray expects to be able to mmap the output file, and pipes cannot be mmapped.
We do support binding to a TCP port, waiting for a client to connect, and then writing the captured data over that socket - https://bloomberg.github.io/memray/api.html#memray.SocketDestination
That's intended for use by the memray live TUI, but you might be able to use that for your needs by spawning a thread or process that connects to that TCP port and writes everything it receives to your FIFO.
After talking about it a bit, we decided we would address this in two separate ways:
- Add documentation for writing a wrapper script that smuggles the capture file out: #598
- Add a way for the Tracker API to take a callback for user post-processing: #599
Closing this in favor of the other two.
