foundry
foundry copied to clipboard
Use rr to trace errors
https://rr-project.org/ It records syscall to enable replayable debugging. It has some limitations like emulating multicore, requiring performance monitor feature (not supported by VirtualBox, many types of AWS instances), but I think we can utilize it for our inner test server.
- [ ] TPS test in our inner test server (solo network)
- [ ] Check whether the record size is too huge or not.
It seems that logging magnifies the overhead of rr. Turning off the logging reduce overhead greatly towards virtually zero. rr record size doesn't seem to grow too much. I'll share exact numbers by next week.
I think we need to buffer the log or to optimize the code (like caching atty::is since it invokes ioctl syscall every time)