Handle file open/close during experimental sessions

Some Gecko debugging helpers dump data to named files. This would require support stdio open/write/close. At least open() would need to be explicitly allowed and executed. Maybe we should only allow open() to create a file that doesn't already exist, to avoid a replaying process accidentally damaging the file system.

Interesting. If we wanted, we could fully emulate open/write/close, because there's no (real) syscall buffering during experiment mode. That would also let us create files in an experiment-specific emufs, although that wouldn't necessarily be obvious to the user.

These files need to be visible to the user. E.g. the user does p gfxUtils::WriteAsPNG(drawTarget, "/tmp/frame.png").

By "emulate", I meant use PTRACE_SYSEMU to reach the syscall but have rr implement the syscall, like we use for writes to STD*_FILENO. rr would open the file, if it chose, on behalf of the tracee. So the file would appear in the fs (if rr chose), but the tracee processes wouldn't have real fds that referred to it.

I also tried dumping the js heap to stdout in replay (crazy, I know) and that just seemed to hang rr. Is that a similar issue to what's being discussed here?

Could be. Is it reproducible? Exactly which function call(s) did you try?

Weird, I tried again and it worked!

I broke here: http://mxr.mozilla.org/mozilla-central/source/js/xpconnect/src/XPCComponents.cpp#2888 (PreciseGCRunnable::Run), then did |call js::DumpHeapComplete(rt, (FILE*)stdout, js::CollectNurseryBeforeDump)|

For what it's worth, I would find it very useful to be able to run call dump_heap_to_file("/tmp/heap.dot") in GDB.

edit: It actually seems that this already works; horray!