zed
zed copied to clipboard
Piping zq output to "less" causes spill directory to be left behind
Repro is with zq
commit 076e53b5 and some DNS test data fdns-5GB.zng.gz I've shared from my Google Drive.
I execute this:
$ zq -version
Version: v1.0.0-6-g076e53b5
$ zq -z 'head 10000000 | count() by name' fdns-5GB.zng.gz | less
After my results show up in less
and I hit q
to quit, there's a spill directory still left behind that needs to be manually deleted.
$ du -sh $TMPDIR/zed-spill-*
356M /var/folders/j3/7lxrfn3n59bghzn2yjfg46tr0000gn/T//zed-spill-4078656352
OTOH, if I don't block the output, the spill directory gets cleaned up just fine. So if I run it this way:
$ zq -z 'head 10000000 | count() by name' fdns-5GB.zng.gz > /dev/null
I can watch this in another window to see the spill directory appear and ultimately get cleaned up on exit.
$ cd $TMPDIR && while true; do ls -l | grep -i spill; du -sh zed-spill-*; sleep 10; done
du: zed-spill-*: No such file or directory
drwx------ 3 phil staff 96 Mar 30 13:33 zed-spill-3745359136
34M zed-spill-3745359136
drwx------ 5 phil staff 160 Mar 30 13:33 zed-spill-3745359136
106M zed-spill-3745359136
drwx------ 7 phil staff 224 Mar 30 13:34 zed-spill-3745359136
179M zed-spill-3745359136
drwx------ 9 phil staff 288 Mar 30 13:34 zed-spill-3745359136
254M zed-spill-3745359136
drwx------ 11 phil staff 352 Mar 30 13:34 zed-spill-3745359136
328M zed-spill-3745359136
drwx------ 11 phil staff 352 Mar 30 13:34 zed-spill-3745359136
321M zed-spill-3745359136
drwx------ 8 phil staff 256 Mar 30 13:34 zed-spill-3745359136
211M zed-spill-3745359136
drwx------ 6 phil staff 192 Mar 30 13:34 zed-spill-3745359136
136M zed-spill-3745359136
du: zed-spill-*: No such file or directory
As discussed in a recent team meeting, @nwt is planning a fix for #1431. He explained how researching that fix revealed that there's several places where this "spill directory left behind" problem may show up, and confirmed that the fix for #1431 should address this one as well. I've marked this one as Blocked by #1431 so that'll serve as a reminder to verify when the fix is in.
To fix this, we'll need to include SIGPIPE in the list of signals for signal.NotifyContext.
Verified in Zed commit a54eba9.
Repeating the repro steps as shown in the attached video, we see now that when we q
out of less
, the spill directory is automatically cleaned up rather than being left behind as it was before.
https://user-images.githubusercontent.com/5934157/200442001-0f9df873-3b93-4d83-abc9-d775bc58c597.mp4
Thanks @nwt!