zed icon indicating copy to clipboard operation
zed copied to clipboard

Piping zq output to "less" causes spill directory to be left behind

Open philrz opened this issue 2 years ago • 2 comments

Repro is with zq commit 076e53b5 and some DNS test data fdns-5GB.zng.gz I've shared from my Google Drive.

I execute this:

$ zq -version
Version: v1.0.0-6-g076e53b5

$ zq -z 'head 10000000 | count() by name' fdns-5GB.zng.gz | less

After my results show up in less and I hit q to quit, there's a spill directory still left behind that needs to be manually deleted.

$ du -sh $TMPDIR/zed-spill-*
356M	/var/folders/j3/7lxrfn3n59bghzn2yjfg46tr0000gn/T//zed-spill-4078656352

OTOH, if I don't block the output, the spill directory gets cleaned up just fine. So if I run it this way:

$ zq -z 'head 10000000 | count() by name' fdns-5GB.zng.gz > /dev/null

I can watch this in another window to see the spill directory appear and ultimately get cleaned up on exit.

$ cd $TMPDIR && while true; do ls -l | grep -i spill; du -sh zed-spill-*; sleep 10; done
du: zed-spill-*: No such file or directory
drwx------   3 phil  staff    96 Mar 30 13:33 zed-spill-3745359136
 34M	zed-spill-3745359136
drwx------   5 phil  staff   160 Mar 30 13:33 zed-spill-3745359136
106M	zed-spill-3745359136
drwx------   7 phil  staff   224 Mar 30 13:34 zed-spill-3745359136
179M	zed-spill-3745359136
drwx------   9 phil  staff   288 Mar 30 13:34 zed-spill-3745359136
254M	zed-spill-3745359136
drwx------  11 phil  staff   352 Mar 30 13:34 zed-spill-3745359136
328M	zed-spill-3745359136
drwx------  11 phil  staff   352 Mar 30 13:34 zed-spill-3745359136
321M	zed-spill-3745359136
drwx------   8 phil  staff   256 Mar 30 13:34 zed-spill-3745359136
211M	zed-spill-3745359136
drwx------   6 phil  staff   192 Mar 30 13:34 zed-spill-3745359136
136M	zed-spill-3745359136
du: zed-spill-*: No such file or directory

philrz avatar Mar 30 '22 20:03 philrz

As discussed in a recent team meeting, @nwt is planning a fix for #1431. He explained how researching that fix revealed that there's several places where this "spill directory left behind" problem may show up, and confirmed that the fix for #1431 should address this one as well. I've marked this one as Blocked by #1431 so that'll serve as a reminder to verify when the fix is in.

philrz avatar Sep 14 '22 18:09 philrz

To fix this, we'll need to include SIGPIPE in the list of signals for signal.NotifyContext.

nwt avatar Oct 13 '22 20:10 nwt

Verified in Zed commit a54eba9.

Repeating the repro steps as shown in the attached video, we see now that when we q out of less, the spill directory is automatically cleaned up rather than being left behind as it was before.

https://user-images.githubusercontent.com/5934157/200442001-0f9df873-3b93-4d83-abc9-d775bc58c597.mp4

Thanks @nwt!

philrz avatar Nov 08 '22 00:11 philrz