vinyl-ftp
vinyl-ftp copied to clipboard
Fix bug: in glob.js scan(), after writing files, properly end writabl…
…e stream by calling stream.end()
- previous code
stream.emit( 'end' )
would cause cut off src() downloads after 16 files, or more accurately, after stream.highWaterMark files
I couldn't figure out why there was an apparent download limit for conn.src() of 16 files.
var conn = ftp.create({
"host": "...",
"user": "...",
"parallel": 1,
"password": "...",
"log":console.log
});
try {
return conn.src(['subdir/*'], {buffer:false})
.pipe(dest('downloaded/'))
} catch (err) {
console.error(err)
}
Everything was fine, no errors; it downloads up to 16 files, then disconnects.
Why 16? It turns out that parallel-transform sets its highWaterMark to 16 by default. Sure enough, when I override the parallel-transform stream's highWaterMark in helpers.js like so...:
var stream = parallel( p, {highWaterMark:2}, transform );
...we now download only 2 files.
CONN
READY
MLSD /subdir
GET \subdir\5FC253.csv
DOWN 100% \subdir\5FC253.csv
GET \subdir\61CEF1.csv
DOWN 100% \subdir\61CEF1.csv
DISC
I'm not sure why we see this particular behavior; stream.highWaterMark shouldn't be relevant to this situation at all. It seems to be an unexpected effect from incorrect use of stream.emit():
Avoid...emitting internal events such as 'error', 'data', 'end', 'finish' and 'close' through .emit().
Switching to stream.end() is correct and was surely the author's original intention.
Calling the writable.end() method signals that no more data will be written to the Writable.
And, in fact, this change corrects the observed behavior in all of my testing; all files are now downloaded regardless of stream.highWaterMark's value.