Line buffering option for Deno.Command stdio streams.

[lrowe]

I'm using Deno.Command to write tests for an external program where I need to process its output line by line to determine when to interact with it. However using "piped" means output is fully buffered rather than line buffered.

It would be convenient if Deno.Command offered a line buffered mode to avoid needing to wrap command invocations with stdbuf, e.g. stdbuf --output=L command <args>.

It would presumably make sense to add an unbuffered option at the same time.

Prior art

Python's subprocess.Popen enables line buffering when specifying bufsize=1. However it could make sense to offer this on a per stream basis, perhaps "lines" as an alternative to "piped".

[bartlomieju]

Do you have an example reproduction that doesn't work as expected?

[lrowe]

Running this demonstrates the problem, although there is probably more research required on the best way to go about it (presumably it would need to fake a tty.)

The program I'm testing was written in C/C++ and worked with stdbuf.

import { TextLineStream } from "jsr:@std/streams/text-line-stream";
const pycode = `
import time
for i in range(100):
    print(i)
    time.sleep(0.1)
`;
// python buffers output when not connected to a tty 
const command1 = new Deno.Command("python3", { args: ["-c", pycode], stdout: "piped" });
// stdbuf does not work for python. https://stackoverflow.com/questions/55654364/why-stdbuf-has-no-effect-on-python
const command2 = new Deno.Command("stdbuf", { args: ["--output=L", "python3", "-c", pycode], stdout: "piped" });
// buf unbuffer does (from expect)
const command3 = new Deno.Command("unbuffer", { args: ["python3", "-c", pycode], stdout: "piped" });

await using proc = command1.spawn();
const lines = proc.stdout.pipeThrough(new TextDecoderStream("latin1")).pipeThrough(new TextLineStream);
for await (const line of lines) {
  console.log(line);
  if (line.trim() === "10") {
    break;
  }
}
console.log("done");

[bartlomieju]

Hey @lrowe, sorry for a slow response. We tried this reproduction code on Linux and this is what we got:

Which to me makes sense - Deno stdio when using subprocesses is line-buffered (because that's the default Tokio library we use provides). So I'm a bit of a loss here what the problem might be.

[lrowe]

Sorry, it's not the clearest repro. While I get the same output the difference is in how it is produced:

• With await using proc = command1.spawn(); I see the results print all at once after 10 seconds.
• With await using proc = command3.spawn(); I see each single line printed once every 0.1 seconds and done after 1 second.

This matters when you are interacting with the child process where you are waiting on child output still buffered in the pipe buffer before interacting in some way while the child is waiting for your interaction before printing anything more. In my case that was waiting for a listening line to see that the port was open before making an http request to the child.

# with command1
$ time deno run --allow-run repro.ts 
0
1
2
3
4
5
6
7
8
9
10
done

real    0m10.084s
user    0m0.030s
sys     0m0.037s

# with command3
$ time deno run --allow-run repro2.ts 
0
1
2
3
4
5
6
7
8
9
10
done

real    0m1.057s
user    0m0.034s
sys     0m0.027s


$ deno upgrade --canary
Current Deno version: v2.3.5+173f26f
Looking up canary version

Local deno version 173f26f391cdb43fb3c46a74365619dd77928623 is the most recent release

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"

[0f-0b]

In Deno, standard I/O streams are always unbuffered. It's the external program that's deciding to fully buffer its output. The way to prevent this depends on what language that program is written in.

• In C, stdout is either line buffered or unbuffered if it's connected to a terminal, and fully buffered otherwise. This can be changed at runtime with setbuf or setvbuf.
  • stdbuf tells the dynamic linker to load libstdbuf (which calls setvbuf) before the actual program, so it only works with programs that use C's stdio interface and dynamically links to the same libc that libstdbuf is compiled for.
• In Python, sys.stdout is unbuffered if the -u flag is passed, line buffered if it's connected to a terminal, and fully buffered otherwise.
  • unbuffer runs the given program in a pseudoterminal, so it works with both C and Python programs whether dynamically linked or not. #3994 might be relevant.

[lrowe]

I think @0f-0b is right about this needing pty support. I just tried with a simple c program and setting Python's subprocess.Popen bufsize to either 0 (unbuffered) or 1 (line buffered) seems to have no effect.

#include <stdio.h>
#include <unistd.h>

int main() {
  for(int i = 0; i < 100; i++) {
    printf("%d\n", i);
    usleep(100000);
  }
  return 0;
}