Deno.command spawn stdout hangs on read

[caperaven]

deno 1.41.1 (release, x86_64-pc-windows-msvc) v8 12.1.285.27 typescript 5.3.3

I am trying to run a python command "python ...path/file.py" I need to read the response to see if it started up correctly.

From deno.

const command = new Deno.Command(this.#command, {
    args: this.#args,
    stdout: "piped"
});

this.#childProcess = await command.spawn();
const success = await read_stdout(this.#childProcess);

in read_stdout I have this

const decoder = new TextDecoder();
const reader = process.stdout.getReader();
const read_value = (await reader.read()).value;
const text = decoder.decode(read_value);

but on the reader.read it just hangs for ever

[lucacasonato]

Did you check that the process did not crash, or that the script is logging to stderr?

Please provide a reproduction that I can run on my machine.

[aunyks]

@lucacasonato I'm running into the same issue.

You can reproduce this using the following script (please have a Python 3 version available in your $PATH). The script should eventually output Printed "Hello World" in Python from JS!, but it instead hangs on the reader.read() line.

deno run --allow-run python-repl.js

const pythonProcess = new Deno.Command('python', {
  args: ['-i'],
  stdin: 'piped',
  stdout: 'piped',
  stderr: 'piped',
})

console.log('Starting Python process')
const process = pythonProcess.spawn()

// Getting an unknown error with the pipeTo function
// ignore for proof of concept
// process.stderr.pipeTo(Deno.stderr)
const reader = process.stdout.getReader()
const writer = process.stdin.getWriter()

const encoder = new TextEncoder()
const decoder = new TextDecoder()

async function writeToPython(command) {
  writer.write(encoder.encode(command + '\n'))
}

async function readFromPython() {
  let result = ''
  while (true) {
    console.log('trying to read from process')
    const { value, done } = await reader.read()
    console.log('read from process')
    if (done) {
      console.log('done reading')
      break
    }
    result += decoder.decode(value)
    console.log(`concatenated result "${result}"`)

    // ignore prompt line
    if (result.includes('>>>')) {
      break
    }
    console.log('finished read loop')
  }
  return result.trim()
}

try {
  console.log('Writing to Python')
  await writeToPython('print("Hello World")')
  console.log('Reading from Python')
  const output = await readFromPython()
  console.log('Python output', output)

  if (output.includes('Hello World')) {
    console.log('Printed "Hello World" in Python from JS!')
  } else {
    console.log(
      "Something went wrong. Expected 'Hello World', but got:",
      output
    )
  }
} catch (error) {
  console.error('An error occurred:', error)
} finally {
  reader.releaseLock()
  writer.close()
  process.kill()
}

[aunyks]

I've looked into this more. In the above example, the second reader.read() call hangs upon execution of this line in 06_streams.js. It's likely caused by the op_read() core function. When I next have time, I'll try to craft a test to reproduce the issue on a more focused level.

[aunyks]

Yep, the following test reproduces the issue: read() works once then hangs on the next. Run cargo test resource_test -- --nocapture against deno_core's testing crate after adding this test to reproduce.

test(async function testPipeLargeRead() {
  const [p1, p2] = op_pipe_create()
  const bufferSize = 65536
  const maxChunkSize = 1024
  const inBuffer = new Uint8Array(bufferSize)
  inBuffer[0] = 1
  assertEquals(maxChunkSize, await Deno.core.write(p1, inBuffer))
  const buf = new Uint8Array(1024)
  for (
    let chunkIndex = 0;
    chunkIndex <= bufferSize / maxChunkSize;
    chunkIndex++
  ) {
    console.log('trying read')
    assertEquals(maxChunkSize, await Deno.core.read(p2, buf))
    console.log('did read')
    // assertArrayEquals(buf.subarray(0), [i])
  }
})

You'll see this in the console:

...
trying read
did read
trying read

then the test doesn't even complete.

[aunyks]

Thanks @bartlomieju. Moving further troubleshooting / debugging discussion to that thread.

[PkmX]

Wondering if there is any movement or workaround (other than redirecting stdout to a temporary file and reading from that) for this. It currently makes Deno.command unusable for running a command and capturing its outputs directly.

[DevRubicate]

I'd also like to know if there is any progress being made on this issue, I got burned rather hard by the strange stalling behavior until I could narrow it down to that Deno.command's output is broken for any non-trivial amount of data. While the workaround of a temporary file is neat, this seems like an open wound in Deno as it stands today. 😱