bun icon indicating copy to clipboard operation
bun copied to clipboard

[company prod issue] pipe file stream to process hangs

Open oguimbal opened this issue 1 year ago • 1 comments

What version of Bun is running?

1.1.9+bb13798d9

What platform is your computer?

Darwin 23.1.0 arm64 arm

What steps can reproduce the bug?

  1. Ensure that you have lz4 available locally (sudo apt-get install lz4 or brew install lz4 or whatever)

  2. Save this script in a repro.js

const { spawn } = require("child_process");
const { createReadStream, mkdirSync } = require("fs");

// download a random archive to extract
downloadRandomArchive() 
  .then(async (archiveFile) => {

    // remove target directory & children
    await toPromise("rm dir", spawn("rm", ["-rf", ".target-dir"]));
    mkdirSync(".target-dir");

    // spawn processes
    const lz4 = spawn("lz4", ["-c", "-d", "-"]);
    const tar = spawn("tar", ["-x", "-C", ".target-dir"]);
    const lz4p = toPromise("lz4", lz4);
    const tarp = toPromise("tar", tar);

    // pipe lz4 to tar
    lz4.stdout.pipe(tar.stdin);

    // open the file to extract
    const f = createReadStream(archiveFile);

    // pipe the file to lz4
    f.pipe(lz4.stdin);

    // wait for both processes to finish
    await Promise.all([lz4p, tarp]);
  });

// ----- utilities -----

function downloadRandomArchive() {
  const target = "archive.tar.lz4";
  const dl = spawn(
    "curl",
    [
      "-L",
      "-r",
      "0-10000000", // only download a bit
      "https://huggingface.co/JCTN/fast-repo/resolve/main/repo.tar.lz4?download=true",
      "-o",
      target,
    ],
    {
      redirect: "manual",
    }
  );
  return toPromise("curl", dl).then(() => target);
}

function toPromise(name, p) {
  return new Promise((resolve, reject) => {
    p.on("exit", (code) =>  code ? reject(new Error(`${name} exited with code ${code}`)) : resolve());
  });
}
  1. Check that it works in node: node repro.js will extract things in .target-dir (will end with an error though - coz' the archive is not fully downloaded)

  2. Check that bun repro.js hangs forever, without extracting anything

What is the expected behavior?

Bun & Node should behave the same

What do you see instead?

Bun hangs.

Additional information

No response

oguimbal avatar May 23 '24 14:05 oguimbal

minimal reproduction

const string = "A".repeat(1_000_000);

const proc = Bun.spawn({
  cmd: ["cat"],
  stdio: ["pipe", "pipe", "inherit"],
});

for (let i = 0; i < 10; i += 1) {
  proc.stdin.write(string);
}
proc.stdin.end();

for await (const chunk of proc.stdout) {
  console.log(chunk.length);
}
await proc.exited;

paperclover avatar May 24 '24 04:05 paperclover