can't read stderr and stdout after one message during remote command execution

[GitBluub]

Get your issue reviewed faster

Sorry, I can't run data-collect.sh on talos. if this issue is missing information I will provide it OS. talos 1.7.2 Kata-container v3.4.0 Cloud-hypervisor from the static kata-container release containerd v1.7.16

Description of problem

During kubectl exec, I only get one message of stdout and one of stderr, the process keeps running (until it doesn't have an issue with stdout and stderr not flushed). It is very similar to #9071 so I began with the same repro script.

// Usage: node ws.js

// Prereqs:
// $ sudo apt install nodejs
// $ npm install ws
// $ kubectl apply -f busybox.yaml
// test.sh in the pod 
// echo '>&2 echo "error1" && sleep 1 && >&2 echo error2 && sleep 5 && echo out1 && sleep 15 && echo out2' > test.sh

const CLUSTER_URL = "wss://k8s-control-plane-url:443";
const POD_NAME = "busybox";

const fs = require('fs');
const WebSocket = require('ws');


let url = `${CLUSTER_URL}/api/v1/namespaces/default/pods/${POD_NAME}/exec?command=sh&command=-e&command=test.sh&stderr=true&stdout=true`;
let sock = new WebSocket(url, 'channel.k8s.io', {

    // Cert only needed when using the wss:// scheme.
    ca: fs.readFileSync(`./ca.crt`),
    cert: fs.readFileSync(`./client.crt`),
    key: fs.readFileSync(`./client.key`),
});

sock.on('open', () => console.log('open'));
sock.on('message', x => console.log('message', JSON.stringify(x.toString())));
sock.on('close', () => console.log('close'));

Expected result (I get it with the same busybox without kata runtime)

open
message "\u0001"
message "\u0002error1\n"
message "\u0002error2\n"
message "\u0001out1\n"
message "\u0001out2\n"
close

Actual result

open
message "\u0001"
message "\u0002error1\n"
message "\u0001out1\n"
close

Further information

containerd-debug.log

I also had the issue in 3.2.0 and I didn't see any other versions that might have fixed it, if you think another version already has a fix I can try.

[sprt]

Hi @GitBluub, this seems very similar to #9071 indeed - we may have to mark this as a duplicate.

Can you repro on 3.9?

[GitBluub]

I tested and have the same behavior on 3.9

[GitBluub]

Hello, any news or idea on how to fix this ? I tested on katacontainers 3.15.0 and the issue remains, It's a shame because it stops us from using kata with github actions runner controller, which we found to be a great use case for kata :/

[sprt]

@GitBluub Are you able to confirm if this is a dupe of #9071? If so we may close either one of these issues.