Passthrough service hangs under high load

[xlight05]

Description: $Subject under load test.

Strand dump - https://gist.github.com/xlight05/af107fdd097dd6825fbe6bd6e13efa1e

Simpler sample -

import ballerina/http;
import ballerina/log;
type Location record {|
    string name;
    string id;
    string address;
|};

type Locations record {|
    *http:Links;
    Location[] locations;
|};

final http:Client snowpeakEP = check new("http://localhost:9090");

service /passthrough on new http:Listener(9091) {
    resource function get .() returns Locations|http:InternalServerError|error {
        log:printInfo("Request received at passthrough service");
        Locations locations = check snowpeakEP->get("/snowpeak/locations");
        return locations;
    }
}

Steps to reproduce:

1. Checkout to HTTP branch with latest u9 changes
2. build a pack using ./gradlew clean build -x check -x test
3. open a new terminal and run the dependant service - docker run -p 9090:9090 ktharmi176/snowpeak:latest
4. run the snowpeak sample with the created pack <http_path>/target/ballerina-runtime/bin/bal run
5. execute the service once - curl http://localhost:9091/passthrough
6. Run a load test against the service - wrk -t12 -c400 -d10s http://localhost:9091/passthrough
7. once the load test is over, execute the service again - curl http://localhost:9091/passthrough

Service should be hanging at this point.

Affected Versions: 2201.9.0-20240418-164300-a37919c4

OS, DB, other environment details and versions:

Related Issues (optional):

Suggested Labels (optional):

Suggested Assignees (optional):