Sockets hanging with node16

[andrzej-woof]

What is your Scenario?

Web app that makes client side fetch to two API endpoints, both on localhost but one is not running, so doesn't accept connections (which is fine, error is handled and app is functional)

Running tests with node v14.x is just fine, after switching to v16 tests start to break after a while Symptoms observed in chrome network tools are that calls to both API endpoints are in pending state

After a while of debugging I've noticed that testcafe's behaviour for requests that are supposed to fail (since there's no server listening) is much different in node16 than node14 Looking at network tools: Node v16

Node v14

As you can see, in node 14 such request fails instantly, as it should. With node 16 the request is pending. I cannot prepare a sample that reproduces my exact problem (where after a while both /api calls are constantly pending in subsequent tests) but I'm assuming maybe sockets are not properly closed in my case and just saturate some system limits

What is the Current behavior?

Fetch requests that are supposed to fail are pending

What is the Expected behavior?

Requests should fail immediately if there's no server listening on the other end

What is your public website URL? (or attach your complete example)

https://github.com/andrzej-kodify/testcafe-hanging-sockets Launch server

npm run serve

Launch tests with node 16

npm run test:debug

and look in chrome network tools on how failed requests behave. Tests are passing, no JS errors.

Do the same with node 14 - requests fail immediately, and there's a JS error (as fetch exception is not handled)

What is your TestCafe test code?

// https://github.com/andrzej-kodify/testcafe-hanging-sockets

const { Selector } = require("testcafe");

fixture('Fixture');

for (let i=0; i<1000; i++) {
    test(`Test ${i}`, async (browser) => {
        await browser.navigateTo('http://localhost:3666/');
        await browser.expect(Selector('#response').innerText).match(/[0-9]+/);
    });
}

Your complete configuration file

No response

Your complete test report

No response

Screenshots

No response

Steps to Reproduce

TestCafe version

1.19.0

Node.js version

version v14.18.1 runs fine, versions 16+ (e.g. v16.15.1) fail

Command-line arguments

chrome

Browser name(s) and version(s)

Version 102.0.5005.61 (Official Build) (arm64)

Platform(s) and version(s)

macOS 12.4

Other

I've made silly workaround by applying this patch in my actual tests https://github.com/andrzej-kodify/testcafe-hanging-sockets/blob/main/testcafe-hammerhead%2B24.5.16.patch

[miherlosev]

Hi @andrzej-kodify

Thank you for the shared example. I've reproduced the issue.

[xiaowanfeng]

I meet this issue to

The request is pending

[marliz97]

Have the same issue, but only with multipart/form-data requests. In my case issue persists in both node14 and node16

[vizdatom]

Hello, I experience the same issue with pending requests, when I create a request to non-existing domain, so the DNS fails to resolve the domain name, and the error is not passed to the client.

[VasilyStrelyaev]

Thank you for the additional information! We increased the frequency level for this issue, which will give it higher priority during our future planning.

[cattermo]

Can someone please work on this?! We are stuck with super old Testcafé since this bug prevents us from upgrading node version. We have this problem with node 14 as well 😢

[miherlosev]

Hi @cattermo,

This issue may be fixed in the latest version of TestCafe (2.1.0). Please try running the tests in the latest TestCafe version and let us know your results.

[cattermo]

Hi @cattermo,

This issue may be fixed in the latest version of TestCafe (2.1.0). Please try running the tests in the latest TestCafe version and let us know your results.

Ok, so we have upgraded now since a couple of weeks back. Problem is, we are having so many intermittent errors where requests are just stuck and causing "Failed to fetch" errors, similar to when we tried last time to upgrade, found this bug reported, and decided to downgrade again. The "intermittent" part means that different random tests will have this error almost every test run but by using quarantine mode we can be lucky and have all tests pass.

Findings from trying to investigate this issue:

• The "stuck" requests are nowhere to be found in our own logs. Seems they are never sent.
• We only see this problem when running Testcafe, never for real users.
• We only see this problem when running tests against a real environment, not against localhost
• The problem is a lot worse on Node 16, on Node v14.18.3 only intermittent
• The process for how it happens (intermittently) is usually
  • Load GUI (React SPA)
  • All ajax requests stuck in pending
  • "Failed to fetch" javascript error fails test OR
  • Element X not found (since API request stuck)

Other info:

• We are using CORS (api on different sub-domain)
• Our load balancer that sits in front of the API is using http2

[miherlosev]

Hi @cattermo,

Thank you for checking tests in the new TestCafe version. I see the shared patch from the description, and it looks like the issue occurs because some request is incorrectly sent/processed in some rare case. At present, we are working on a special test run mode that uses the native Google Chrome functionality to send a request. It allows you to avoid issues with sending requests. We call it the Proxyless mode. In this mode, this issue may be fixed. Try running your tests in Proxyless mode and let us know the results. This option is available in all interfaces:

// Command-line
testcafe chrome tests --experimental-proxyless
// Programmatic
const testcafe = await createTestCafe({ experimentalProxyless: true });

// Configuration file
{
"experimentalProxyless": "true"
} 

Note that at present it is an experimental mode. Also, the Proxyless mode is implemented only in Google Chrome. It will not work correctly if you run tests in a non-Chrome browser or in a combination of other browsers.

[dcsaszar]

FYI after upgrading to 2.2.0 I learned about "experimentalProxyless": "true" and gave it a try. With this option, I run into this issue. I've never experienced this issue without the experimentalProxyless option. Node v16.14.2.

Update 2023-01-05: when I was able to reproduce this consistently, I was on a low-bandwidth mobile tethered connection. Now, with a cable or high bandwidth mobile connection, I can no longer reproduce it. 😿

[aleks-pro]

Hello @dcsaszar ,

Thank you for pointing out the issue to us. According to your description, the cause of the problem most likely is different from the original issue in this thread, although the symptoms are the same. Could you please create a separate issue with a project sample that we can use to reproduce the problem? It will help us to improve the Proxyless mode and take it out of the experimental state.

[cattermo]

One more comment to this, we really can't get Testcafe to work on node 16/18 without a lot of requests stuck in pending (intermittently). The experimentalProxyless option did not make any difference. It works well with node v14.18.3 and Testcafe v2.1.0.

[aleks-pro]

Hi @cattermo ,

Since the problem in your project can be reproduced with the experimental Proxyless mode enabled, I'm not sure that this issue and the issue from the original post are the same. Could you share your project sample and a test with which we can reproduce the problematic behavior?

[dandoniev]

This problem is also relevant to REST API requests if there's several of them at the same moment. Node v16.16.0, Testcafe v2.2.0

[miherlosev]

Hi @dandoniev,

Thank you for sharing this information. Did you try using the experimentalProxyless option?

[dandoniev]

Hi @miherlosev, yes I tried it, but application's authorization fails in case experimentalProxyless option enabled

[miherlosev]

@dandoniev

In this case, could you please share an example where the issue can be reproduced?

[Artem-Babich]

Hi, Would you please check if this issue is reproduced in Node 18 with the latest TestCafe version? Please also note that in v2.2.5, we renamed the experimental-proxyless option to native-automation.

[github-actions[bot]]

This issue was automatically closed because there was no response to our request for more information from the original author. Currently, we don't have enough information to take action. Please reach out to us if you find the necessary information and are able to share it. We are also eager to know if you resolved the issue on your own and can share your findings with everyone.

[cattermo]

We we're testing this last week but had some other issues so could not finalise the testing yet. Please do not close.

[aleks-pro]

Hello @cattermo ,

If you manage to reproduce the problematic behavior, feel free to reopen the issue. If you encounter some new problem, please create a separate issue with a reproducible example.

[tomasVizdal]

Hello, this problem still persists in Firefox and Safari. I am using tetscafe version 3.2.0 and node 18.17.1. Do you plan to fix that for Firefox and Safari?

[aleks-pro]

I reproduced the problematic behavior using the example shared in the original post in the environment with the following configuration:

• TestCafe version 3.0.0,
• NodeJS versions 16.17.0, 18.4.0, 20.1.0,
• Browsers: Chrome 116.0.0.0, Safari 16.6, Firefox 117.0,
• MacOS Ventura 13.5.1.

I will reopen the issue, but currently, we cannot give any estimates on when we will fix it.

[github-actions[bot]]

This issue was automatically closed because there was no response to our request for more information from the original author. Currently, we don't have enough information to take action. Please reach out to us if you find the necessary information and are able to share it. We are also eager to know if you resolved the issue on your own and can share your findings with everyone.

[cattermo]

@aleks-pro And there it was automatically closed again 😅 . Can someone re-open and remove the "Need response" label?