ES connection handling

[revlis-x]

In various places in the code new Elasticsearch-Clients are instantiated for every request, e.g. in /src/platform/magento2/tax.js - Line 83 (with tax.calulateServerSide = true): const client = es.getClient(this._config)

or in src/api/extensions/elastic-stock/index.js - Line 20: return getElasticClient(config).search(esQuery).then((products) => {

This causes new TCP-connections to be established on the system level.

Where in v.1.10 and before the legacy elasticsearch-js library would be used ("elasticsearch": "^15.2.0",), this was not a problem, because these TCP connections would timeout by default.

In 1.11 new elasticsearch-library is used ("@elastic/elasticsearch": "^7.3.0",) and with that TCP connections are not timed out by default and basically stay open forever. This leads to a situation where the system runs out of available TCP ports eventually and no more connections to ES can be established. (EADDRNOTAVAIL errors).

One can see established TCP connections rising and never falling again using watch -n 1 ss -s and hitting affected routes (e.g. /api/ext/elastic-stock/check/SKU or /api/catalog/vue_storefront_catalog/product/_search? with tax.calculateServerSide = true)

To mitigate one could actively close the connection by calling .close(), but better solution would be to re-use existing connection instead of instantiating new clients.

[ldc321]

In various places in the code new Elasticsearch-Clients are instantiated for every request, e.g. in /src/platform/magento2/tax.js - Line 83 (with tax.calulateServerSide = true): const client = es.getClient(this._config)

or in src/api/extensions/elastic-stock/index.js - Line 20: return getElasticClient(config).search(esQuery).then((products) => {

This causes new TCP-connections to be established on the system level.

Where in v.1.10 and before the legacy elasticsearch-js library would be used ("elasticsearch": "^15.2.0",), this was not a problem, because these TCP connections would timeout by default.

In 1.11 new elasticsearch-library is used ("@elastic/elasticsearch": "^7.3.0",) and with that TCP connections are not timed out by default and basically stay open forever. This leads to a situation where the system runs out of available TCP ports eventually and no more connections to ES can be established. (EADDRNOTAVAIL errors).

One can see established TCP connections rising and never falling again using watch -n 1 ss -s and hitting affected routes (e.g. /api/ext/elastic-stock/check/SKU or /api/catalog/vue_storefront_catalog/product/_search? with tax.calculateServerSide = true)

To mitigate one could actively close the connection by calling .close(), but better solution would be to re-use existing connection instead of instantiating new clients.

I also have the same problem, I want to ask how to solve it, this problem is bothering me, thanks