Connection error with parallel testing

[mahmoudyusof]

Describe the feature:

Elasticsearch version (bin/elasticsearch --version): 7.14.1

elasticsearch-py version (elasticsearch.__versionstr__): 7.14.1

Description of the problem including expected versus actual behavior: When running tests sequentially using pytest everything works fine, however, once I try to run tests in parallel using pytest -n=4 I get the following error: elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f201879e978>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f201879e978>: Failed to establish a new connection: [Errno 111] Connection refused) I make sure before every test that the elasticsearch server is running, after this error I check the server status to see that is failed.

[sethmlarson]

Our test suite isn't parallelizable in all cases due to setup and teardown of the Elasticsearch server. However when I try running with what I assume is pytest-xdist I don't receive connection errors, you're likely not running Elasticsearch properly for the test suite to connect to.

[mahmoudyusof]

@sethmlarson I don't know why you closed the issue, the problem doesn't appear only with testing, it appears also when using uwsgi on multiple workers, parallel connections doesn't seem to work properly, it throws the same error and it stops the elasticsearch server

[mahmoudyusof]

before running uwsgi I make sure to run sudo service elasticsearch status and it is running, after I run uwsgi it fails with the given error, and it shuts down elasticsearch servers

[mahmoudyusof]

before running uwsgi I make sure to run sudo service elasticsearch status and it is running, after I run uwsgi it fails with the given error, and it shuts down elasticsearch servers

[sethmlarson]

@mahmoudyusof Sorry for closing the issue early, you only mentioned thr test suite in your initial issue. Could you share details and code you're using for your web/worker so I can try reproducing the issue on Monday?

[mahmoudyusof]

Configuration

sure, I am using uwsgi to serve a flask application with the configuration below

[uwsgi]
wsgi-file = wsgi.py
callable = app
socket = :8080
processes = 4
threads = 2
master = true
chmod-socket = 660
vacuum = true
die-on-term = true
cheaper = 0
lazy-apps = true
enable-threads=true
die-on-term = true
need-app = true
disable-logging = true
master = true

and as mentioned earlier, I make sure before running anything that the elasticsearch server is up and running

Code

I am using Elasticsearch client which is, as the documentation suggests, abstracts the handling of a connection pool
I define the client locally in the functions that uses it as follows

es = Elasticsearch("localhost:9200")

I've also tried defining the client globally and tried using a list instead of a string without any luck

es = Elasticsearch(["localhost:9200"])

Error

The code runs fine if I run flask the normal way (python app.py) however, when I run it using uwsgi on multiple processes and multiple threads, it shows the error mentioned in the first comment in this issue. Same happens with testing, when I run tests sequentially everything runs fine, once I use xdist run tests in parallel, the same error shows up and the server of elasticsearch goes down.