clickhouse-sqlalchemy icon indicating copy to clipboard operation
clickhouse-sqlalchemy copied to clipboard

session hangout without any error output

Open flyly0755 opened this issue 2 years ago • 1 comments

Describe the bug https://github.com/xzkostyan/clickhouse-sqlalchemy/blob/master/clickhouse_sqlalchemy/drivers/http/transport.py#L102

        self.timeout = float(timeout) if timeout is not None else None

with this line code, is there any issue here when timeout is None, then self.timeout is None? when self.timeout is None, means code below the post method with self.timout this inarg will not timeout forever?

 r = self.http.post(
            self.db_url, auth=self.auth, params=params, data=data,
            stream=stream, timeout=self.timeout, headers=self.headers,
            verify=self.verify, cert=self.cert
        )

I met a problem below, with a batch of files need to parse and parsed data to insert into clickhouse. in the python script loop the files, every file build a database connection session for inserting data; after running the script a long time, then the run will hangout and without raising any error info. manual stop the run, and with output: site-packages\clickhouse_sqlalchemy\drivers\http\transport.py, line 169, in _send r = self.http.post( site-packages\requests\sessions.py, line 577, in post return self.request("POST", url, data=data, json=json, **kwargs) site-packages\requests\sessions.py, line 529, in request resp = self.send(prep, **send_kwargs) site-packages\requests\sessions.py, line 645, in send r = adapter.send(request, **kwargs) site-packages\requests\adapters.py, line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: (‘Connection aborted.’, RmeoteDisconnected('Remote end closed connection without response'))

The clickhosue server maybe sometime miss handle some request, so without response. But with this situation, i think the corresponding request will timeout and raise error. but now with this hangout issue, is there any relation with the timeout parameter? coz is None, so the session will not timeout, and always wait the response from clickhouse server side no matter how long time?

To Reproduce as long as the script running of file loop operation doesn't finish, the hangout issue always occurs, just a time issue.

Expected behavior session with a default timeout config, if not provide timeout with the inarg, then use the default value as timeout setting, once a response is not reveived after timeout time, then raise error instead of hangout.

Versions

python 3.8.5 clickhouse-sqlalchemy 0.2.0 requests 2.27.1

flyly0755 avatar Aug 01 '22 07:08 flyly0755

engine = create_engine(dbsession.bind.url, echo=False, connect_args={'timeout': 600}, poolclass=NullPool)
session = sessionmaker(bind=engine)()

the seesion settings as above

flyly0755 avatar Aug 01 '22 08:08 flyly0755