opendata.cern.ch
opendata.cern.ch copied to clipboard
Published
20 hours ago •
cernopendata
cernopendata
files: downloads hang intermittently
$ git checkout master
$ docker-compose -f docker-compose-dev.yml down -v
$ docker-compose -f docker-compose-dev.yml build
$ docker-compose -f docker-compose-dev.yml up
$ docker-compose exec web /code/scripts/populate-instance.sh --skip-records
$ docker-compose exec web cernopendata fixtures records -f /code/cernopendata/modules/fixtures/data/records/cms-csv-files.json --mode insert
$ curl http://0.0.0.0:5000/record/700/files/MuRun2010B_0.csv
.. hangs ..
Seeing the following logs on server side:
web_1 | 172.22.0.1 - - [29/Jan/2021 12:09:48] "GET /record/700/files/MuRun2010B_0.csv HTTP/1.1" 500 - [1278/1868]
web_1 | Traceback (most recent call last):
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/flask/app.py", line 2464, in __call__
web_1 | return self.wsgi_app(environ, start_response)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/werkzeug/middleware/dispatcher.py", line 66, in __call__
web_1 | return app(environ, start_response)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/flask/app.py", line 2450, in wsgi_app
web_1 | response = self.handle_exception(e)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/flask/app.py", line 1867, in handle_exception
web_1 | reraise(exc_type, exc_value, tb)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
web_1 | raise value
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
web_1 | response = self.full_dispatch_request()
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
web_1 | rv = self.handle_user_exception(e)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
web_1 | reraise(exc_type, exc_value, tb)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
web_1 | raise value
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
web_1 | rv = self.dispatch_request()
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
web_1 | return self.view_functions[rule.endpoint](**req.view_args)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/invenio_records_ui/views.py", line 169, in record_view
web_1 | pid, record = resolver.resolve(pid_value)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/invenio_pidstore/resolver.py", line 47, in resolve
web_1 | pid = PersistentIdentifier.get(self.pid_type, pid_value)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/invenio_pidstore/models.py", line 194, in get
web_1 | return cls.query.filter_by(**args).one()
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3490, in one
web_1 | ret = self.one_or_none()
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none
web_1 | ret = list(self)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
web_1 | return self._execute_and_instances(context)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3557, in _execute_and_instances
web_1 | querycontext, self._connection_from_session, close_with_result=True
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3572, in _get_bind_args
web_1 | mapper=self._bind_mapper(), clause=querycontext.statement, **kw
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3550, in _connection_from_session
web_1 | conn = self.session.connection(**kw)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1141, in connection
web_1 | execution_options=execution_options,
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1147, in _connection_for_bind
web_1 | engine, execution_options
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
web_1 | conn = bind._contextual_connect()
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2302, in _contextual_connect
web_1 | self._wrap_pool_connect(self.pool.connect, None),
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2336, in _wrap_pool_connect
web_1 | return fn()
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
web_1 | return _ConnectionFairy._checkout(self)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
web_1 | fairy = _ConnectionRecord.checkout(pool)
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
web_1 | rec = pool._do_get()
web_1 | File "/opt/invenio/var/instance/python/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 132, in _do_get
web_1 | code="3o7r",
web_1 | sqlalchemy.exc.TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30 (Background on this error at: http://sqlalche.me/e/13/3o7r)
For now (and only for developers) it helps to do:
$ docker-compose exec web cernopendata fixtures records -f /path/to/record.json --mode insert-or-replace
$ docker-compose restart web
Findings so far:
- The issue can be reproduced:
- a. Before the error appears: I can only get to the
.openfunction fromxrootdPython library. There either one of the two following things happen:- i. The call returns with a big delay
- ii. The call hangs indefinitely
- b. After the error appears: the code hangs/fails at PID resolution
- a. Before the error appears: I can only get to the
- The issue can't be reproduced (happening most of the time)
