kolibri icon indicating copy to clipboard operation
kolibri copied to clipboard
verified publisher icon learningequality

Error raised during insufficient storage handling

Open rtibbles opened this issue 2 years ago • 8 comments

Observed behavior

When handling insufficient storage errors, an integrity error can occur while attempting to log the learner device status.

Errors and logs

ERROR    2024-04-02 08:47:16,010 Job f5ed1eb7ef6a4b588bff3b0ed3bf6cdd raised an exception: Traceback (most recent call last):
  File "/home/richard/github/kolibri/kolibri/core/content/utils/content_request.py", line 505, in process_content_requests
    _process_content_requests(incomplete_downloads)
  File "/home/richard/github/kolibri/kolibri/core/content/utils/content_request.py", line 745, in _process_content_requests
    raise InsufficientStorage(
kolibri.core.content.utils.content_request.InsufficientStorage: Content download requests need 1.61GB of free space

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/query.py", line 480, in update_or_create
    obj = self.select_for_update().get(**lookup)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/query.py", line 378, in get
    raise self.model.DoesNotExist(
kolibri.core.device.models.DoesNotExist: LearnerDeviceStatus matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.IntegrityError: UNIQUE constraint failed: device_learnerdevicestatus.user_id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/richard/github/kolibri/kolibri/core/tasks/job.py", line 329, in execute
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/github/kolibri/kolibri/core/tasks/registry.py", line 238, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/github/kolibri/kolibri/core/content/tasks.py", line 403, in automatic_resource_import
    process_content_requests()
  File "/home/richard/github/kolibri/kolibri/core/content/utils/content_request.py", line 511, in process_content_requests
    LearnerDeviceStatus.save_statuses(DeviceStatus.InsufficientStorage)
  File "/home/richard/github/kolibri/kolibri/core/device/models.py", line 652, in save_statuses
    cls.save_learner_status(user_id, status)
  File "/home/richard/github/kolibri/kolibri/core/device/models.py", line 674, in save_learner_status
    cls.objects.update_or_create(
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/query.py", line 484, in update_or_create
    obj, created = self._create_object_from_params(lookup, params, lock=True)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/query.py", line 509, in _create_object_from_params
    six.reraise(*exc_info)
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/utils/six.py", line 686, in reraise
    raise value
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/query.py", line 500, in _create_object_from_params
    obj = self.create(**params)
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/query.py", line 394, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/richard/github/kolibri/kolibri/core/auth/models.py", line 322, in save
    super(AbstractFacilityDataModel, self).save(*args, **kwargs)
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/morango/models/core.py", line 832, in save
    super(SyncableModel, self).save(*args, **kwargs)
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/morango/models/fields/uuids.py", line 125, in save
    super(UUIDModelMixin, self).save(*args, **kwargs)
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/base.py", line 807, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/base.py", line 924, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/base.py", line 962, in _do_insert
    return manager._insert([self], fields=fields, return_id=update_pk,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/query.py", line 1079, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1112, in execute_sql
    cursor.execute(sql, params)
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/backends/utils.py", line 60, in execute
    with self.db.wrap_database_errors:
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/richard/.virtualenvs/kolibripy3.11/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.IntegrityError: UNIQUE constraint failed: device_learnerdevicestatus.user_id

Expected behavior

Not sure why update_or_create is not handling this properly here, but we shouldn't get the integrity error here.

User-facing consequences

Device status won't get properly updated.

Steps to reproduce

Do a full facility import from a facility with a lot of lessons and quizzes, and with low disk space locally.

Context

Kolibri 0.16.1b0

rtibbles avatar Apr 02 '24 15:04 rtibbles

Hi @rtibbles! I have been trying to replicate the use case by mounting device with 20MB of free space and running kolibri with KOLIBRI_HOME="/mnt/test/.kolibri" yarn devserver, and doing a full import of a facility with 1k+ lessons and 4k+ quizzes. But I havent been able to replicate the InsufficientStorage exception, but instead I get a sqlite3.OperationalError: disk I/O error when the device runs out of space.

[python-devserver] DEBUG    2024-04-15 11:03:42,155 http://192.168.112.1:8080 "GET /api/morango/v1/buffers/?limit=200&offset=2400&transfer_session_id=318c8078009d4263b6403221f7b46f98 HTTP/1.1" 200 2739098
[python-devserver] DEBUG    2024-04-15 11:03:42,238 http://192.168.112.1:8080 "PATCH /api/morango/v1/transfersessions/318c8078009d4263b6403221f7b46f98/ HTTP/1.1" 200 604
[python-devserver] DEBUG    2024-04-15 11:03:42,240 [morango:pull] NetworkSessionContext -> NetworkPullTransferOperation = pending
[python-devserver] INFO     2024-04-15 11:03:42,240 Receiving data (2600/8034, 10.11MB)
[python-devserver] --- Logging error ---
[python-devserver] Traceback (most recent call last):
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
[python-devserver]     self.dialect.do_execute(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
[python-devserver]     cursor.execute(statement, parameters)
[python-devserver] sqlite3.OperationalError: disk I/O error
[python-devserver] 
[python-devserver] The above exception was the direct cause of the following exception:
[python-devserver] 
[python-devserver] Traceback (most recent call last):
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/syncsession.py", line 667, in run
[python-devserver]     self.proceed_to_and_wait_for(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/syncsession.py", line 628, in proceed_to_and_wait_for
[python-devserver]     result = self.controller.proceed_to_and_wait_for(stage, callback=callback)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/controller.py", line 224, in proceed_to_and_wait_for
[python-devserver]     callback()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 192, in fire
[python-devserver]     handler(**fire_kwargs)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 192, in fire
[python-devserver]     handler(**fire_kwargs)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/utils.py", line 700, in handler
[python-devserver]     self.update_progress(
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/utils.py", line 314, in update_progress
[python-devserver]     self.job.update_progress(new_progress, self.job.total_progress)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/job.py", line 258, in update_progress
[python-devserver]     self.storage.update_job_progress(self.job_id, progress, total_progress)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 506, in update_job_progress
[python-devserver]     self._update_job(job_id, progress=progress, total_progress=total_progress)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 657, in _update_job
[python-devserver]     job, orm_job = self._get_job_and_orm_job(job_id, session)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 693, in _get_job_and_orm_job
[python-devserver]     orm_job = session.query(ORMJob).filter_by(id=job_id).one_or_none()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2850, in one_or_none
[python-devserver]     return self._iter().one_or_none()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
[python-devserver]     result = self.session.execute(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
[python-devserver]     result = conn._execute_20(statement, params or {}, execution_options)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
[python-devserver]     return meth(self, args_10style, kwargs_10style, execution_options)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
[python-devserver]     return connection._execute_clauseelement(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
[python-devserver]     ret = self._execute_context(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
[python-devserver]     self._handle_dbapi_exception(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
[python-devserver]     util.raise_(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
[python-devserver]     raise exception
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
[python-devserver]     self.dialect.do_execute(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
[python-devserver]     cursor.execute(statement, parameters)
[python-devserver] sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error
[python-devserver] [SQL: SELECT jobs.id AS jobs_id, jobs.state AS jobs_state, jobs.func AS jobs_func, jobs.priority AS jobs_priority, jobs.queue AS jobs_queue, jobs.saved_job AS jobs_saved_job, jobs.time_created AS jobs_time_created, jobs.time_updated AS jobs_time_updated, jobs.interval AS jobs_interval, jobs.retry_interval AS jobs_retry_interval, jobs.repeat AS jobs_repeat, jobs.scheduled_time AS jobs_scheduled_time, jobs.worker_host AS jobs_worker_host, jobs.worker_process AS jobs_worker_process, jobs.worker_thread AS jobs_worker_thread, jobs.worker_extra AS jobs_worker_extra 
[python-devserver] FROM jobs 
[python-devserver] WHERE jobs.id = ?]
[python-devserver] [parameters: ('398a66d2fa024b139b48d705718d2f8f',)]
[python-devserver] (Background on this error at: https://sqlalche.me/e/14/e3q8)
[python-devserver] 
[python-devserver] During handling of the above exception, another exception occurred:
[python-devserver] 
[python-devserver] Traceback (most recent call last):
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
[python-devserver]     self.dialect.do_execute(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
[python-devserver]     cursor.execute(statement, parameters)
[python-devserver] sqlite3.OperationalError: disk I/O error
[python-devserver] 
[python-devserver] The above exception was the direct cause of the following exception:
[python-devserver] 
[python-devserver] Traceback (most recent call last):
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/job.py", line 329, in execute
[python-devserver]     result = func(*args, **kwargs)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/registry.py", line 238, in __call__
[python-devserver]     return self.func(*args, **kwargs)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/tasks.py", line 393, in peerfacilityimport
[python-devserver]     call_command(command, **kwargs)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/django/core/management/__init__.py", line 131, in call_command
[python-devserver]     return command.execute(*args, **defaults)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/django/core/management/base.py", line 330, in execute
[python-devserver]     output = self.handle(*args, **options)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/management/commands/base.py", line 28, in handle
[python-devserver]     return self.handle_async(*args, **options)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/commands/sync.py", line 180, in handle_async
[python-devserver]     self._sync(sync_session_client, **options)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/utils.py", line 437, in _sync
[python-devserver]     self._pull(
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/utils.py", line 567, in _pull
[python-devserver]     sync_client.run()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/syncsession.py", line 667, in run
[python-devserver]     self.proceed_to_and_wait_for(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 232, in __exit__
[python-devserver]     self.completed.fire()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 192, in fire
[python-devserver]     handler(**fire_kwargs)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 192, in fire
[python-devserver]     handler(**fire_kwargs)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/utils.py", line 700, in handler
[python-devserver]     self.update_progress(
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/utils.py", line 314, in update_progress
[python-devserver]     self.job.update_progress(new_progress, self.job.total_progress)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/job.py", line 258, in update_progress
[python-devserver]     self.storage.update_job_progress(self.job_id, progress, total_progress)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 506, in update_job_progress
[python-devserver]     self._update_job(job_id, progress=progress, total_progress=total_progress)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 657, in _update_job
[python-devserver]     job, orm_job = self._get_job_and_orm_job(job_id, session)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 693, in _get_job_and_orm_job
[python-devserver]     orm_job = session.query(ORMJob).filter_by(id=job_id).one_or_none()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2850, in one_or_none
[python-devserver]     return self._iter().one_or_none()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
[python-devserver]     result = self.session.execute(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
[python-devserver]     result = conn._execute_20(statement, params or {}, execution_options)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
[python-devserver]     return meth(self, args_10style, kwargs_10style, execution_options)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
[python-devserver]     return connection._execute_clauseelement(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
[python-devserver]     ret = self._execute_context(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
[python-devserver]     self._handle_dbapi_exception(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
[python-devserver]     util.raise_(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
[python-devserver]     raise exception
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
[python-devserver]     self.dialect.do_execute(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
[python-devserver]     cursor.execute(statement, parameters)
[python-devserver] sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error
[python-devserver] [SQL: SELECT jobs.id AS jobs_id, jobs.state AS jobs_state, jobs.func AS jobs_func, jobs.priority AS jobs_priority, jobs.queue AS jobs_queue, jobs.saved_job AS jobs_saved_job, jobs.time_created AS jobs_time_created, jobs.time_updated AS jobs_time_updated, jobs.interval AS jobs_interval, jobs.retry_interval AS jobs_retry_interval, jobs.repeat AS jobs_repeat, jobs.scheduled_time AS jobs_scheduled_time, jobs.worker_host AS jobs_worker_host, jobs.worker_process AS jobs_worker_process, jobs.worker_thread AS jobs_worker_thread, jobs.worker_extra AS jobs_worker_extra 
[python-devserver] FROM jobs 
[python-devserver] WHERE jobs.id = ?]
[python-devserver] [parameters: ('398a66d2fa024b139b48d705718d2f8f',)]
[python-devserver] (Background on this error at: https://sqlalche.me/e/14/e3q8)
[python-devserver] 
[python-devserver] During handling of the above exception, another exception occurred:
[python-devserver] 
[python-devserver] Traceback (most recent call last):
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/lib/python3.9/logging/__init__.py", line 1086, in emit
[python-devserver]     stream.write(msg + self.terminator)
[python-devserver] OSError: [Errno 28] No space left on device
[python-devserver] Call stack:
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/lib/python3.9/threading.py", line 937, in _bootstrap
[python-devserver]     self._bootstrap_inner()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/lib/python3.9/threading.py", line 980, in _bootstrap_inner
[python-devserver]     self.run()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/lib/python3.9/threading.py", line 917, in run
[python-devserver]     self._target(*self._args, **self._kwargs)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/lib/python3.9/concurrent/futures/thread.py", line 83, in _worker
[python-devserver]     work_item.run()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/lib/python3.9/concurrent/futures/thread.py", line 58, in run
[python-devserver]     result = self.fn(*self.args, **self.kwargs)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/worker.py", line 55, in execute_job_with_python_worker
[python-devserver]     execute_job(
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/worker.py", line 31, in execute_job
[python-devserver]     job.execute()
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/job.py", line 337, in execute
[python-devserver]     logger.error(
[python-devserver] Message: 'Job 398a66d2fa024b139b48d705718d2f8f raised an exception: Traceback (most recent call last):\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context\n    self.dialect.do_execute(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n    cursor.execute(statement, parameters)\nsqlite3.OperationalError: disk I/O error\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/syncsession.py", line 667, in run\n    self.proceed_to_and_wait_for(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/syncsession.py", line 628, in proceed_to_and_wait_for\n    result = self.controller.proceed_to_and_wait_for(stage, callback=callback)\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/controller.py", line 224, in proceed_to_and_wait_for\n    callback()\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 192, in fire\n    handler(**fire_kwargs)\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 192, in fire\n    handler(**fire_kwargs)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/utils.py", line 700, in handler\n    self.update_progress(\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/utils.py", line 314, in update_progress\n    self.job.update_progress(new_progress, self.job.total_progress)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/job.py", line 258, in update_progress\n    self.storage.update_job_progress(self.job_id, progress, total_progress)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 506, in update_job_progress\n    self._update_job(job_id, progress=progress, total_progress=total_progress)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 657, in _update_job\n    job, orm_job = self._get_job_and_orm_job(job_id, session)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 693, in _get_job_and_orm_job\n    orm_job = session.query(ORMJob).filter_by(id=job_id).one_or_none()\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2850, in one_or_none\n    return self._iter().one_or_none()\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n    result = self.session.execute(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1717, in execute\n    result = conn._execute_20(statement, params or {}, execution_options)\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20\n    return meth(self, args_10style, kwargs_10style, execution_options)\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n    return connection._execute_clauseelement(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement\n    ret = self._execute_context(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context\n    self._handle_dbapi_exception(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception\n    util.raise_(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n    raise exception\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context\n    self.dialect.do_execute(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n    cursor.execute(statement, parameters)\nsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error\n[SQL: SELECT jobs.id AS jobs_id, jobs.state AS jobs_state, jobs.func AS jobs_func, jobs.priority AS jobs_priority, jobs.queue AS jobs_queue, jobs.saved_job AS jobs_saved_job, jobs.time_created AS jobs_time_created, jobs.time_updated AS jobs_time_updated, jobs.interval AS jobs_interval, jobs.retry_interval AS jobs_retry_interval, jobs.repeat AS jobs_repeat, jobs.scheduled_time AS jobs_scheduled_time, jobs.worker_host AS jobs_worker_host, jobs.worker_process AS jobs_worker_process, jobs.worker_thread AS jobs_worker_thread, jobs.worker_extra AS jobs_worker_extra \nFROM jobs \nWHERE jobs.id = ?]\n[parameters: (\'398a66d2fa024b139b48d705718d2f8f\',)]\n(Background on this error at: https://sqlalche.me/e/14/e3q8)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context\n    self.dialect.do_execute(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n    cursor.execute(statement, parameters)\nsqlite3.OperationalError: disk I/O error\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/job.py", line 329, in execute\n    result = func(*args, **kwargs)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/registry.py", line 238, in __call__\n    return self.func(*args, **kwargs)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/tasks.py", line 393, in peerfacilityimport\n    call_command(command, **kwargs)\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/django/core/management/__init__.py", line 131, in call_command\n    return command.execute(*args, **defaults)\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/django/core/management/base.py", line 330, in execute\n    output = self.handle(*args, **options)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/management/commands/base.py", line 28, in handle\n    return self.handle_async(*args, **options)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/commands/sync.py", line 180, in handle_async\n    self._sync(sync_session_client, **options)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/utils.py", line 437, in _sync\n    self._pull(\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/utils.py", line 567, in _pull\n    sync_client.run()\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/syncsession.py", line 667, in run\n    self.proceed_to_and_wait_for(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 232, in __exit__\n    self.completed.fire()\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 192, in fire\n    handler(**fire_kwargs)\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 192, in fire\n    handler(**fire_kwargs)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/utils.py", line 700, in handler\n    self.update_progress(\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/utils.py", line 314, in update_progress\n    self.job.update_progress(new_progress, self.job.total_progress)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/job.py", line 258, in update_progress\n    self.storage.update_job_progress(self.job_id, progress, total_progress)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 506, in update_job_progress\n    self._update_job(job_id, progress=progress, total_progress=total_progress)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 657, in _update_job\n    job, orm_job = self._get_job_and_orm_job(job_id, session)\n  File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 693, in _get_job_and_orm_job\n    orm_job = session.query(ORMJob).filter_by(id=job_id).one_or_none()\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2850, in one_or_none\n    return self._iter().one_or_none()\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter\n    result = self.session.execute(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1717, in execute\n    result = conn._execute_20(statement, params or {}, execution_options)\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20\n    return meth(self, args_10style, kwargs_10style, execution_options)\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection\n    return connection._execute_clauseelement(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement\n    ret = self._execute_context(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context\n    self._handle_dbapi_exception(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception\n    util.raise_(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_\n    raise exception\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context\n    self.dialect.do_execute(\n  File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute\n    cursor.execute(statement, parameters)\nsqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error\n[SQL: SELECT jobs.id AS jobs_id, jobs.state AS jobs_state, jobs.func AS jobs_func, jobs.priority AS jobs_priority, jobs.queue AS jobs_queue, jobs.saved_job AS jobs_saved_job, jobs.time_created AS jobs_time_created, jobs.time_updated AS jobs_time_updated, jobs.interval AS jobs_interval, jobs.retry_interval AS jobs_retry_interval, jobs.repeat AS jobs_repeat, jobs.scheduled_time AS jobs_scheduled_time, jobs.worker_host AS jobs_worker_host, jobs.worker_process AS jobs_worker_process, jobs.worker_thread AS jobs_worker_thread, jobs.worker_extra AS jobs_worker_extra \nFROM jobs \nWHERE jobs.id = ?]\n[parameters: (\'398a66d2fa024b139b48d705718d2f8f\',)]\n(Background on this error at: https://sqlalche.me/e/14/e3q8)\n'
[python-devserver] Arguments: ()
[python-devserver] ERROR    2024-04-15 11:03:42,247 Job 398a66d2fa024b139b48d705718d2f8f raised an exception: Traceback (most recent call last):
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
[python-devserver]     self.dialect.do_execute(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
[python-devserver]     cursor.execute(statement, parameters)
[python-devserver] sqlite3.OperationalError: disk I/O error
[python-devserver] 
[python-devserver] The above exception was the direct cause of the following exception:
[python-devserver] 
[python-devserver] Traceback (most recent call last):
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/syncsession.py", line 667, in run
[python-devserver]     self.proceed_to_and_wait_for(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/syncsession.py", line 628, in proceed_to_and_wait_for
[python-devserver]     result = self.controller.proceed_to_and_wait_for(stage, callback=callback)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/controller.py", line 224, in proceed_to_and_wait_for
[python-devserver]     callback()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 192, in fire
[python-devserver]     handler(**fire_kwargs)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/morango/sync/utils.py", line 192, in fire
[python-devserver]     handler(**fire_kwargs)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/auth/management/utils.py", line 700, in handler
[python-devserver]     self.update_progress(
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/utils.py", line 314, in update_progress
[python-devserver]     self.job.update_progress(new_progress, self.job.total_progress)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/job.py", line 258, in update_progress
[python-devserver]     self.storage.update_job_progress(self.job_id, progress, total_progress)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 506, in update_job_progress
[python-devserver]     self._update_job(job_id, progress=progress, total_progress=total_progress)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 657, in _update_job
[python-devserver]     job, orm_job = self._get_job_and_orm_job(job_id, session)
[python-devserver]   File "/home/alexvelezll/Documents/jobs/learningequality/repos/kolibri/kolibri/core/tasks/storage.py", line 693, in _get_job_and_orm_job
[python-devserver]     orm_job = session.query(ORMJob).filter_by(id=job_id).one_or_none()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2850, in one_or_none
[python-devserver]     return self._iter().one_or_none()
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
[python-devserver]     result = self.session.execute(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
[python-devserver]     result = conn._execute_20(statement, params or {}, execution_options)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
[python-devserver]     return meth(self, args_10style, kwargs_10style, execution_options)
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
[python-devserver]     return connection._execute_clauseelement(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
[python-devserver]     ret = self._execute_context(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
[python-devserver]     self._handle_dbapi_exception(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
[python-devserver]     util.raise_(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
[python-devserver]     raise exception
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
[python-devserver]     self.dialect.do_execute(
[python-devserver]   File "/home/alexvelezll/.pyenv/versions/3.9.18/envs/kolibri-py3.9/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
[python-devserver]     cursor.execute(statement, parameters)
[python-devserver] sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) disk I/O error
[python-devserver] [SQL: SELECT jobs.id AS jobs_id, jobs.state AS jobs_state, jobs.func AS jobs_func, jobs.priority AS jobs_priority, jobs.queue AS jobs_queue, jobs.saved_job AS jobs_saved_job, jobs.time_created AS jobs_time_created, jobs.time_updated AS jobs_time_updated, jobs.interval AS jobs_interval, jobs.retry_interval AS jobs_retry_interval, jobs.repeat AS jobs_repeat, jobs.scheduled_time AS jobs_scheduled_time, jobs.worker_host AS jobs_worker_host, jobs.worker_process AS jobs_worker_process, jobs.worker_thread AS jobs_worker_thread, jobs.worker_extra AS jobs_worker_extra 
[python-devserver] FROM jobs 
[python-devserver] WHERE jobs.id = ?]
[python-devserver] [parameters: ('398a66d2fa024b139b48d705718d2f8f',)]
[python-devserver] (Background on this error at: https://sqlalche.me/e/14/e3q8)
[python-devserver] 
[python-devserver] During handling of the above exception, another exception occurred:
....

Meanwhile, the import progress just stops without any error message:

image

AlexVelezLl avatar Apr 15 '24 18:04 AlexVelezLl

Hi @AlexVelezLl - to avoid the issues you're seeing it might be simpler to artificially restrict the space available by setting this option: https://github.com/learningequality/kolibri/blob/develop/kolibri/utils/options.py#L659

If you set it to close to (or even exceeding) your regular drive's free space, it will restrict the space available for imports, but not impact the space available for the SQLite DB.

To update this, you can either edit the options.ini file in your KOLIBRI_HOME folder, or you can set it temporarily using the KOLIBRI_MINIMUM_DISK_SPACE environment variable. It accepts B, KB, MB, GB, TB, and PB - as it uses the logic in this module: https://github.com/learningequality/kolibri/blob/develop/kolibri/utils/data.py#L4

rtibbles avatar Apr 15 '24 18:04 rtibbles

@rtibbles The only way to run into this is that we have the same user with different Morango instanceID. That would be why update_or_create does not find a record, because the pair of (instance_id, user_id) would not be the same, and when trying to write the record, we would be writing a record with a repeated user_id, and that is why it would be violating the UNIQUE constraint of the user_id field.

Could this happen when doing a full import of a facility that already had a record of the admin device in the device_learnerdevicestatus table with the morango instanceID of one device, and then when trying to import to the other device?, the user id would be kept, but not the morango instanceID. I don't have two different devices to test this case, but I have manually changed the instanceId of morango and this way I have managed to replicate the error.

In any case, I think the solution could be to change the UNIQUE constraint to have both properties (instance_id, user_id), or remove the UNIQUE constraint from user_id in the database and have this constraint handled from the application layer.

AlexVelezLl avatar Apr 16 '24 15:04 AlexVelezLl

OK, so as I understand it the issue is that the user field on the LearnerDeviceStatus model is set as OneToOneField: https://github.com/learningequality/kolibri/blob/release-v0.16.x/kolibri/core/device/models.py#L613

Which enforces the constraint on both sides of the relationship, so if we have a LearnerDeviceStatus for multiple devices for the same learner, then we will have an issue.

So it seems that what we actually need is for it to be a ForeignKeyField so that it is only enforced on one side.

However, this may have downstream effects on where this is referenced, so I think we would need to dig in more into where this is used to make sure this wouldn't cause additional issues (as it may have been referenced for lookups on the FacilityUser assuming it will only ever return a single status).

rtibbles avatar Apr 18 '24 23:04 rtibbles

The other thing I think we'd need to consider is the fact this is a syncable model, so I'd need to get a double check on how this might impact syncing!

rtibbles avatar Apr 18 '24 23:04 rtibbles

However, this may have downstream effects on where this is referenced, so I think we would need to dig in more into where this is used to make sure this wouldn't cause additional issues

I have been searching the code for references to the LearnerDeviceStatus model, and I have not found places where the query is done only with user_id, but with both (user_id, instance_id), or by dataset_id. Within the model, the pair (instance_id, user_id) is also always taken into account, for example to delete the learner_status. Even in tests, it is always filtered by both fields. Other than those places, I haven't found any others where we refer to the LearnerDeviceStatus model to query data.

And it seems that at one point an attempt was made to put a unique_together contraint on both fields https://github.com/learningequality/kolibri/blob/5eeeb75141a2215436e668237f0e8fa8c89594f0/kolibri/core/device/migrations/0018_add_learner_device_status.py#L67

AlexVelezLl avatar Apr 22 '24 15:04 AlexVelezLl

OK, in that case loosening the OneToOneField to a ForeignKeyField seems appropriate.

Alternatively, if the intention is only to have a single LearnerDeviceStatus model per user, then we should be doing a different update.

rtibbles avatar Apr 23 '24 20:04 rtibbles

So, it seems that we want to change the OneToOneField to a ForeignKeyField.

To handle the backwards compatibility issue, we will need to add some special sauce to ensure that we do not try to sync multiple LearnerDeviceStatuses for a single user to a Kolibri device with a version less than 0.16.2.

There are three cases we need to consider:

  1. a learner only device that has Kolibri less than 0.16.2 installed, syncing with a server that has 0.16.2 installed, but that has already synced to another learner only device where the same learner had been imported - so the server would have a LearnerDeviceStatus for a different LOD device that would then sync to the current LOD.

  2. a learner only device that has Kolibri 0.16.2 installed, but has learner device statuses for itself, and for at least one other learner only device (this could have happened by syncing to with the 0.16.2 server in 1), syncing with a server with Kolibri less than 0.16.2 installed.

  3. A full facility server with Kolibri 0.16.2 has multiple LearnerDeviceStatuses for the same learner, does a full facility sync with a Kolibri with a version less than 0.16.2

For each scenario we should do the following:

  1. Subclass https://github.com/learningequality/kolibri/blob/develop/kolibri/core/auth/sync_operations.py#L154 in the downgrade method check if a single user sync, and if so delete LearnerDeviceStatus objects for the user being synced that are not for the instance_id of the remote that is syncing.
  2. Subclass https://github.com/learningequality/kolibri/blob/develop/kolibri/core/auth/sync_operations.py#L154 in the downgrade method check if a single user sync, and if so delete LearnerDeviceStatus objects for the user being synced that are not for the instance_id of the local that is syncing.
  3. Subclass https://github.com/learningequality/kolibri/blob/develop/kolibri/core/auth/sync_operations.py#L154 in the downgrade method check if a single user sync, and if so delete LearnerDeviceStatus objects for the every user in the facility being synced that are not for the instance_id of the remote that is syncing.

I think all 3 can be implemented in the same subclass with some conditional logic.

For an example of how to register the operation to be used, see here: https://github.com/learningequality/kolibri/blob/develop/kolibri/core/logger/kolibri_plugin.py#L40

Would need to add register a new FacilityDataSyncHook subclass in here: https://github.com/learningequality/kolibri/blob/develop/kolibri/core/device/kolibri_plugin.py and set the operations as initializing_operations

rtibbles avatar Apr 25 '24 20:04 rtibbles

Closed with #12153

marcellamaki avatar May 15 '24 20:05 marcellamaki