No such file or directory: '/tmp/celery_once/ using file based backend

[Cally99]

My tasks crash and do not run anymore due to a no such file error. I'm using the file based backend and this is my set up.

settings.py

CELERY_ONCE = { 'backend': 'celery_once.backends.File', 'settings': { 'location': '/tmp/celery_once/', 'default_timeout': 60 * 60 } }

CELERY_BROKER_URL = 'pyamqp://rabbitmq:5672' CELERY_RESULT_BACKEND = 'django-db'

CELERYD_HIJACK_ROOT_LOGGER = False

use json format for everything

CELERY_ACCEPT_CONTENT = ['json'] CELERY_TASK_SERIALIZER = 'json' CELERY_TIMEZONE = 'UTC' CELERYBEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'

CELERY_ONCE = { 'backend': 'celery_once.backends.File', 'settings': { 'location': '/tmp/celery_once/', 'default_timeout': 60 * 60 } }

celery.py

from future import absolute_import import os

from celery import Celery from django.conf import settings

set the default Django settings module for the 'celery' program.

all = [ 'celery', 'QueueOnce', ] os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'autobets.settings') os.environ.setdefault('DJANGO_CONFIGURATION', 'Development') import configurations

configurations.setup()

app = Celery('autobets')

Using a string here means the worker will not have to

pickle the object when using Windows.

app.config_from_object('django.conf:settings', namespace='CELERY') app.conf.ONCE = settings.CELERY_ONCE app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

tasks.py

@shared_task(bind=True,base=QueueOnce, once={'graceful': True}) def get_events(self): do stuff bla bla bla

Before using celeryonce tasks would run as normal.

Do I need to create a tmp backend file first or does celery once create this?

stack trace.

[2020-02-13 20:11:43,505: INFO/MainProcess] mingle: searching for neighbors
[2020-02-13 20:11:44,810: INFO/MainProcess] mingle: all alone
[2020-02-13 20:11:44,865: WARNING/MainProcess] /usr/local/lib/python3.6/site-packages/celery/fixups/django.py:200: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2020-02-13 20:11:44,866: INFO/MainProcess] celery@9f2b7c337579 ready.
[2020-02-13 20:11:44,868: INFO/MainProcess] Received task: apimb.tasks.get_events[b423ad7e-cf2b-4137-8f2d-25bb934dc148]  
[2020-02-13 20:11:44,872: INFO/MainProcess] Received task: apimb.tasks.get_balance[6af923d7-f1a1-451a-bc8d-e4f07f3b2f06]  
[2020-02-13 20:11:44,876: INFO/MainProcess] Received task: apimb.tasks.get_orders[3c30ddee-22a2-4831-b256-0511ed1b8cc6]  
[2020-02-13 20:11:45,416: WARNING/ForkPoolWorker-1] login has been calledAPIClient
[2020-02-13 20:11:45,502: WARNING/ForkPoolWorker-1] GET Balance task has been called now
[2020-02-13 20:11:45,586: WARNING/ForkPoolWorker-2] login has been calledAPIClient
[2020-02-13 20:11:45,637: INFO/ForkPoolWorker-1] Task apimb.tasks.get_balance[6af923d7-f1a1-451a-bc8d-e4f07f3b2f06] succeeded in 0.6551041000020632s: None
[2020-02-13 20:11:45,652: WARNING/ForkPoolWorker-2] GET EVENTS task has been called now
[2020-02-13 20:11:45,706: WARNING/ForkPoolWorker-1] /usr/local/lib/python3.6/site-packages/celery/app/trace.py:561: RuntimeWarning: Exception raised outside body: FileNotFoundError(2, 'No such file or directory'):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_52094e3196ca05f36eb013e2b749ce03'

  exc, exc_info.traceback)))
[2020-02-13 20:11:45,750: ERROR/ForkPoolWorker-1] Task apimb.tasks.get_balance[6af923d7-f1a1-451a-bc8d-e4f07f3b2f06] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_52094e3196ca05f36eb013e2b749ce03'
[2020-02-13 20:11:45,813: WARNING/ForkPoolWorker-1] GET ORDERS task has been called now
[2020-02-13 20:11:45,860: INFO/ForkPoolWorker-1] Task apimb.tasks.get_orders[3c30ddee-22a2-4831-b256-0511ed1b8cc6] succeeded in 0.10413699999844539s: None
[2020-02-13 20:11:45,875: WARNING/ForkPoolWorker-1] /usr/local/lib/python3.6/site-packages/celery/app/trace.py:561: RuntimeWarning: Exception raised outside body: FileNotFoundError(2, 'No such file or directory'):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_55d76a4bebdad047aea38a8fd7e07c8e'

  exc, exc_info.traceback)))
[2020-02-13 20:11:45,937: ERROR/ForkPoolWorker-1] Task apimb.tasks.get_orders[3c30ddee-22a2-4831-b256-0511ed1b8cc6] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_55d76a4bebdad047aea38a8fd7e07c8e'
[2020-02-13 20:11:48,857: INFO/ForkPoolWorker-2] Task apimb.tasks.get_events[b423ad7e-cf2b-4137-8f2d-25bb934dc148] succeeded in 3.861357500001759s: None
[2020-02-13 20:11:48,878: WARNING/ForkPoolWorker-2] /usr/local/lib/python3.6/site-packages/celery/app/trace.py:561: RuntimeWarning: Exception raised outside body: FileNotFoundError(2, 'No such file or directory'):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_f98116d19c287c5287360c9ef915d089'

  exc, exc_info.traceback)))
[2020-02-13 20:11:48,910: ERROR/ForkPoolWorker-2] Task apimb.tasks.get_events[b423ad7e-cf2b-4137-8f2d-25bb934dc148] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 465, in trace_task
    state, retval, uuid, args, kwargs, None,
  File "/usr/local/lib/python3.6/site-packages/celery_once/tasks.py", line 141, in after_return
    self.once_backend.clear_lock(key)
  File "/usr/local/lib/python3.6/site-packages/celery_once/backends/file.py", line 76, in clear_lock
    os.remove(lock_path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/celery_once/qo_apimb.tasks.ge_f98116d19c287c5287360c9ef915d089'```

[Cally99]

As per documentation there default path location does not work. I created a file myself and copied the relative path to 'location' :'backend/project/celery_once'

Tasks are firing now:)

Please check if documentation needs to be updated or default tmp file is not being created. By the way I'm using docker

[xuhcc]

celery-once should create all necessary parent directories and the lock file itself automatically. Yes, it may fail to do that due to permission problem or something else, but I think it's not what is happening in your case. The No such file or directory error in your log is thrown when celery-once tries to remove the lock file after the completion of the task. It's like the lock file has been removed already, and celery-once tries to remove it again.

[cameronmaske]

@Cally99 Your /tmp/ will be cleared each time you start/stop the container. Hence, the locks will not be ensured as the directory, they are stored in does not persist between worker restarts.

I'm not sure if the FileNotFoundError is the best error to raise here, but there should be some sort of warning (or exception) to the user to indicate the lock has failed.

[xuhcc]

@cameronmaske But according to the provided traceback, the FileNotFoundError was thrown on the lock removal. Do you have any idea why this might happen? I'm seeing this in my setup too, albeit very rarely.

[cameronmaske]

My theory is that it's due to the task still being in the broker (i.e. unprocessed) but the lock file is removed before the task is run. The lock is created before the task is sent into the broker.

If we break down the timeline, the following happens in order:

• Schedule task.
• Lock file is checked, and created if needs be. If no lock, the task is now in brokers (i.e. RabbitMQ) waiting to be picked up.
• Container/server restarts, or /tmp/ is cleared some other way.
• Lock file no longer exists, but the task is still in the broker.
• Task get picked up by worker, taken out of the broker.
• Task runs, goes to clean up lock file, but it no longer exists. Raises FileNotFoundError.

[xuhcc]

Yeah, that makes sense. Redis ignores the missing keys when DEL command is executed. So I think we should do the same with the file backend.

[eyeteajay]

I have a nearly identical setup and also getting the same traceback. I think @cameronmaske 's diagnosis is right on.