taiga-back Celery service crashes

Celery service crashes

Open paulkernstock opened this issue 6 years ago • 0 comments

I'm doing a fresh install of Taiga which is running now except the celery service which is constantly restarting.

What is the current behavior? The service crashes without information of what went wrong, even on DEBUG loglevel:

Jan 08 15:57:11 shg-server celery[24632]: Trying import local.py settings...
Jan 08 15:57:12 shg-server celery[24632]:  
Jan 08 15:57:12 shg-server celery[24632]:  -------------- celery@shg-server v4.0.2 (latentcall)
Jan 08 15:57:12 shg-server celery[24632]: ---- **** -----
Jan 08 15:57:12 shg-server celery[24632]: --- * ***  * -- Linux-4.15.0-43-generic-x86_64-with-Ubuntu-18.04-bionic 2019-01-08 14:57:11
Jan 08 15:57:12 shg-server celery[24632]: -- * - **** ---
Jan 08 15:57:12 shg-server celery[24632]: - ** ---------- [config]
Jan 08 15:57:12 shg-server celery[24632]: - ** ---------- .> app:         taiga:0x7fb4098b3358
Jan 08 15:57:12 shg-server celery[24632]: - ** ---------- .> transport:   amqp://guest:**@localhost:5672//
Jan 08 15:57:12 shg-server celery[24632]: - ** ---------- .> results:     redis://localhost:6379/0
Jan 08 15:57:12 shg-server celery[24632]: - *** --- * --- .> concurrency: 4 (prefork)
Jan 08 15:57:12 shg-server celery[24632]: -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
Jan 08 15:57:12 shg-server celery[24632]: --- ***** -----
Jan 08 15:57:12 shg-server celery[24632]:  -------------- [queues]
Jan 08 15:57:12 shg-server celery[24632]:                 .> tasks            exchange=(direct) key=task.#
Jan 08 15:57:12 shg-server celery[24632]:                 .> transient        exchange=(direct) key=transient.#
Jan 08 15:57:12 shg-server celery[24632]: [tasks]
Jan 08 15:57:12 shg-server celery[24632]:   . celery.accumulate
Jan 08 15:57:12 shg-server celery[24632]:   . celery.backend_cleanup
Jan 08 15:57:12 shg-server celery[24632]:   . celery.chain
Jan 08 15:57:12 shg-server celery[24632]:   . celery.chord
Jan 08 15:57:12 shg-server celery[24632]:   . celery.chord_unlock
Jan 08 15:57:12 shg-server celery[24632]:   . celery.chunks
Jan 08 15:57:12 shg-server celery[24632]:   . celery.group
Jan 08 15:57:12 shg-server celery[24632]:   . celery.map
Jan 08 15:57:12 shg-server celery[24632]:   . celery.starmap
Jan 08 15:57:12 shg-server celery[24632]:   . djmail.tasks.retry_send_messages
Jan 08 15:57:12 shg-server celery[24632]:   . djmail.tasks.send_messages
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.export_import.tasks.delete_project_dump
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.export_import.tasks.dump_project
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.export_import.tasks.load_project_dump
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.projects.contact.services.send_contact_email
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.projects.services.projects.delete_project
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.timeline.service.push_to_timelines
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.webhooks.tasks.change_webhook
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.webhooks.tasks.create_webhook
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.webhooks.tasks.delete_webhook
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.webhooks.tasks.resend_webhook
Jan 08 15:57:12 shg-server celery[24632]:   . taiga.webhooks.tasks.test_webhook
Jan 08 15:57:12 shg-server celery[24632]: [2019-01-08 14:57:12,185: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
Jan 08 15:57:12 shg-server celery[24632]: [2019-01-08 14:57:12,193: INFO/MainProcess] mingle: searching for neighbors
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,211: INFO/MainProcess] mingle: all alone
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,229: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[53758dd4-de8b-42b1-93c7-7434d4ee0b1e]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,230: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fb4097e2e18> (args:('taiga.timeline.service.push_to_timelines', '53758dd4-de8b-42b1-93c7-7434d4ee0b1e', {'lang': 'py', 'task': 'taiga.timeline.service.push_to_timelines', 'id': '53758dd4-de8b-42b1-93c7-7434d4ee0b1e', 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '53758dd4-de8b-42b1-93c7-7434d4ee0b1e', 'parent_id': None, 'argsrepr': "(8, 32, 'wiki', 'wikipage', 60, 'change', datetime.datetime(2019, 1, 8, 9, 39, 1, 656612, tzinfo=<UTC>))", 'kwargsrepr': "{'extra_data': {'values_diff': {'content_diff': (...), 'content_html': (...)}, 'user': {'id': 32}, 'comment': '', 'comment_html': ''}, 'refresh_totals': True}", 'origin': 'gen4338@shg-server', 'reply_to': 'e8383f37-1c30-39d3-ba08-90cb238b1ad5', 'correlation_id': '53758dd4-de8b-42b1-93c7-7434d4ee0b1e', 'delivery_info': {'exchange': '', 'routing_key': 'tasks', 'priority': 0, 'redelivered': True}}, b'\x80\x02(K\x08K... kwargs:{})
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,232: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[97f1aab2-4fde-41e7-8251-c4e38b53fd52]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,232: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fb4097e2e18> (args:('taiga.timeline.service.push_to_timelines', '97f1aab2-4fde-41e7-8251-c4e38b53fd52', {'lang': 'py', 'task': 'taiga.timeline.service.push_to_timelines', 'id': '97f1aab2-4fde-41e7-8251-c4e38b53fd52', 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '97f1aab2-4fde-41e7-8251-c4e38b53fd52', 'parent_id': None, 'argsrepr': "(8, 32, 'wiki', 'wikipage', 60, 'change', datetime.datetime(2019, 1, 8, 9, 39, 33, 170124, tzinfo=<UTC>))", 'kwargsrepr': "{'extra_data': {'values_diff': {'content_diff': (...), 'content_html': (...)}, 'user': {'id': 32}, 'comment': '', 'comment_html': ''}, 'refresh_totals': True}", 'origin': 'gen4345@shg-server', 'reply_to': 'eae2f2ec-bb89-3ab1-898a-ef18d5ad4d11', 'correlation_id': '97f1aab2-4fde-41e7-8251-c4e38b53fd52', 'delivery_info': {'exchange': '', 'routing_key': 'tasks', 'priority': 0, 'redelivered': True}}, b'\x80\x02(K\x08K... kwargs:{})
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,233: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[7c7c0447-d6e2-4b7e-b4ce-480e5b135293]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,233: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fb4097e2e18> (args:('taiga.timeline.service.push_to_timelines', '7c7c0447-d6e2-4b7e-b4ce-480e5b135293', {'lang': 'py', 'task': 'taiga.timeline.service.push_to_timelines', 'id': '7c7c0447-d6e2-4b7e-b4ce-480e5b135293', 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '7c7c0447-d6e2-4b7e-b4ce-480e5b135293', 'parent_id': None, 'argsrepr': "(8, 32, 'wiki', 'wikipage', 60, 'change', datetime.datetime(2019, 1, 8, 9, 44, 29, 307821, tzinfo=<UTC>))", 'kwargsrepr': "{'extra_data': {'values_diff': {'content_diff': (...), 'content_html': (...)}, 'user': {'id': 32}, 'comment': '', 'comment_html': ''}, 'refresh_totals': True}", 'origin': 'gen4338@shg-server', 'reply_to': 'e8383f37-1c30-39d3-ba08-90cb238b1ad5', 'correlation_id': '7c7c0447-d6e2-4b7e-b4ce-480e5b135293', 'delivery_info': {'exchange': '', 'routing_key': 'tasks', 'priority': 0, 'redelivered': True}}, b'\x80\x02(K\x08K... kwargs:{})
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,233: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[8b98dca8-c729-4c74-91b4-c4e35e7ce61b]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,233: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fb4097e2e18> (args:('taiga.timeline.service.push_to_timelines', '8b98dca8-c729-4c74-91b4-c4e35e7ce61b', {'lang': 'py', 'task': 'taiga.timeline.service.push_to_timelines', 'id': '8b98dca8-c729-4c74-91b4-c4e35e7ce61b', 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '8b98dca8-c729-4c74-91b4-c4e35e7ce61b', 'parent_id': None, 'argsrepr': "(8, 32, 'wiki', 'wikipage', 68, 'change', datetime.datetime(2019, 1, 8, 9, 45, 2, 809305, tzinfo=<UTC>))", 'kwargsrepr': "{'extra_data': {'values_diff': {'content_diff': (...), 'content_html': (...)}, 'user': {'id': 32}, 'comment': '', 'comment_html': ''}, 'refresh_totals': True}", 'origin': 'gen4341@shg-server', 'reply_to': 'b6124899-97e8-3e7d-afa1-3c23682a3dac', 'correlation_id': '8b98dca8-c729-4c74-91b4-c4e35e7ce61b', 'delivery_info': {'exchange': '', 'routing_key': 'tasks', 'priority': 0, 'redelivered': True}}, b'\x80\x02(K\x08K... kwargs:{})
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,234: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[1dde8358-44a7-4931-928e-b79ffef448c9]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,234: DEBUG/MainProcess] TaskPool: Apply <function _fast_trace_task at 0x7fb4097e2e18> (args:('taiga.timeline.service.push_to_timelines', '1dde8358-44a7-4931-928e-b79ffef448c9', {'lang': 'py', 'task': 'taiga.timeline.service.push_to_timelines', 'id': '1dde8358-44a7-4931-928e-b79ffef448c9', 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '1dde8358-44a7-4931-928e-b79ffef448c9', 'parent_id': None, 'argsrepr': "(8, 32, 'wiki', 'wikipage', 68, 'change', datetime.datetime(2019, 1, 8, 9, 45, 15, 724292, tzinfo=<UTC>))", 'kwargsrepr': "{'extra_data': {'values_diff': {'content_diff': (...), 'content_html': (...)}, 'user': {'id': 32}, 'comment': '', 'comment_html': ''}, 'refresh_totals': True}", 'origin': 'gen4341@shg-server', 'reply_to': 'b6124899-97e8-3e7d-afa1-3c23682a3dac', 'correlation_id': '1dde8358-44a7-4931-928e-b79ffef448c9', 'delivery_info': {'exchange': '', 'routing_key': 'tasks', 'priority': 0, 'redelivered': True}}, b'\x80\x02(K\x08K... kwargs:{})
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,234: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[6d49804c-f747-4216-9b22-6f79abd7d929]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,235: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[1700f3e7-0858-4740-a3e6-afb7df50d0af]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,235: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[a4298efc-1c0c-4c01-bb06-042d8f6adecf]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,235: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[c7c7f45a-961e-4926-b437-69aa0a084144]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,235: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[2c055779-1a3b-43e6-a8bf-66f6563d109d]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,236: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[2728498a-dff9-411a-acb9-79912f9c8663]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,236: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[8f738557-822a-49c8-8b32-f801f6d9288a]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,236: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[5da4437e-420e-4c86-ae5c-22f2dd58a636]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,236: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[a04568ca-4959-41e0-bc2f-121b92345eb2]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,237: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[3046cd4c-85a1-4f3c-82b3-1ebfe059f1de]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,237: INFO/MainProcess] Received task: taiga.timeline.service.push_to_timelines[79f4777f-91f9-4dc5-98b4-3d7eee7887fc]
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,237: DEBUG/MainProcess] Canceling task consumer...
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,241: DEBUG/MainProcess] Canceling task consumer...
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,241: DEBUG/MainProcess] Closing consumer channel...
Jan 08 15:57:13 shg-server celery[24632]: [2019-01-08 14:57:13,243: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
Jan 08 15:57:13 shg-server systemd[1]: taiga_celery.service: Main process exited, code=exited, status=1/FAILURE
Jan 08 15:57:13 shg-server systemd[1]: taiga_celery.service: Failed with result 'exit-code'.

There is no error being logged but the service exits with failure.

If the current behavior is a bug, please provide the steps to reproduce. local.py:

from .common import *

MEDIA_URL = "http://mydomain/media/"
STATIC_URL = "http://mydomain/static/"
ADMIN_MEDIA_PREFIX = "http://mydomain/static/admin/"
SITES["front"]["scheme"] = "http"
SITES["front"]["domain"] = "mydomain"
SITES["api"]["domain"] = "mydomain"

SECRET_KEY = "theveryultratopsecretkey"

DEBUG = False
TEMPLATE_DEBUG = False
PUBLIC_REGISTER_ENABLED = False

#Celery
CELERY_ENABLED = True
from .celery import *
broker_url = 'amqp://guest:guest@localhost:5672//'
result_backend = 'redis://localhost:6379/0'

EVENTS_PUSH_BACKEND = "taiga.events.backends.rabbitmq.EventsPushBackend"
EVENTS_PUSH_BACKEND_OPTIONS = {"url": "amqp://taiga:PASS@localhost:5672/taiga"}

# Uncomment and populate with proper connection parameters
# for enable github login/singin.
#GITHUB_API_CLIENT_ID = "yourgithubclientid"
#GITHUB_API_CLIENT_SECRET = "yourgithubclientsecret"

taiga_celery.service:

[Unit]
Description=taiga_celery
After=network.target

[Service]
User=taiga
Environment=PYTHONUNBUFFERED=true
WorkingDirectory=/home/taiga/taiga-back
ExecStart=/home/taiga/.virtualenvs/taiga/bin/celery -A taiga worker --concurrency 4 -l DEBUG
Restart=always
RestartSec=3
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=default.target

Is it happening in taiga.io or in your own instance? Own instance

What browser/version are you using? Firefox 64

Jan 08 '19 15:01 paulkernstock

taiga-back taiga-back copied to clipboard

Celery service crashes

taiga-back
taiga-back copied to clipboard