Refusing to deserialize untrusted content of type pickle

[monegim]

File "/usr/local/lib/python3.9/site-packages/kombu/messaging.py", line 620, in _receive_callback
decoded = None if on_m else message.decode()
File "/usr/local/lib/python3.9/site-packages/kombu/message.py", line 194, in decode
self._decoded_cache = self._decode()
File "/usr/local/lib/python3.9/site-packages/kombu/message.py", line 198, in _decode
return loads(self.body, self.content_type,
File "/usr/local/lib/python3.9/site-packages/kombu/serialization.py", line 242, in loads
raise self._for_untrusted_content(content_type, 'untrusted')
kombu.exceptions.ContentDisallowed: Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)

Deployed celery-exporter using helm chart.

[danihodovic]

Could you post your helm config?

[monegim]

Could you post your helm config?

Do you mean values.yaml?

[monegim]

replicaCount: 1

image:
  repository: danihodovic/celery-exporter
  pullPolicy: IfNotPresent
  tag: ""

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  create: true
  annotations: {}
  name: ""

env: []

podAnnotations: {}

podSecurityContext: {}

securityContext: {}

service:
  type: ClusterIP
  port: 9808

ingress:
  enabled: false
  className: ""
  annotations: {}
  hosts:
    - host: celery-exporter.example
      paths:
        - path: /
          pathType: ImplementationSpecific
  tls: []

serviceMonitor:
  enabled: true
  additionalLabels: {}
  namespace: ""
  namespaceSelector: {}
  scrapeInterval: 30s
  targetLabels: []
  relabelings: []
  metricRelabelings: []

resources: 
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

[danihodovic]

cc @adinhodovic

[adinhodovic]

@monegim What Chart version are you using?

[monegim]

@monegim What Chart version are you using?

version: 0.4.1
appVersion: 0.5.3

[adinhodovic]

@monegim What Chart version are you using?

version: 0.4.1
appVersion: 0.5.3

https://docs.celeryq.dev/en/3.1/configuration.html

Try to see if you override the CELERY_ACCEPT_CONTENT environment variable. By default it should accept pickle content or any content. echo $CELERY_ACCEPT_CONTENT, or printenv in the celery container - not the celery-exporter container.

[monegim]

This has been set to this: https://github.com/DefectDojo/django-DefectDojo/blob/master/dojo/settings/settings.dist.py#L991

CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

[adinhodovic]

This has been set to this: https://github.com/DefectDojo/django-DefectDojo/blob/master/dojo/settings/settings.dist.py#L991

CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

Try adding application/x-python-serialize to the list.

[monegim]

CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml', 'application/x-python-serialize']

This did not work. Is this issue related to the celery-exporter or celery?

[adinhodovic]

CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml', 'application/x-python-serialize']

This did not work. Is this issue related to the celery-exporter or celery?

I believe celery does not accept the content celery-exporter wants.

[monegim]

So, should not something change in celery-exporter?

[danihodovic]

Feel free to submit a PR

[adinhodovic]

So, should not something change in celery-exporter?

Celery-exporter runs fine with celery by default. There are some settings which you've added that disabled celery-exporter from fetching the data needed. So yeah either extend the exporter to support a different content format or change the settings for celery.

[kblcuk]

At least for me similar thing happens because of task_serializer = "msgpack" in celery config with accept content being accept_content = ["msgpack", "json"]. Note though that I'm just running celery-exporter as docker image, no helm charts.

INFO     | src.http_server:start_http_server:66 - Started celery-exporter at port='9808'
ERROR in app: Exception on /metrics [GET]
  Traceback (most recent call last):
    File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2073, in wsgi_app
      response = self.full_dispatch_request()
    File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1518, in full_dispatch_request
      rv = self.handle_user_exception(e)
    File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
      rv = self.dispatch_request()
    File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request
      return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
    File "/app/src/http_server.py", line 32, in metrics
      current_app.config["metrics_puller"]()
    File "/app/src/exporter.py", line 116, in track_queue_length
      queues = self.app.control.inspect().active_queues() or {}
    File "/usr/local/lib/python3.9/site-packages/celery/app/control.py", line 338, in active_queues
      return self._request('active_queues')
    File "/usr/local/lib/python3.9/site-packages/celery/app/control.py", line 106, in _request
      return self._prepare(self.app.control.broadcast(
    File "/usr/local/lib/python3.9/site-packages/celery/app/control.py", line 741, in broadcast
      return self.mailbox(conn)._broadcast(
    File "/usr/local/lib/python3.9/site-packages/kombu/pidbox.py", line 344, in _broadcast
      return self._collect(reply_ticket, limit=limit,
    File "/usr/local/lib/python3.9/site-packages/kombu/pidbox.py", line 386, in _collect
      self.connection.drain_events(timeout=timeout)
    File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 316, in drain_events
      return self.transport.drain_events(self.connection, **kwargs)
    File "/usr/local/lib/python3.9/site-packages/kombu/transport/virtual/base.py", line 971, in drain_events
      get(self._deliver, timeout=timeout)
    File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 584, in get
      ret = self.handle_event(fileno, event)
    File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 566, in handle_event
      return self.on_readable(fileno), self
    File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 562, in on_readable
      chan.handlers[type]()
    File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 967, in _brpop_read
      self.connection._deliver(loads(bytes_to_str(item)), dest)
    File "/usr/local/lib/python3.9/site-packages/kombu/transport/virtual/base.py", line 991, in _deliver
      callback(message)
    File "/usr/local/lib/python3.9/site-packages/kombu/transport/virtual/base.py", line 624, in _callback
      return callback(message)
    File "/usr/local/lib/python3.9/site-packages/kombu/messaging.py", line 620, in _receive_callback
      decoded = None if on_m else message.decode()
    File "/usr/local/lib/python3.9/site-packages/kombu/message.py", line 194, in decode
      self._decoded_cache = self._decode()
    File "/usr/local/lib/python3.9/site-packages/kombu/message.py", line 198, in _decode
      return loads(self.body, self.content_type,
    File "/usr/local/lib/python3.9/site-packages/kombu/serialization.py", line 242, in loads
      raise self._for_untrusted_content(content_type, 'untrusted')
  kombu.exceptions.ContentDisallowed: Refusing to deserialize untrusted content of type msgpack (application/x-msgpack)

[howardjones]

Hitting this too, while looking around for a prometheus option for celery. We use pickle encoding to allow us to pass dataclass objects to tasks (which the default json serializer can't handle). Does the exporter need to be given the same config object as the celery worker somehow? (with the task_serializer and accept_content set). My celery config object looks like this:

class CeleryConfig:
    task_serializer = 'pickle'
    result_serializer = 'pickle'
    event_serializer = 'pickle'
    result_accept_content = ['pickle']
    accept_content = ['application/json', 'application/x-python-serialize', 'pickle']
    worker_send_task_events = True
    task_send_sent_event = True

[howardjones]

(confirming - if you add your Celery config object in exporter.py, everything seems to work. Just need a way to do that without altering the code! :-) Not sure if everything we have can be done with just environment vars)

My ugly test case:

▶ git diff src/exporter.py                                    
diff --git a/src/exporter.py b/src/exporter.py
index b68a300..090096d 100644
--- a/src/exporter.py
+++ b/src/exporter.py
@@ -13,6 +13,15 @@ from prometheus_client import CollectorRegistry, Counter, Gauge, Histogram
 
 from .http_server import start_http_server
 
+class CeleryConfig:
+    task_serializer = 'pickle'
+    result_serializer = 'pickle'
+    event_serializer = 'json'
+    result_accept_content = ['pickle']
+    accept_content = ['application/json', 'application/x-python-serialize', 'pickle']
+    worker_send_task_events = True
+    task_send_sent_event = True
+
 
 class Exporter:  # pylint: disable=too-many-instance-attributes,too-many-branches
     state: State = None
@@ -199,6 +208,7 @@ class Exporter:  # pylint: disable=too-many-instance-attributes,too-many-branche
         logger.remove()
         logger.add(sys.stdout, level=click_params["log_level"])
         self.app = Celery(broker=click_params["broker_url"])
+        self.app.config_from_object(CeleryConfig)
         transport_options = {}
         for transport_option in click_params["broker_transport_option"]:
             if transport_option is not None:

[danihodovic]

Resolved by @amandahla in #186 . Run the container with CE_ACCEPT_CONTENT="json,pickle". Released in docker.io/danihodovic/celery-exporter:0.5.8