Unable to push stats

[prashanthellina]

I installed statsd python wrapper from here: https://pypi.python.org/pypi/statsd/2.1.2 and tried the following:

>>> import statsd
>>> c = statsd.StatsClient('localhost', 8125)
>>> c.incr('prashanth.test', 100)

In Grafana dashboard I tried to look for the metric but it wasn't showing up. I asked Docker to dump logs from the container and see this:

2014-05-01 12:30:04,083 CRIT Supervisor running as root (no user in config file)
2014-05-01 12:30:04,083 WARN Included extra file "/etc/supervisor/conf.d/supervisord.conf" during parsing
2014-05-01 12:30:04,099 INFO RPC interface 'supervisor' initialized
2014-05-01 12:30:04,099 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2014-05-01 12:30:04,099 INFO supervisord started with pid 1
2014-05-01 12:30:05,101 INFO spawned: 'nginx' with pid 8
2014-05-01 12:30:05,102 INFO spawned: 'carbon-cache' with pid 9
2014-05-01 12:30:05,103 INFO spawned: 'elasticsearch' with pid 10
2014-05-01 12:30:05,104 INFO spawned: 'graphite-webapp' with pid 11
2014-05-01 12:30:05,105 INFO spawned: 'statsd' with pid 12
2014-05-01 12:30:05,310 INFO exited: carbon-cache (exit status 1; not expected)
2014-05-01 12:30:06,541 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-05-01 12:30:06,542 INFO spawned: 'carbon-cache' with pid 58
2014-05-01 12:30:06,542 INFO success: elasticsearch entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-05-01 12:30:06,542 INFO success: graphite-webapp entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-05-01 12:30:06,542 INFO success: statsd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-05-01 12:30:06,664 INFO exited: carbon-cache (exit status 1; not expected)
2014-05-01 12:30:09,191 INFO spawned: 'carbon-cache' with pid 100
2014-05-01 12:30:09,302 INFO exited: carbon-cache (exit status 1; not expected)
2014-05-01 12:30:13,195 INFO spawned: 'carbon-cache' with pid 133
2014-05-01 12:30:13,307 INFO exited: carbon-cache (exit status 1; not expected)
2014-05-01 12:30:14,308 INFO gave up: carbon-cache entered FATAL state, too many start retries too quickly

Here is what Docker reports for the "top" command

CONTAINER ID        IMAGE                        COMMAND                CREATED             STATUS              PORTS                                                                NAMES
a6659900b509        kamon/grafana_graphite:1.0   /usr/bin/supervisord   2 days ago          Up 26 minutes       0.0.0.0:81->80/tcp, 0.0.0.0:8125->8125/udp, 0.0.0.0:8126->8126/tcp   kamon-grafana-dashboard   
prashanth@almatrix1:~$ sudo docker.io top a6659900b509
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                13726               2007                0                   12:30               ?                   00:00:00            /usr/bin/python /usr/bin/supervisord
root                13771               13726               0                   12:30               ?                   00:00:00            nginx: master process /usr/sbin/nginx
sshd                13773               13726               0                   12:30               ?                   00:00:09            /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.1.jar:/usr/share/elasticsearch/lib/*:/usr/share/elasticsearch/lib/sigar/* -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
root                13774               13726               0                   12:30               ?                   00:00:00            /usr/bin/python /usr/bin/gunicorn_django -b127.0.0.1:8000 -w2 graphite/settings.py
root                13775               13726               0                   12:30               ?                   00:00:00            statsd /src/statsd/config.js
www-data            13776               13771               0                   12:30               ?                   00:00:00            nginx: worker process
root                13816               13774               0                   12:30               ?                   00:00:00            /usr/bin/python /usr/bin/gunicorn_django -b127.0.0.1:8000 -w2 graphite/settings.py
root                13817               13774               0                   12:30               ?                   00:00:00            /usr/bin/python /usr/bin/gunicorn_django -b127.0.0.1:8000 -w2 graphite/settings.py

Does this mean that the Carbon process did indeed fail to start? How can I fix this?

[ivantopo]

I would suggest that you attach to the container and review the logs under the /var/log/supervisor folder, there should be a hint of what is going wrong there, let us know what you find there.

[prashanthellina]

I was in the process of trying this. Attaching to an existing container seems less than straightforward on Docker. I am still trying to figure out. Thanks.

[mohdaliiqbal]

I'm having similar issue - in my case graphite-webapp fails to start with following error in supervisord.log

    2016-01-30 09:46:36,713 INFO exited: graphite-webapp (exit status 1; not expected)
    2016-01-30 09:46:40,300 INFO spawned: 'graphite-webapp' with pid 54
    2016-01-30 09:46:40,536 INFO exited: graphite-webapp (exit status 1; not expected)
    2016-01-30 09:46:40,599 INFO gave up: graphite-webapp entered FATAL state, too many start retries too quickly

And following errors in graphite-webapp.log

    `ImportError: Could not import settings 'graphite/settings.py' (Is it on sys.path?): Import by filename is not supported.
    2016-01-30 09:46:40 [59] [INFO] Worker exiting (pid: 59)
    2016-01-30 09:46:40 [60] [ERROR] Exception in worker process:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 473, in spawn_worker
        worker.init_process()
      File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 100, in init_process
        self.wsgi = self.app.wsgi()
      File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 115, in wsgi
        self.callable = self.load()
      File "/usr/lib/python2.7/dist-packages/gunicorn/app/djangoapp.py", line 102, in load
        return mod.make_wsgi_application()
      File "/usr/lib/python2.7/dist-packages/gunicorn/app/django_wsgi.py", line 36, in make_wsgi_application
        if get_validation_errors(s):
      File "/usr/local/lib/python2.7/dist-packages/django/core/management/validation.py", line 28, in get_validation_errors
        from django.db import models, connection
      File "/usr/local/lib/python2.7/dist-packages/django/db/__init__.py", line 11, in <module>
        if settings.DATABASES and DEFAULT_DB_ALIAS not in settings.DATABASES:
      File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 52, in __getattr__
        self._setup(name)
      File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 47, in _setup
        self._wrapped = Settings(settings_module)
      File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 132, in __init__
        raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
    ImportError: Could not import settings 'graphite/settings.py' (Is it on sys.path?): Import by filename is not supported.
    Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 473, in spawn_worker
        worker.init_process()
      File "/usr/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 100, in init_process
        self.wsgi = self.app.wsgi()
      File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 115, in wsgi
        self.callable = self.load()
      File "/usr/lib/python2.7/dist-packages/gunicorn/app/djangoapp.py", line 102, in load
        return mod.make_wsgi_application()
      File "/usr/lib/python2.7/dist-packages/gunicorn/app/django_wsgi.py", line 36, in make_wsgi_application
        if get_validation_errors(s):
      File "/usr/local/lib/python2.7/dist-packages/django/core/management/validation.py", line 28, in get_validation_errors
        from django.db import models, connection
      File "/usr/local/lib/python2.7/dist-packages/django/db/__init__.py", line 11, in <module>
        if settings.DATABASES and DEFAULT_DB_ALIAS not in settings.DATABASES:
      File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 52, in __getattr__
        self._setup(name)
      File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 47, in _setup
        self._wrapped = Settings(settings_module)
      File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 132, in __init__
        raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
    ImportError: Could not import settings 'graphite/settings.py' (Is it on sys.path?): Import by filename is not supported.
    2016-01-30 09:46:40 [60] [INFO] Worker exiting (pid: 60)
    Traceback (most recent call last):
      File "/usr/bin/gunicorn_django", line 9, in <module>
        load_entry_point('gunicorn==17.5', 'console_scripts', 'gunicorn_django')()
      File "/usr/lib/python2.7/dist-packages/gunicorn/app/djangoapp.py", line 143, in run
        DjangoApplication("%(prog)s [OPTIONS] [SETTINGS_PATH]").run()
      File "/usr/lib/python2.7/dist-packages/gunicorn/app/base.py", line 144, in run
        Arbiter(self).run()
      File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 196, in run
        self.halt(reason=inst.reason, exit_status=inst.exit_status)
      File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 291, in halt
        self.stop()
      File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 334, in stop
        self.reap_workers()
      File "/usr/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 430, in reap_workers
        raise HaltServer(reason, self.WORKER_BOOT_ERROR)
    gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>

[akauppi]

I was in the process of trying this. Attaching to an existing container seems less than straightforward on Docker. I am still trying to figure out. Thanks.

There is now a make shell command to easily get into the container.