opencensus-python icon indicating copy to clipboard operation
opencensus-python copied to clipboard

Adding Celery integration

Open dsborets opened this issue 6 years ago • 9 comments

Celery integration to be able to trace asynchronous tasks

dsborets avatar Nov 02 '19 20:11 dsborets

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google. In order to pass this check, please resolve this problem and then comment @googlebot I fixed it.. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

googlebot avatar Nov 02 '19 20:11 googlebot

Unfortunately travis build keeps failing because of two things:

  1. pkg_resources.ContextualVersionConflict: (setuptools 39.1.0 (/home/travis/build/census-instrumentation/opencensus-python/.tox/py36-unit/lib/python3.6/site-packages), Requirement.parse('setuptools>=40.3.0'), {'google-auth'}) It's only for python 3.4, 3.6 I cannot reproduce it locally. When I'm trying to run tox it's failing but for different reason that I don't see on travis, weird...
  2. For python 3.7 tests keeps failing because of sort: it suggesting to remove an empty line, but if I do it python 2.7 will fail because of the same sort is suggesting to add an empty line in the same position Any ideas how to fix it?

dsborets avatar Nov 05 '19 20:11 dsborets

@dsborets Hi, thanks so much for contributing! For (1), when you run tox locally, what error are you getting? For (2), try running isort -rc . on the root folder of opencensus. If it is still failing for one of the versions, take a look at isort configurations to see if you can ignore the error.

lzchen avatar Nov 05 '19 21:11 lzchen

@lzchen thanks for helping with troubleshooting this

For (1) I'm getting multiple of:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 101, in worker
    inqueue._writer.close()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 177, in close
    self._close()
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 361, in _close
    _close(self._handle)
OSError: [Errno 9] Bad file descriptor```

which, seems like could be ignored. And than an exception:

================================================================================================================== FAILURES ==================================================================================================================
____________________________________________________________________________________________________ Test_threading_trace.test_wrap_pool _____________________________________________________________________________________________________

self = <test_threading_trace.Test_threading_trace testMethod=test_wrap_pool>

    def test_wrap_pool(self):
        _tracer = tracer.Tracer()
        execution_context.set_opencensus_tracer(tracer)
    
        trace.trace_integration()
        context = tracer.Tracer().span_context
        print(context.trace_id)
    
        pool = Pool(processes=1)
        with _tracer.span(name='span1'):
>           result = pool.apply_async(fake_pooled_func, ()).get(timeout=1)

contrib/opencensus-ext-threading/tests/test_threading_trace.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <multiprocessing.pool.ApplyResult object at 0x117305810>, timeout = 1

    def get(self, timeout=None):
        self.wait(timeout)
        if not self.ready():
>           raise TimeoutError
E           multiprocessing.context.TimeoutError

/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py:653: TimeoutError
------------------------------------------------------------------------------------------------------------ Captured stdout call ------------------------------------------------------------------------------------------------------------

For (2): When I ran isort -rc . I've got nothing as an output so the code has no problem with import sorting. What is really weird why with the same .isort.cong file the results are different for different python versions?

dsborets avatar Nov 05 '19 22:11 dsborets

@googlebot I fixed it.

dsborets avatar Nov 07 '19 18:11 dsborets

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

googlebot avatar Nov 07 '19 18:11 googlebot

Hi @songy23, @reyang, @c24t. Could you please review this PR? We are going to use this instrumentation for our services soon. Will be great to have it as part of the standard instrumentation and not as our local package. I'm going to port it to the Opentelemetry project soon

dsborets avatar Dec 10 '19 23:12 dsborets

@dsborets can you look into the circle failure?

tmc avatar Jan 16 '20 22:01 tmc

Hi @tmc That failure happened after I merged the code from the 'master' to my branch. I didn't have this problem before in my branch. Also, it seems like something wrong with sqlite module importing:

Traceback (most recent call last):
  File "/root/repo/.nox/unit-2-7/bin/py.test", line 8, in <module>
    sys.exit(main())
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/config/__init__.py", line 63, in main
    config = _prepareconfig(args, plugins)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/config/__init__.py", line 207, in _prepareconfig
    pluginmanager=pluginmanager, args=args
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/hooks.py", line 286, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/manager.py", line 93, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/manager.py", line 87, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/callers.py", line 203, in _multicall
    gen.send(outcome)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/helpconfig.py", line 94, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/callers.py", line 81, in get_result
    _reraise(*ex)  # noqa
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/config/__init__.py", line 687, in pytest_cmdline_parse
    self.parse(args)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/config/__init__.py", line 895, in parse
    self._preparse(args, addopts=addopts)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/config/__init__.py", line 841, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pluggy/manager.py", line 299, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/importlib_metadata/__init__.py", line 94, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/_pytest/assertion/rewrite.py", line 304, in load_module
    exec(co, mod.__dict__)
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/pytest_cov/plugin.py", line 6, in <module>
    import coverage
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/coverage/__init__.py", line 15, in <module>
    from coverage.control import Coverage, process_startup
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/coverage/control.py", line 20, in <module>
    from coverage.data import CoverageData, combine_parallel_data
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/coverage/data.py", line 17, in <module>
    from coverage.sqldata import CoverageData
  File "/root/repo/.nox/unit-2-7/lib/python2.7/site-packages/coverage/sqldata.py", line 15, in <module>
    import sqlite3
  File "/usr/local/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/usr/local/lib/python2.7/sqlite3/dbapi2.py", line 28, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3

I cannot reproduce it locally, all tests are passing. It could be some temporary glitch so the next build will be successful, unfortunately I cannot re-run only that step.

dsborets avatar Jan 16 '20 22:01 dsborets