ImportError: cannot import name 'StatsdStatsLogger' from 'superset.stats_logger' (/app/superset/stats_logger.py)

[rajesh-bijja]

Bug description

This ticket reports the same issue that was reported in https://github.com/apache/superset/issues/21067 and which was closed as stale. 

We are trying to uptake the superset statsd logging capability as mentioned in the document: https://superset.apache.org/docs/configuration/event-logging/#statsd-logging

from superset.stats_logger import StatsdStatsLogger
STATS_LOGGER = StatsdStatsLogger(host='localhost', port=8125, prefix='superset')

But we are running into below error while starting the superset docker container:

Found but failed to import local superset_config
Traceback (most recent call last):
  File "/app/superset/config.py", line 1704, in <module>
    import superset_config
  File "/app/docker/pythonpath_dev/superset_config.py", line 119, in <module>
    from superset.stats_logger import StatsdStatsLogger
ImportError: cannot import name 'StatsdStatsLogger' from 'superset.stats_logger' (/app/superset/stats_logger.py)
Found but failed to import local superset_config


superset_app          | During handling of the above exception, another exception occurred:
superset_app          |
superset_app          | Traceback (most recent call last):
superset_app          |   File "/app/superset/config.py", line 1704, in <module>
superset_app          |     import superset_config
superset_app          |   File "/app/docker/pythonpath_dev/superset_config.py", line 119, in <module>
superset_app          |     from superset.stats_logger import StatsdStatsLogger
superset_app          | ImportError: cannot import name 'StatsdStatsLogger' from 'superset.stats_logger' (/app/superset/stats_logger.py)
superset_app          | Failed to create app
superset_app          | Traceback (most recent call last):
superset_app          |   File "/app/superset/app.py", line 37, in create_app
superset_app          |     app.config.from_object(config_module)
superset_app          |   File "/usr/local/lib/python3.10/site-packages/flask/config.py", line 227, in from_object
superset_app          |     obj = import_string(obj)
superset_app          |   File "/usr/local/lib/python3.10/site-packages/werkzeug/utils.py", line 611, in import_string
superset_app          |     raise ImportStringError(import_name, e).with_traceback(
superset_app          |   File "/usr/local/lib/python3.10/site-packages/werkzeug/utils.py", line 603, in import_string
superset_app          |     module = __import__(module_name, globals(), locals(), [obj_name])
superset_app          |   File "/app/superset/config.py", line 1704, in <module>
superset_app          |     import superset_config
superset_app          |   File "/app/docker/pythonpath_dev/superset_config.py", line 119, in <module>
superset_app          |     from superset.stats_logger import StatsdStatsLogger
superset_app          | werkzeug.utils.ImportStringError: import_string() failed for 'superset.config'. Possible reasons are:
superset_app          |
superset_app          | - missing __init__.py in a package;
superset_app          | - package or module path not included in sys.path;
superset_app          | - duplicated package or module name taking precedence in sys.path;
superset_app          | - missing module, class, function or variable;
superset_app          |
superset_app          | Debugged import:
superset_app          |
superset_app          | - 'superset' found in '/app/superset/__init__.py'.
superset_app          | - 'superset.config' not found.
superset_app          |

Superset version - v4.0.0
Superset instance - On docker statsd==3.3.0 python-statsd==2.1.0

How to reproduce the bug

1. Add the following two lines at the end of the file: superset_config.py

from superset.stats_logger import StatsdStatsLogger
STATS_LOGGER = StatsdStatsLogger(host='localhost', port=8125, prefix='superset')

2. Start superset docker container and the above error will be thrown.

Screenshots/recordings

No response

Superset version

4.0.0

Python version

3.10

Node version

Not applicable

Browser

Not applicable

Additional context

No response

Checklist

• [X] I have searched Superset docs and Slack and didn't find a solution to my problem.
• [X] I have searched the GitHub issue tracker and didn't find a similar bug report.
• [ ] I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

[dosubot[bot]]

It looks like the issue you're encountering with the ImportError for StatsdStatsLogger from superset.stats_logger could be due to the statsd library not being properly installed or recognized in your environment. To resolve this, ensure that the statsd library is installed by running:

pip install statsd

This step is crucial because the StatsdStatsLogger class depends on the successful import of the statsd library. If the library is missing or there's an import issue, the class won't be defined, leading to the error you're seeing.

Additionally, verify that your Docker environment is correctly configured to communicate with the statsd service. This involves setting the correct host and port values in your configuration to match where your statsd instance is running. In a Docker setup, the host might need to be set to the service name of your statsd container or the IP address of the host machine if statsd is running outside Docker.

If after ensuring the statsd library is installed and your Docker configuration is correct you still face issues, it might be helpful to double-check the version compatibility of Superset and the statsd library, as well as ensuring there are no path or import conflicts in your setup.

Details

Sources

• ImportError: cannot import name 'StatsdStatsLogger' from 'superset.stats_logger' (/app/superset/stats_logger.py)
• ImportError: cannot import name 'StatsdStatsLogger'
• superset/stats_logger.py
• tests/integration_tests/stats_logger_tests.py
• docker/superset-websocket/config.json
• RELEASING/release-notes-4-0/README.md
• pyproject.toml

About Dosu This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[rusackas]

I think the superset.config it says it's not finding is actually referring to config.py. Is that present? I'm not well versed in this app bootstrapping workflow... maybe @dpgaspar can spot a problem here.

If there's not much outcry about something broken here, then it's probably a configuration problem rather than a bug, so we might be forced to migrate this to a Discussion rather than an Issue.

[mistercrunch]

Looks like you're in docker-compose. To repro/debug you can

$ docker-compose exec superset bash
root@4625ad55f961:/app# python
Python 3.10.14 (main, Apr 24 2024, 07:50:19) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from superset.stats_logger import StatsdStatsLogger
>>>

This works for me as is on latest master.

The comment from the error message you provided above seems generic but probably captures the issue:

superset_app          | - missing __init__.py in a package;
superset_app          | - package or module path not included in sys.path;
superset_app          | - duplicated package or module name taking precedence in sys.path;
superset_app          | - missing module, class, function or variable;

[dpgaspar]

Just checked and I was able to configure STATS_LOGGER exactly as the documentation says

[rusackas]

Sounds like it's a config issue, not a bug. I'll close this for now, but I'm happy to move it to a Q&A Discussion if you'd like. Just say the word :)

[rajesh-bijja]

Thank you all for your comments/answers.

I tried connecting to the docker container and ran the above commands:

docker exec -it superset_app bash
root@e24559e47f80:/app# python
Python 3.10.14 (main, Apr 10 2024, 04:55:18) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from superset.stats_logger import StatsdStatsLogger
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'StatsdStatsLogger' from 'superset.stats_logger' (/app/superset/stats_logger.py)
>>>

I am getting the same error. I am not sure what is wrong in my config file or in my env vars file.

Let me try configuring this statsd in our kubernetes pod as the configuration style is helm based and is different from docker compose. If it works in kubernetes then we can skip this in docker compose as we want to get this working finally in kubernetes.

[rajesh-bijja]

I just tried importing StatsClient from python console and that is also failing.

>>> from statsd import StatsClient
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/statsd/__init__.py", line 2, in <module>
    from statsd.client import Client
ImportError: cannot import name 'Client' from 'statsd.client' (/usr/local/lib/python3.10/site-packages/statsd/client/__init__.py)

pip freeze | grep stats
python-statsd==2.1.0
statsd==4.0.1

Not sure how can statsd be a problem in importing here.

[rajesh-bijja]

I think I am able to avoid this issue. Currently I am using statsd 4.0.1 version. I tried moving to 3.3.0 version and it worked fine. I am assuming the reason why it is working for others with statsd 4.0.1 version and not in my env is that I am installing other database dependency libraries that might have cause conflict in one of them.

If others also run in to this issue then though try switching the version of statsd and retry.

[mistercrunch]

If we know for sure that certain versions of a lib work or don't work, we should set that range here: https://github.com/apache/superset/blob/master/pyproject.toml#L196

[SamraHanifCareem]

Moving from 4.0.1 to statsd >= 3.3.0 in setup.py worked for me also