great-expectations
Importing Great Expectations within an Azure Function throws an exception
Describe the bug Within an azure function (and possibly elsewhere) importing great expectations throws an exception (it seems like this is something to do with convert_to_json_serializable, and it's use of IPython, although looking at the gitblame around this I'm not clear why it would suddenly happen now (possible something has happened on the azure function side)
To Reproduce Steps to reproduce the behavior:
- Within an Azure Function, import great_expectations
- This should throw a "ModuleNotFoundError: No module names 'main'
I've only ran this using a "GET" azure function based on South West UK Linux servers, and it appears to have started since last Thursday/Friday.
Expected behavior Importing great_expectations should be possible within the context of an azure function.
Environment (please complete the following information):
- Operating System: Linux, on azure function
- Great Expectations Version: Latest (though appears to persist with previous versions)
Additional context Here's the full stack trace for the error I'm seeing (ignore "azure_function_to_wrap" which is within the erroring code and catches the exception)
Python error occured during execution: Traceback (most recent call last):
File "/home/site/wwwroot/utils/azure_function_helpers.py", line 24, in wrapped_function
result = azure_function_to_wrap(req)
File "/home/site/wwwroot/ExpectationsCheck/__init__.py", line 27, in main
from utils import expectations as ex
File "/home/site/wwwroot/utils/expectations.py", line 2, in <module>
from utils import datalake as dl
File "/home/site/wwwroot/utils/datalake.py", line 9, in <module>
import great_expectations as ge
File "/home/site/wwwroot/.python_packages/lib/site-packages/great_expectations/__init__.py", line 7, in <module>
from great_expectations.data_context import DataContext
File "/home/site/wwwroot/.python_packages/lib/site-packages/great_expectations/data_context/__init__.py", line 1, in <module>
from great_expectations.data_context.data_context import (
File "/home/site/wwwroot/.python_packages/lib/site-packages/great_expectations/data_context/data_context/__init__.py", line 1, in <module>
from great_expectations.data_context.data_context.abstract_data_context import (
File "/home/site/wwwroot/.python_packages/lib/site-packages/great_expectations/data_context/data_context/abstract_data_context.py", line 29, in <module>
from great_expectations.core import ExpectationSuite
File "/home/site/wwwroot/.python_packages/lib/site-packages/great_expectations/core/__init__.py", line 3, in <module>
from .expectation_suite import (
File "/home/site/wwwroot/.python_packages/lib/site-packages/great_expectations/core/expectation_suite.py", line 11, in <module>
from great_expectations.core.evaluation_parameters import (
File "/home/site/wwwroot/.python_packages/lib/site-packages/great_expectations/core/evaluation_parameters.py", line 27, in <module>
from great_expectations.core.util import convert_to_json_serializable
File "/home/site/wwwroot/.python_packages/lib/site-packages/great_expectations/core/util.py", line 17, in <module>
from IPython import get_ipython
File "/home/site/wwwroot/.python_packages/lib/site-packages/IPython/__init__.py", line 53, in <module>
from .terminal.embed import embed
File "/home/site/wwwroot/.python_packages/lib/site-packages/IPython/terminal/embed.py", line 16, in <module>
from IPython.terminal.interactiveshell import TerminalInteractiveShell
File "/home/site/wwwroot/.python_packages/lib/site-packages/IPython/terminal/interactiveshell.py", line 46, in <module>
from .debugger import TerminalPdb, Pdb
File "/home/site/wwwroot/.python_packages/lib/site-packages/IPython/terminal/debugger.py", line 6, in <module>
from IPython.core.completer import IPCompleter
File "/home/site/wwwroot/.python_packages/lib/site-packages/IPython/core/completer.py", line 143, in <module>
import __main__
ModuleNotFoundError: No module named '__main__'
If anyone is looking into this for triage etc, I should also mention:
- The error is (probably pretty obvious if you read the stack) being caused by the fact that IPython.core.completer imports main, which in certain contexts (i.e. azure functions) doesn't seem to exist
- For my purposes, it's easy enough to work around this error in a hacky way by just adding in a completely empty main module
I don't know if this is likely to cause issues elsewhere? (or if it's really a "great expectations" issue, given it pretty clearly seems to come from dependencies)
Hi @benrutter - thank you for raising this and for all of the background context. To confirm, your project was working fine, and then this stopped working suddenly? Are you able to tell the versions of your dependencies when this was last working, and the version when this stopped working?
I agree that I'm not sure exactly what we would be able to do here aside from pinning versions, but anything else you can provide that might help us narrow in would be a huge help.
Hi @talagluck 👋 yes, its as part of a pipeline that routinely checks new files - it spontaneously stopped working sometime between late Thursday and early Friday. Doesn't appear to be an error based on GE versions as other releases fail with the same error (also, doesn't look like the relevant function has been touched in a significant way for a while?)
The error is coming from IPython, but again, that hasn't been updated for a while?
I guess having a conditional import of IPython would on theory fix it, that's probably overkill if this issue hasn't been spotted anywhere else?
HI @talagluck @benrutter our team was facing this exact issue as well. When testing today it appears to have been resolved with no change on our end. Can you confirm on yours?
Sorry for the slow reply - but yes, all resolved now (no longer have the versions of everything that the function app was running on at the initial time of the error, so possible that there's some specific version combination where this error still exists, but seems resolved on most recent versions of dependencies)