Execute/Jobs not returning the logs

[vsmarinescu]

trafficstars

Expected Behavior

I would expect the stdout output to be returned when a job or dbx execute is concluded.

I am running a kedro pipeline as a python wheel file. (Python 3.9)

Current Behavior

Both the jobs and the execute functionality return the following warning (and nothing else):

/databricks/python_shell/dbruntime/PythonPackageImportsInstrumentation/__init__.py:171: 
DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses  original_result = python_builtin_import(name, globals, locals, fromlist, level)

This warning is issued by the imp library.

I can see the right output in the cluster logs. If I keep the context and run the same pipeline again, all the right output is returned (I assume the warning is not reissued).

I've tried running the cluster with spark.databricks.instrumentation.packageImports.enabled false but the warning was reached from a different path and the result was the same.

Steps to Reproduce (for bugs)

I am not sure what causes these, besides the warning, there is nothing else returned or printed in the cluster logs. I assume any python wheel that uses importlib may cause this issue.

Context

Your Environment

I am running a kedro project built with Poetry. The dependencies installed are:

python = ">=3.9,<3.11"
kedro = { version = "~=0.18.2", extras = ["spark.sparkdataset"] }
nbstripout = "~=0.4"
wheel = ">=0.35,<0.37"
mlflow = "~=1.27.0"
memory-profiler = ">=0.60.0,<1.0"
shap = "~=0.41.0"
numba = "~=0.55.1"
sql-formatter = "^0.6.2"
statsmodels = "^0.13.2"
defusedxml = "^0.7.1"
catboost = "~=1.0.6"
mypy = "~=0.971"
click = "8.0.0"
pydantic = "~=1.9.0"
optuna = "^3.0.0"
seaborn = "^0.11.2"
scikit-learn = "1.1.1"

• dbx version used: 0.7.4
• Databricks Runtime version: 11.2 (also tried 11.1)

[renardeinside]

hi @vsmarinescu , the imp library warning is irrelevant to the issue. Could you please verify that you're looking for logs in the expected location as described here, and that you initialize the logger as expected?