kedro
kedro copied to clipboard
kedro-org
Kedro tracebacks are overwhelming
Description
Kedro shows nice, colored tracebacks (which is great), but sometimes we don't need to show the full traceback to the user, and in fact it can be quite overwhelming. I propose we try to control the amount of tracebacks that bubble up to the user, and reserve those for user code or internal breakage.
Context
Today I did something silly: tried a kedro run right after a kedro new --starter=pandas-iris, without first installing the dependencies. This is what I got:
juan_cano@M-PH9T4K3P3C /t/test-omegaconf> kedro run (kedro310-dev-test-omegaconf)
[03/08/23 11:29:06] INFO Kedro project test-omegaconf session.py:355
WARNING /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/kedro/framework warnings.py:109
/project/__init__.py:330: UserWarning: An error occurred while importing the 'test_omegaconf.pipeline'
module. Nothing defined therein will be returned by 'find_pipelines'.
Traceback (most recent call last):
File
"/Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/kedro/framewor
k/project/__init__.py", line 327, in find_pipelines
pipeline_module = importlib.import_module(pipeline_module_name)
File
"/Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/importlib/__init__.py", line
126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/private/tmp/test-omegaconf/src/test_omegaconf/pipeline.py", line 8, in <module>
from .nodes import make_predictions, report_accuracy, split_data
File "/private/tmp/test-omegaconf/src/test_omegaconf/nodes.py", line 9, in <module>
import numpy as np
ModuleNotFoundError: No module named 'numpy'
warnings.warn(
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/bin/kedro:8 in <module> │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/kedro │
│ /framework/cli/cli.py:211 in main │
│ │
│ 208 │ """ │
│ 209 │ _init_plugins() │
│ 210 │ cli_collection = KedroCLI(project_path=Path.cwd()) │
│ ❱ 211 │ cli_collection() │
│ 212 │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/click │
│ /core.py:1130 in __call__ │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/kedro │
│ /framework/cli/cli.py:139 in main │
│ │
│ 136 │ │ ) │
│ 137 │ │ │
│ 138 │ │ try: │
│ ❱ 139 │ │ │ super().main( │
│ 140 │ │ │ │ args=args, │
│ 141 │ │ │ │ prog_name=prog_name, │
│ 142 │ │ │ │ complete_var=complete_var, │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/click │
│ /core.py:1055 in main │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/click │
│ /core.py:1657 in invoke │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/click │
│ /core.py:1404 in invoke │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/click │
│ /core.py:760 in invoke │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/kedro │
│ /framework/cli/project.py:455 in run │
│ │
│ 452 │ with KedroSession.create( │
│ 453 │ │ env=env, conf_source=conf_source, extra_params=params │
│ 454 │ ) as session: │
│ ❱ 455 │ │ session.run( │
│ 456 │ │ │ tags=tag, │
│ 457 │ │ │ runner=runner(is_async=is_async), │
│ 458 │ │ │ node_names=node_names, │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/kedro │
│ /framework/session/session.py:380 in run │
│ │
│ 377 │ │ │ │ f"by the 'register_pipelines' function." │
│ 378 │ │ │ ) from exc │
│ 379 │ │ │
│ ❱ 380 │ │ filtered_pipeline = pipeline.filter( │
│ 381 │ │ │ tags=tags, │
│ 382 │ │ │ from_nodes=from_nodes, │
│ 383 │ │ │ to_nodes=to_nodes, │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-dev-test-omegaconf/lib/python3.10/site-packages/kedro │
│ /pipeline/pipeline.py:769 in filter │
│ │
│ 766 │ │ │ filtered_pipeline &= subset_pipeline │
│ 767 │ │ │
│ 768 │ │ if not filtered_pipeline.nodes: │
│ ❱ 769 │ │ │ raise ValueError( │
│ 770 │ │ │ │ "Pipeline contains no nodes after applying all provided filters" │
│ 771 │ │ │ ) │
│ 772 │ │ return filtered_pipeline │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: Pipeline contains no nodes after applying all provided filters
For the user to understand what happened here they are forced to scroll to the top of the error message, which most likely will not be shown (unless the user has a really tall screen). This is an actual screenshot of how my screen was looking right after running the command:
The actual problem here was that a dependency was missing, and therefore no pipelines were left. But the user is presented with a huge traceback (colored and nice, but still huge) and a lengthy warning, when the actual solution is to do pip install -r src/requirements.txt.
I argue that tracebacks should be shown to the user when something inside Kedro breaks, or when user code fails for whatever reason. But "pipeline contains no nodes" is hardly a broken condition: it can come up when certain tags or filters are applied. This is just the example that prompted me to open this issue, but there are more.
Possible Implementation
For managed exceptions, maybe a one line red warning would be more than enough:
juan_cano@M-PH9T4K3P3C /t/test-omegaconf> kedro run --pipeline=_default_ # glaring typo (kedro310-dev-test-omegaconf)
[03/08/23 11:43:36] INFO Kedro project test-omegaconf session.py:355
[03/08/23 11:43:36] WARNING Failed to find the pipeline named '_default_'. It needs to be generated and returned by the 'register_pipelines' function. Maybe you meant `__default__`?
juan_cano@M-PH9T4K3P3C /t/test-omegaconf [1]> echo $status (kedro310-dev-test-omegaconf)
1
(And the "maybe you meant" suggestion would be the icing in the cake)
Possible Alternatives
Alternatively, the full traceback should only be shown after a --verbose/-v or --debug flag is passed. At the moment kedro run does not have any such flags.
Let me first explain what's happening above because it's not obvious unless you really look carefully:
- a
WARNINGis emitted byfind_pipelines, which is used for automatic registration of kedro pipelines (see https://github.com/kedro-org/kedro/issues/1664 and https://github.com/kedro-org/kedro/issues/1284 for context). The warning saysAn error occurred while importing the 'test_omegaconf.pipeline'. Nothing defined therein will be returned by 'find_pipelines'. - the warning also gives the original exception traceback that caused the error while importing
test_omegaconf.pipeline. This is the plain (non-rich) traceback that culminates inModuleNotFoundError: No module named 'numpy'. Note that nothing yet has halted execution; the exception has been caught and just re-issued as part of a warning message. - since
find_pipelinescan't see anything to register, it returns{"__default__": pipeline([])}. Ultimately this is what your project'sregister_pipelineswill also return - hence the
kedro runcommand is trying to run an empty pipeline. This results in the rich exception that you see that culminates inValueError: Pipeline contains no nodes after applying all provided filtersand is the one that halts execution.
Is this confusing? Yes, definitely. We should try to improve it so that it's more obvious to the user that the way to fix the problem is to pip install pandas, which as you say is very buried at the moment.
There's a couple of possible things we might want to do here and it's worth breaking down more...
1. find_pipelines behaviour
This is tricky to get right because there's two conflicting requirements:
- we want to fail fast to expose the underlying exception ASAP so it doesn't get buried and a user knows how to fix it
- we actually don't want to fail fast, because you still should be able to do a
kedro runwith a missing dependency if that dependency is not required in the pipeline you're actually running
By design we have prioritised the second of these, but maybe there's a better compromise to be made. @deepyaman any thoughts?
2. Managed exceptions
This is an excellent idea I think, and actually we have an exception class that basically does this already: KedroCliError. If I change ValueError("Pipeline contains no nodes after applying all provided filters") to KedroCliError then here's what happens:
Note the error message is wrong since --verbose doesn't actually exist for run as it doesn't use command_with_verbosity (not sure why tbh, maybe it should).
This isn't an immediate fix because it doesn't really make sense to raise KedroCliError outside the CLI (e.g. if you're doing a programmatic kedro run), but I am very much in favour of having some sort of class that could generate user-friendly error messages rather than detailed tracebacks for the cases of "expected" errors like this where nothing is actually broken. Like you say, there's many such examples across kedro.
As mentioned during backlog grooming (and without having fully read the above):
- I think there's clear value in showing a shorter traceback in the above case. Furthermore, showing shorter tracebacks would mean a verbose mode would provide more value.
- That being said, we definitely need to be cautious about hiding tracebacks; users will be very unhappy if they run for 3 hours, run into an error, and get prompted to rerun with
--verbose.
Couple quick thoughts:
- Maybe we can distinguish pre-run errors from errors while running pipelines, and be more conservative around hiding context for errors during pipeline runs?
- Maybe we can display a shorter traceback via the CLI but point to a log file with fuller logs.
We discussed this at Technical Design today. Some notes:
Ideally, by default the user should see a traceback only in two situations:
- User code failed (they have to fix it)
- Kedro had an internal, unexpected failure (they have to open an issue here)
Otherwise, a friendlier error message should be shown. Some users might want to actually see the full traceback or will be quick to spot the actual error, but we want to err on the side of favoring beginners here.
Ideally, these cases would look as follows:
$ kedro run --tags=notexist
ValueError: Pipeline contains no nodes after applying all provided filters
(Add `-v` to show extra information)
$ kedro run --pipeline=__defaulXX__
ValueError: Pipeline contains no nodes after applying all provided filters
Available pipelines are:
- __default__
(Add `-v` to show extra information)
$ kedro run --from-nodes=make_predictionXX
ValueError: Pipeline does not contain nodes named ['make_predictionXX']
(Add `-v` to show extra information)
There are essentially two possible workflows:
- Interactive (user is running Kedro commands on the CLI)
- Unattended (Kedro is running in production or as part of a background process)
We focused on some kedro run cases, and left aside for now the "missing dependencies" problem (because find_pipelines could be more difficult to fix).
We settled on concealing the traceback by default behind a flag, which could be -v/--verbose, --debug, --full-traceback, or something else. We'd like to avoid situations in which long-running processess need to be restarted with -v - however, it looks like the cases at hand can be detected very early on.
As an alternative, we considered having full logs are always written somewhere to disk, and only shown with a flag. But we quickly found that this could lead to problems in read-only filesystems, and does not necessarily make the situation better (for example, the user might not be able to easily extract artifacts from CI). Above all, "the last thing you want to do is write logs and crash the process/make the situation worse".
We already have some machinery for this: the KedroCLIError class and the _click_verbose function, but we're not using the consistently.
Possible implementation:
- Controlling for a "managed error" class in
kedro.framework.cli.project:runto conceal tracebacks fromkedro run(and offer them behind a flag). The relevantValueErroretc. that become managed exception and are raised insession.runwould need to be changed toManagedKedroError.
with KedroSession.create(
env=env, conf_source=conf_source, extra_params=params
) as session:
try:
session.run(
tags=tag,
runner=runner(is_async=is_async),
node_names=node_names,
from_nodes=from_nodes,
to_nodes=to_nodes,
from_inputs=from_inputs,
to_outputs=to_outputs,
load_versions=load_version,
pipeline_name=pipeline,
namespace=namespace,
)
except ManagedKedroError as e:
# Handle error appropriately, conceal traceback
...
# else:
# raise as before
- Improve error messages for
kedro runby adding suggestions (no such pipeline, no nodes with such tags, etc) - Apply this to the rest of the CLI for consistency after some research (for example, decorate every click command with
managed_kedro_errordecorator which does the try/except)
Notice that in some places we are already making suggestions, but it's debatable whether the full traceback should be shown (this is not CLI usage, but Python usage):
In [1]: catalog.load("example_iris_dataX")
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:1 │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-errors/lib/python3.10/site-packages/kedro/io/data_cat │
│ alog.py:341 in load │
│ │
│ 338 │ │ │ >>> df = io.load("cars") │
│ 339 │ │ """ │
│ 340 │ │ load_version = Version(version, None) if version else None │
│ ❱ 341 │ │ dataset = self._get_dataset(name, version=load_version) │
│ 342 │ │ │
│ 343 │ │ self._logger.info( │
│ 344 │ │ │ "Loading data from '%s' (%s)...", name, type(dataset).__name__ │
│ │
│ /Users/juan_cano/.micromamba/envs/kedro310-errors/lib/python3.10/site-packages/kedro/io/data_cat │
│ alog.py:300 in _get_dataset │
│ │
│ 297 │ │ │ │ │ suggestions = ", ".join(matches) │
│ 298 │ │ │ │ │ error_msg += f" - did you mean one of these instead: {suggestions}" │
│ 299 │ │ │ │
│ ❱ 300 │ │ │ raise DataSetNotFoundError(error_msg) │
│ 301 │ │ │
│ 302 │ │ data_set = self._data_sets[data_set_name] │
│ 303 │ │ if version and isinstance(data_set, AbstractVersionedDataSet): │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
DataSetNotFoundError: DataSet 'example_iris_dataX' not found in the catalog - did you mean one of these instead:
example_iris_data
Note the _click_verbose function we have already uses -v/--verbose for this, so I'd continue using those flags unless there's a particular reason to change to something else like --debug.
https://github.com/kedro-org/kedro/blob/653145771fa440a46b36d2c96c57a5997613da5e/kedro/framework/cli/utils.py#L234-L242
@astrojuanlu do you have any examples of packages that have their own ManagedException class like this?
I don't have any examples in mind, we could do some research. Unless I'm missing something I don't think they need to be too sophisticated, just a subclass of RuntimeError carrying the appropriate error message, generated internally.
A user complained this week to me in person that when a node fails it's very difficult to see the actual error, because the traceback includes too much information from the Kedro internals.
Hi everyone,
If I may jump in with my own clumsy formulation:
The traceback's "signal-to-noise" ratio is indeed something that could be improved.
The relevant information is almost never immediately visible "at very bottom" and almost always concealed somewhere "in the middle".
As @astrojuanlu previously said: This is often confusing for new users. And even for experience users: Not having to systematically scroll the traceback would result in a better user experience, with less "friction", resulting in a greater productivity.
Hoping this helps :-)
Regards
M
Happy 1 year anniversary to this issue 🍰
I was prescient:
Alternatively, the full traceback should only be shown after a --verbose/-v or --debug flag is passed. At the moment kedro run does not have any such flags.
Probably we should tackle #3446 and #3591 before this one, so that people can do kedro run -v or kedro run -vv to show the full tracebacks. @noklam what do you think?
Agree with the sentiments mostly. I have an attempt to reduce the noise coming from Kedro during a kedro run.
- Managed Exception, I think it's a nice idea, but I don't see how it can help for
kedro run, maybe more useful for pre-kedro run errors
Like @deepyaman said, I would be careful about hiding traceback particular for long runs. I like the idea having less crowd terminal and more verbose in logging, but I suspect no one is going to look at the log files during development.
I think #3446 is ready to go and there is not much drawback. It won't work for 100% use case, but I think it will improve the developer experience a lot.
#3591 can be broken down to two separate tickets (edited: I just did it)
- Change the default of
default_logging.ymland the templatelogging.yml#3687 - Add verbose mode
-vand-vv- #3591
In my mind, once -v is available, it will be the preferred options, logging.yml is more useful for advance use cases or production logging. The more controversial one seems to be #3687, I suggest we go with #3446 and #3591, because that solves 95% of the problem already.
I think https://github.com/kedro-org/kedro/issues/3446 is ready to go
Been participating and following the conversation there and I'm not sure that I saw a clear action plan, could you clarify?
logging.yml is more useful for advance use cases or production logging.
I would very much welcome some historical context on why Kedro decided to depart from the 12 Factor App guidance on logging but at this point and especially after seeing @noklam's demo I am of the opinion that production logging should be handled by production tools and that we should plan a smooth transition towards phasing out our *logging.yml handling completely.
Probably not a super valuable comment but just saw this issue pop up in slack.
The way Kedro will swallow actual errors and return its own errors has always been a big pet peeve of mine. Having to scroll up about 2/3 of the way up the trace back to find the true cause of the error, when the bottom 2/3 are totally irrelevant isn't great UX so big support for any development which might improve this 😃
Do you mean the custom Kedro error class or things like auto pipeline discovery that generate a warning message instead of an error?
The former one is created to provide better explicit message, if we are actually making it worse it's bad. Can you give an example of it?
On Tue, 9 Apr 2024, 23:48 Juan Luis Cano Rodríguez, < @.***> wrote:
Also, related: #3794 https://github.com/kedro-org/kedro/issues/3794
— Reply to this email directly, view it on GitHub https://github.com/kedro-org/kedro/issues/2401#issuecomment-2046161884, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELAWL4FTSKRRGFBDPEHLFDY4RV4XAVCNFSM6AAAAAAVTTVWGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBWGE3DCOBYGQ . You are receiving this because you were mentioned.Message ID: @.***>
Since rich/textual support some css / Web like feature, would it be possible to support something that collapse kedro error by default?
There are concern that we tried too hard to hide important message, and a rerun is too expensive just to see the error again.
On Thu, 11 Apr 2024, 19:34 Nok Chan, @.***> wrote:
Do you mean the custom Kedro error class or things like auto pipeline discovery that generate a warning message instead of an error?
The former one is created to provide better explicit message, if we are actually making it worse it's bad. Can you give an example of it?
On Tue, 9 Apr 2024, 23:48 Juan Luis Cano Rodríguez, < @.***> wrote:
Also, related: #3794 https://github.com/kedro-org/kedro/issues/3794
— Reply to this email directly, view it on GitHub https://github.com/kedro-org/kedro/issues/2401#issuecomment-2046161884, or unsubscribe https://github.com/notifications/unsubscribe-auth/AELAWL4FTSKRRGFBDPEHLFDY4RV4XAVCNFSM6AAAAAAVTTVWGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBWGE3DCOBYGQ . You are receiving this because you were mentioned.Message ID: @.***>
From just this morning.
This might be personal preference: the traceback is 244 lines long, but only the first 70 lines which show the error and the last two lines which identify the dataset are actually relevant to me, and the other 170 (70%) are kedro stuff which in my experience has never been helpful to me.
Traceback
You can resume the pipeline run from the nearest nodes with persisted inputs by adding the following argument to your previous command:
--from-nodes ""
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/lib/python3.10/site-packages/kedro/io/core.py:193 in load │
│ │
│ 190 │ │ self._logger.debug("Loading %s", str(self)) │
│ 191 │ │ │
│ 192 │ │ try: │
│ ❱ 193 │ │ │ return self._load() │
│ 194 │ │ except DatasetError: │
│ 195 │ │ │ raise │
│ 196 │ │ except Exception as exc: │
│ │
│ /usr/local/lib/python3.10/site-packages/axpo/kedro/datasets/pandas_arrow_dat │
│ aset.py:338 in _load │
│ │
│ 335 │ │ │ pads.Dataset: The loaded dataset. │
│ 336 │ │ """ # noqa │
│ 337 │ │ │
│ ❱ 338 │ │ dataset = self.arrow_dataset │
│ 339 │ │ # Here we are deleting the partition so then we can't filter │
│ 340 │ │ if self.partition_by is not None: │
│ 341 │ │ │ for partition_col in self.partition_by: │
│ │
│ /usr/local/lib/python3.10/site-packages/axpo/kedro/datasets/pandas_arrow_dat │
│ aset.py:312 in arrow_dataset │
│ │
│ 309 │ │ """Instantiate the pyarrow dataset object. │
│ 310 │ │ Used for loading data. │
│ 311 │ │ """ │
│ ❱ 312 │ │ dataset = pads.dataset( │
│ 313 │ │ │ self.upath.path, │
│ 314 │ │ │ filesystem=self.filesystem, │
│ 315 │ │ │ partitioning=self._partitioning(flavor_only=True), # infe │
│ │
│ /usr/local/lib/python3.10/site-packages/pyarrow/dataset.py:782 in dataset │
│ │
│ 779 │ ) │
│ 780 │ │
│ 781 │ if _is_path_like(source): │
│ ❱ 782 │ │ return _filesystem_dataset(source, **kwargs) │
│ 783 │ elif isinstance(source, (tuple, list)): │
│ 784 │ │ if all(_is_path_like(elem) for elem in source): │
│ 785 │ │ │ return _filesystem_dataset(source, **kwargs) │
│ │
│ /usr/local/lib/python3.10/site-packages/pyarrow/dataset.py:465 in │
│ _filesystem_dataset │
│ │
│ 462 │ if isinstance(source, (list, tuple)): │
│ 463 │ │ fs, paths_or_selector = _ensure_multiple_sources(source, file │
│ 464 │ else: │
│ ❱ 465 │ │ fs, paths_or_selector = _ensure_single_source(source, filesys │
│ 466 │ │
│ 467 │ options = FileSystemFactoryOptions( │
│ 468 │ │ partitioning=partitioning, │
│ │
│ /usr/local/lib/python3.10/site-packages/pyarrow/dataset.py:441 in │
│ _ensure_single_source │
│ │
│ 438 │ elif file_info.type == FileType.File: │
│ 439 │ │ paths_or_selector = [path] │
│ 440 │ else: │
│ ❱ 441 │ │ raise FileNotFoundError(path) │
│ 442 │ │
│ 443 │ return filesystem, paths_or_selector │
│ 444 │
╰──────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError:
The above exception was the direct cause of the following exception:
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/bin/kedro:8 in <module> │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/framework/cli/cli.py:211 in │
│ main │
│ │
│ 208 │ """ │
│ 209 │ _init_plugins() │
│ 210 │ cli_collection = KedroCLI(project_path=Path.cwd()) │
│ ❱ 211 │ cli_collection() │
│ 212 │
│ │
│ /usr/local/lib/python3.10/site-packages/click/core.py:1157 in __call__ │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/framework/cli/cli.py:139 in │
│ main │
│ │
│ 136 │ │ ) │
│ 137 │ │ │
│ 138 │ │ try: │
│ ❱ 139 │ │ │ super().main( │
│ 140 │ │ │ │ args=args, │
│ 141 │ │ │ │ prog_name=prog_name, │
│ 142 │ │ │ │ complete_var=complete_var, │
│ │
│ /usr/local/lib/python3.10/site-packages/click/core.py:1078 in main │
│ │
│ /usr/local/lib/python3.10/site-packages/click/core.py:1688 in invoke │
│ │
│ /usr/local/lib/python3.10/site-packages/click/core.py:1434 in invoke │
│ │
│ /usr/local/lib/python3.10/site-packages/click/core.py:783 in invoke │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/framework/cli/project.py:453 │
│ in run │
│ │
│ 450 │ with KedroSession.create( │
│ 451 │ │ env=env, conf_source=conf_source, extra_params=params │
│ 452 │ ) as session: │
│ ❱ 453 │ │ session.run( │
│ 454 │ │ │ tags=tag, │
│ 455 │ │ │ runner=runner(is_async=is_async), │
│ 456 │ │ │ node_names=node_names, │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/framework/session/session.py:4 │
│ 36 in run │
│ │
│ 433 │ │ ) │
│ 434 │ │ │
│ 435 │ │ try: │
│ ❱ 436 │ │ │ run_result = runner.run( │
│ 437 │ │ │ │ filtered_pipeline, catalog, hook_manager, session_id │
│ 438 │ │ │ ) │
│ 439 │ │ │ self._run_called = True │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/runner/runner.py:103 in run │
│ │
│ 100 │ │ │ self._logger.info( │
│ 101 │ │ │ │ "Asynchronous mode is enabled for loading and saving d │
│ 102 │ │ │ ) │
│ ❱ 103 │ │ self._run(pipeline, catalog, hook_manager, session_id) │
│ 104 │ │ │
│ 105 │ │ self._logger.info("Pipeline execution completed successfully." │
│ 106 │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/runner/sequential_runner.py:70 │
│ in _run │
│ │
│ 67 │ │ │
│ 68 │ │ for exec_index, node in enumerate(nodes): │
│ 69 │ │ │ try: │
│ ❱ 70 │ │ │ │ run_node(node, catalog, hook_manager, self._is_async, s │
│ 71 │ │ │ │ done_nodes.add(node) │
│ 72 │ │ │ except Exception: │
│ 73 │ │ │ │ self._suggest_resume_scenario(pipeline, done_nodes, cat │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/runner/runner.py:329 in │
│ run_node │
│ │
│ 326 │ │ ) │
│ 327 │ │
│ 328 │ if is_async: │
│ ❱ 329 │ │ node = _run_node_async(node, catalog, hook_manager, session_id │
│ 330 │ else: │
│ 331 │ │ node = _run_node_sequential(node, catalog, hook_manager, sessi │
│ 332 │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/runner/runner.py:474 in │
│ _run_node_async │
│ │
│ 471 │ │ │ inputs[name] = pool.submit(_synchronous_dataset_load, name │
│ 472 │ │ │
│ 473 │ │ wait(inputs.values(), return_when=ALL_COMPLETED) │
│ ❱ 474 │ │ inputs = {key: value.result() for key, value in inputs.items() │
│ 475 │ │ is_async = True │
│ 476 │ │ additional_inputs = _collect_inputs_from_hook( │
│ 477 │ │ │ node, catalog, inputs, is_async, hook_manager, session_id= │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/runner/runner.py:474 in │
│ <dictcomp> │
│ │
│ 471 │ │ │ inputs[name] = pool.submit(_synchronous_dataset_load, name │
│ 472 │ │ │
│ 473 │ │ wait(inputs.values(), return_when=ALL_COMPLETED) │
│ ❱ 474 │ │ inputs = {key: value.result() for key, value in inputs.items() │
│ 475 │ │ is_async = True │
│ 476 │ │ additional_inputs = _collect_inputs_from_hook( │
│ 477 │ │ │ node, catalog, inputs, is_async, hook_manager, session_id= │
│ │
│ /usr/local/lib/python3.10/concurrent/futures/_base.py:451 in result │
│ │
│ 448 │ │ │ │ if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]: │
│ 449 │ │ │ │ │ raise CancelledError() │
│ 450 │ │ │ │ elif self._state == FINISHED: │
│ ❱ 451 │ │ │ │ │ return self.__get_result() │
│ 452 │ │ │ │ │
│ 453 │ │ │ │ self._condition.wait(timeout) │
│ 454 │
│ │
│ /usr/local/lib/python3.10/concurrent/futures/_base.py:403 in __get_result │
│ │
│ 400 │ def __get_result(self): │
│ 401 │ │ if self._exception: │
│ 402 │ │ │ try: │
│ ❱ 403 │ │ │ │ raise self._exception │
│ 404 │ │ │ finally: │
│ 405 │ │ │ │ # Break a reference cycle with the exception in self._ │
│ 406 │ │ │ │ self = None │
│ │
│ /usr/local/lib/python3.10/concurrent/futures/thread.py:58 in run │
│ │
│ 55 │ │ │ return │
│ 56 │ │ │
│ 57 │ │ try: │
│ ❱ 58 │ │ │ result = self.fn(*self.args, **self.kwargs) │
│ 59 │ │ except BaseException as exc: │
│ 60 │ │ │ self.future.set_exception(exc) │
│ 61 │ │ │ # Break a reference cycle with the exception 'exc' │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/runner/runner.py:461 in │
│ _synchronous_dataset_load │
│ │
│ 458 │ │ """Minimal wrapper to ensure Hooks are run synchronously │
│ 459 │ │ within an asynchronous dataset load.""" │
│ 460 │ │ hook_manager.hook.before_dataset_loaded(dataset_name=dataset_n │
│ ❱ 461 │ │ return_ds = catalog.load(dataset_name) │
│ 462 │ │ hook_manager.hook.after_dataset_loaded( │
│ 463 │ │ │ dataset_name=dataset_name, data=return_ds, node=node │
│ 464 │ │ ) │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/io/data_catalog.py:506 in load │
│ │
│ 503 │ │ │ "Loading data from '%s' (%s)...", name, type(dataset).__na │
│ 504 │ │ ) │
│ 505 │ │ │
│ ❱ 506 │ │ result = dataset.load() │
│ 507 │ │ │
│ 508 │ │ return result │
│ 509 │
│ │
│ /usr/local/lib/python3.10/site-packages/kedro/io/core.py:202 in load │
│ │
│ 199 │ │ │ message = ( │
│ 200 │ │ │ │ f"Failed while loading data from data set {str(self)}. │
│ 201 │ │ │ ) │
│ ❱ 202 │ │ │ raise DatasetError(message) from exc │
│ 203 │ │
│ 204 │ def save(self, data: _DI) -> None: │
│ 205 │ │ """Saves data by delegation to the provided save method. │
╰──────────────────────────────────────────────────────────────────────────────╯
DatasetError: Failed while loading data from data set
ParquetArrowDataset(credentials={REDACTED},
root_path=REDACTED)
Do you mean the custom Kedro error class or things like auto pipeline discovery that generate a warning message instead of an error?
In the specific case of https://github.com/kedro-org/kedro/issues/3794, if there aren't any pipelines, I'd expect the error message to be exactly 1 line long and say "there are no pipelines defined, you can create one with kedro pipeline create ..." or something like that.
Btw @noklam can you explain the rich css stuff?
Without knowing specifics, I'm not sure how useful that will be outside of interactive/local development, as when running in our production environment (argo workflows) we only get access to whatever is logged to stdout, I don't know if this sort of interactivity would work outside of a terminal environment.
@inigohidalgo We had some discussion about logging internally recently. There are different opinion but I think it's clear that rich is not suitable for production logging in many cases. So anything related to rich here I am referring to development flow in my head.
For production logging, you would at least using something like a json that are easy to parse and can integrate with observability tool. In most cast you won't be reading the log inside the box but rather through some other UI/alert
We need to be precise when we discuss the verbosity, because there are few things mentioned here. I don't think we have a good idea how to solve all of them together, so maybe it's easier to tackle the obvious case first.
Do you have some examples in mind?
For me the only major example I have is the one I put here https://github.com/kedro-org/kedro/issues/2401#issuecomment-2051295192 which I think is totally separate from some other things being discussed above
- adding more helpful error messages like @astrojuanlu said https://github.com/kedro-org/kedro/issues/2401#issuecomment-2051303986
- not swallowing errors and reraising custom errors which make finding the "true" error harder. As a user I care about why something failed, not where in the kedro framework stack the error was raised. Refer to my previous comment's traceback.
- more specific verbosity control?
I suggest splitting this issue into some actionable tasks. There are mixed discussion about logs and tracebacks, some are very specific.
cc @astrojuanlu
- We can try produce some explicit error. I think the tricky point here is that we may need to assume one is using the entire Kedro framework while in theory someone can use one specific class and that error message may be misleading.
- I think custom error mean to make it easier. Are those created because we need to handle
on_pipeline_error,on_node_error? - verbosity control (not sure about this, if 2 is addressed is this still needed? @inigohidalgo )