Nok Lam Chan

Results 535 comments of Nok Lam Chan
trafficstars

@notniknot From what I see your pipelines structure should works already, is the only problem you cannot move `pipeline_registry.py` one level down? >In Organisations there are often already established cookiecutter...

> with kedro_src folder bundling settings.py and pipeline_registry.py (and maybe even __main__.py?): To communicate this better, the only fix point is `settings.py` and `pipeline_registry.py`, `__main__.py` is not a kedro specific...

Would the ibis dataset already support polar as a backend?

Make sense, @AntonNikishin is this something you would like to work on?

@yury-fedotov I think the change should go to starter, and I don't like the current test example. I just found another problem https://github.com/kedro-org/kedro/issues/3966 with the current test. My point is...

https://linen-slack.kedro.org/t/22711373/is-there-any-people-want-to-use-the-kedro-vscode-extension-s#9a92d9f4-8083-4e74-acf3-ef811df08201 Gathered some topics from Kedro Slack Archive & Github - slow s3 glob https://linen-slack.kedro.org/t/22727466/hi-fellows-we-are-using-the-versioned-pandas-parquetdataset-#e411b958-a918-4874-88b2-2396a9d726c3 - I/O related issue (i.e. CacheDataset/kedro-accelerator) https://linen-slack.kedro.org/t/16947241/using-the-last-version-of-the-0-18-i-found-out-that-when-you#70c641e4-98d6-48fe-8e6a-2a0a4cee65dd - Slow catalog load https://linen-slack.kedro.org/t/15474644/hey-everyone-im-trying-to-add-some-tests-to-my-project-but-i#4796efa8-d9a0-4c40-b73f-36b15f412225 - Slow Config...

I suggest focus on two things: 1. Kedro vs Without Kedro 2. Config/Pipeline/Catalog creation time when the # of entry scales.

Speak to @rashidakanchwala today and we conclude that size of the pipeline is usually not the bottleneck for viz, so we will forgo creating project with complex (nested) modular pipeline....

@ElenaKhaustova > I would suggest adding tests for different types of runners to Component stress test; What do you have in mind for stress testing Runners? Generate some dummy node...

It may be interesting to have memory profiling too, it will be helpful to address issues like - https://github.com/kedro-org/kedro/discussions/4111