Joel

Gotcha it was more of a hairbrained thought than anything more :)

Not sure if super helpful, but I rand the following in a blank Databricks notebook: ```python import os [{"env_var":k, "char_count":len(v), "example":v[:30] } for k,v in os.environ.items()] ``` `TERM` sticks out...

Are you running this with ParallelRunner? That's a common issue here.

Hi @brendalf I've just realised this is possibly resolved by tweaking the `copy_mode` of the memory dataset when passed into the next node: https://kedro.readthedocs.io/en/latest/_modules/kedro/io/memory_dataset.html

@noklam do you think we could catch this pickling error and recommend the solution? It's a hard one to debug for users in this situation.

Sorry - `MemoryDataSet` is used to dynamically pass data between nodes automatically, if you look at the implementation we automatically do this for native Spark dataframes:  So you can...

@brendalf or perhaps - why can't you just return runtime data as inputs to another node, does it need to be in the DataSet implementation?

I think this option is most common amongst the community: > Create a class that accepts the data and the replace where query, so the node can send everything I...

We can get far with a Slack workflow without actually building a fully fledged bot :)

I'm just worried a chatbot is an unwinnable game, we could get quite far with emoji reactions for the some of the common ones like: `kedro-dynamic-pipelines` `kedro-hooks` `kedro-migration-guide`