[FEATURE] Override default picke + base64 serialization when setting task values from Brickflow Context

[riccamini]

Is your feature request related to a problem? Please describe. Brickflow context allows setting task values through BrickflowTaskComs:

brickflow_ctx.task_coms.put(task_id="some_task_id", key="some_key", value="some_value"))

This will get by default serialized through pickle + base64 before being pushed to databricks workflows and will make the task value not always suitable to be used in if/else condition tasks.

Cloud Information

• [x] AWS
• [x] Azure
• [x] GCP
• [x] Other

Describe the solution you'd like I would like to give the user the possibility disable pickle + base64 when setting / reading task values. If disabled, task values should be serialized to simple python strings. On the other side, when reading a task value without picke + base64 serialization, the user should expect a string as a result and cast it to the original type if needed outside of the brickflow context.

Describe alternatives you've considered Simply not relying on brickflow context for setting up such task values, hence using dbutils directly from workflow code.