Niels Bantilan

Cutting a `0.18.1` release this weekend to capture these changes

This also solves the problem with [eager mode](https://github.com/flyteorg/flytekit/pull/1579), i.e. we can re-render the Flyte deck of subtasks being kicked-off by the eager workflow in real-time.

@jeevb

yeah, the regex approach doesn't actually work... will need to create some kind of special encoding of this in the flyte literal

> You should remove your token! :) How do I rotate this? Googling around yields no useful info on this

> Additional context: this command was working for me yesterday. I tried a fresh miniforge install, but that didn't get around the issue. The weird thing is this was working...

@NeerajMalhotra-QB @jaskaransinghsidana the `unique=True` core check was never implemented for pyspark right? I don't see it here: https://github.com/unionai-oss/pandera/blob/main/pandera/backends/pyspark/column.py#L38-L42 Would this be as simple as: ``` df.select(schema.name).distinct().count() == df.select(schema.name).count() ``` If...

> By default it should be disabled and only runs when absolutely needed. Cool, `unique=False` by default, so opting in with `unique=True` should make sense

Happy to review a PR to [update documentation](https://github.com/unionai-oss/pandera/blob/main/docs/source/pyspark_sql.md) and a warning/error if it's specified @zippeurfou . > It breaks trust with the user to not have documentation or a warning...

pinging this issue again, in case anyone has the capacity to make a PR for it. Basically the PR just needs to implement the solution described [here](https://github.com/unionai-oss/pandera/issues/1344#issuecomment-1732342719), with unit tests