Eventual-Inc
add support for adding file path as a new column when read from csv/json etc
a common pattern when reading from csv. json etc, is to add a column in the destination table with the files processed already, so the next time you add new csv files, you will not endup with duplicate values, duckdb/polars for example support this function using filename = true
Thanks for raising this up! This has been something we have been thinking about adding as well.
As an added bonus: if we could figure out that the dataframe is partitioned by filename (if no file splitting was performed) that could be really cool.
This could enable easy and cheap data manipulation such as: df.read_parquet("...", filename=True).groupby("filename").count()
Example use-case for counting number of distinct rows, grouped by filename:
Hi @djouallah , sorry for the delay, I'm currently finalizing the PR for this, will let you know once it is ready
