eland icon indicating copy to clipboard operation
eland copied to clipboard

Add functionality for `pandas_to_eland` to handle columns with `.`

Open Ashton-Sidhu opened this issue 4 years ago • 0 comments

Resolves #418

This PR adds the functionality to handle nested Elasticsearch fields and ETL processes that have dataframes that use periods in column names that map to nested fields in the Elastic mapping.

Instead of trying to take each column with a . in the pandas dataframe, create a nested mapping to map the Elasticsearch mapping format and change the verify_mapping_compatability function to be able to compare nested schemas - I took the Elastic mapping schema and then compressed it so it's column_name: type ( ex. {"a.b.c.d": {"type": "long"}} ). This significantly simplified the implementation.

Ashton-Sidhu avatar Dec 30 '21 07:12 Ashton-Sidhu