eland
eland copied to clipboard
Add functionality for `pandas_to_eland` to handle columns with `.`
Resolves #418
This PR adds the functionality to handle nested Elasticsearch fields and ETL processes that have dataframes that use periods in column names that map to nested fields in the Elastic mapping.
Instead of trying to take each column with a . in the pandas dataframe, create a nested mapping to map the Elasticsearch mapping format and change the verify_mapping_compatability function to be able to compare nested schemas - I took the Elastic mapping schema and then compressed it so it's column_name: type ( ex. {"a.b.c.d": {"type": "long"}} ). This significantly simplified the implementation.