pandavro icon indicating copy to clipboard operation
pandavro copied to clipboard

Unable to infer timestamp type

Open numersoz opened this issue 4 months ago • 0 comments

I have my Avro Schema as below:

schema = { "namespace": "example.avro", "type": "record", "name": "IoTData", "fields": [ {"name": "nodeId", "type": ["null", "string"], "default": None}, {"name": "displayName", "type": ["null", "string"], "default": None}, {"name": "dataType", "type": ["null", "string"], "default": None}, {"name": "statusCode", "type": ["null", "string"], "default": None}, {"name": "timestamp", "type": ["null", {"type": "string", "logicalType": "timestamp-micros"}], "default": None}, {"name": "sourceTimestamp", "type": ["null", {"type": "string", "logicalType": "timestamp-micros"}], "default": None}, {"name": "value", "type": ["null", "double"], "default": None} ] }

I had to put default values as None as sometimes these values maybe blank.

File content is as below:

[{'nodeId': 'ns=2;s=SCD30_CO2', 'displayName': 'SCD30_CO2', 'dataType': 'Double', 'statusCode': 'Good', 'timestamp': '2024-02-25T22:56:21.622480', 'sourceTimestamp': '2024-02-26T03:56:20.224859', 'value': 61.83}, {'nodeId': 'ns=2;s=SCD30_TEMPERATURE', 'displayName': 'SCD30_TEMPERATURE', 'dataType': 'Double', 'statusCode': 'Good', 'timestamp': '2024-02-25T22:56:21.622480', 'sourceTimestamp': '2024-02-26T03:56:20.224859', 'value': 27.35}, {'nodeId': 'ns=2;s=SCD30_HUMIDITY', 'displayName': 'SCD30_HUMIDITY', 'dataType': 'Double', 'statusCode': 'Good', 'timestamp': '2024-02-25T22:56:21.622480', 'sourceTimestamp': '2024-02-26T03:56:20.224859', 'value': 41.49}, {'nodeId': 'ns=2;s=SCD30_CO2', 'displayName': 'SCD30_CO2', 'dataType': 'Double', 'statusCode': 'Good', 'timestamp': '2024-02-25T22:56:22.704777', 'sourceTimestamp': '2024-02-26T03:56:22.250094', 'value': 63.27}]

When I convert this to Pandas data types are as below:


0 nodeId 60 non-null string 1 displayName 60 non-null string 2 dataType 60 non-null string 3 statusCode 60 non-null string 4 timestamp 60 non-null object 5 sourceTimestamp 60 non-null object 6 value 60 non-null float64

Its unable to infer timestamp type.

numersoz avatar Feb 26 '24 04:02 numersoz