IMCtermite
IMCtermite copied to clipboard
RecordEvolution
JSONDecode Error with dataset containing CAN-Bus signals
Hello, as mentioned in the "README.md incorrect" issue, I have a problem with importing a dataset of an imc cronos measurement. I get an JSONDecode Error (s. below). In the meantime I discovered, that the error only is raised, if the dataset contains CAN-Bus signals. If I remove them, I do not get this error. But nevertheless, reading the data seems then only work for one channel - in all other channels the xdata and ydata are empty.
Please find below the error and attached an exemplary dataset. Measurement.zip
---------------------------------------------------------------------------
JSONDecodeError Traceback (most recent call last)
Input In [18], in <cell line: 1>()
----> 1 data = raw.get_channels(True)
File IMCtermite.pyx:25, in IMCtermite.imctermite.get_channels()
File ~\.conda\envs\myAnaconda\lib\json\__init__.py:346, in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
341 s = s.decode(detect_encoding(s), 'surrogatepass')
343 if (cls is None and object_hook is None and
344 parse_int is None and parse_float is None and
345 parse_constant is None and object_pairs_hook is None and not kw):
--> 346 return _default_decoder.decode(s)
347 if cls is None:
348 cls = JSONDecoder
File ~\.conda\envs\myAnaconda\lib\json\decoder.py:337, in JSONDecoder.decode(self, s, _w)
332 def decode(self, s, _w=WHITESPACE.match):
333 """Return the Python representation of ``s`` (a ``str`` instance
334 containing a JSON document).
335
336 """
--> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
338 end = _w(s, end).end()
339 if end != len(s):
File ~\.conda\envs\myAnaconda\lib\json\decoder.py:355, in JSONDecoder.raw_decode(self, s, idx)
353 obj, end = self.scan_once(s, idx)
354 except StopIteration as err:
--> 355 raise JSONDecodeError("Expecting value", s, err.value) from None
356 return obj, end
JSONDecodeError: Expecting value: line 1 column 400 (char 399)
[Measurement.zip](https://github.com/RecordEvolution/IMCtermite/files/9615982/Measurement.zip)
`import IMCtermite import json import os import datetime import pdb
declare and initialize instance of "imctermite" by passing a raw-file
try : imcraw = IMCtermite.imctermite(b"dn1.raw") # multiple channel raw file except RuntimeError as e : raise Exception("failed to load/parse raw-file: " + str(e)) pdb.set_trace()
obtain list of channels as list of dictionaries (without data)
channels = imcraw.get_channels(False) print(json.dumps(channels,indent=4, sort_keys=False)) print("number of channels: " + str(len(channels)))
#pdb.set_trace()
obtain all channels (including full data)
channelsdata = imcraw.get_channels(True)
everything that follows is an example that specifically makes use only of
the first (index = 0) channel ...
idx = 0
if len(channelsdata) > 0 :
# get first channel's data
chnydata = channelsdata[idx]['ydata']
chnxdata = channelsdata[idx]['xdata']
print("xdata: " + str(len(chnxdata)))
print("ydata: " + str(len(chnydata)))
# extract trigger-time
trigtim = datetime.datetime.fromisoformat(channels[idx]["trigger-time"])
print(trigtim)
# file output of data with absolute timestamp in 1st column
filname = os.path.join("./",channelsdata[idx]['name']+".csv")
print("writing output into " + filname)
with open(filname,'w') as fout :
# include column header
fout.write( str(channelsdata[idx]['xname']) + '[' + str(channelsdata[idx]['xunit']) + "]"
+ ","
+ str(channelsdata[idx]['yname']) + '[' + str(channelsdata[idx]['yunit']) + "]"
+ "\n" )
# add data (introduce time shift according to trigger-time)
for row in range(0,len(chnxdata)) :
fout.write( str( (trigtim + datetime.timedelta(seconds=chnxdata[row])).isoformat() )
+ ","
+ str( chnydata[row])
+ "\n" )
`
but getting this error...
number of channels: 6
Traceback (most recent call last):
File "C:\Users\guptap15\Desktop\RAW\Raw4.py", line 20, in
@Priyanka260895 on https://github.com/RecordEvolution/IMCtermite/commit/ef0bb7550d4845bc4481783e22e885a90b62e2d6#commitcomment-100284938 : Have you tried installing the latest version 2.0.6 of the package (https://pypi.org/project/IMCtermite/) ? Any of the given Python examples should be appropriate to decode a multichannel dataset. Are you able to share a sample file of the data you're trying to read?
@mario-fink Multi-channel file like in the above comment measurement.raw is having multiple channels and that means when it will be converted to csv every channel will have a different column. But after updating the package, that conversion to csv is happening but data of only one channel is being generated. I'm stuck because I'm having raw files with multiple channels which if we convert to csv should have multiple columns. Due to some reasons, I'm not able to share the raw file with you. But if you find some solution to the problem I have explained, then it would be of great help.
In order to extract the different channels of the above Measurement.raw file you might, for instance, use the following snippet:
import IMCtermite
import pandas
if __name__ == "__main__" :
imctm = IMCtermite.imctermite(b"Measurement.raw")
chns = imctm.get_channels(True)
df = pandas.DataFrame()
for idx,chn in enumerate(chns) :
xcol = (chn['xname'] if chn['xname'] != '' else "x_"+str(idx))+" ["+chn['xunit']+"]"
df[xcol] = pandas.Series(chn['xdata'])
ycol = chn['yname']+" ["+chn['yunit']+"]"
df[ycol] = pandas.Series(chn['ydata'])
print(df)
df.to_csv("Measurement.csv",header=True,sep='\t')
that writes all 12 channels and their specific dependencies in the time domain in individual columns.
@mario-fink Thankyou for your Continuous support. It is work fine but, I have converted "Measurement.raw" file into csv format using FAMOS software and I am getting the data with timestamp in first column.
Data example: meas.csv
could we get the output csv file in the same format (with time stamp) as given above?
@Priyanka260895 Please check out the new Python example in order to get the requested output format. And don't forget to upgrade to version v2.0.7 :)
@mario-fink Thankyou I have updated the version but for given code output is as below:-
time [s] AI_a01_x [m/s^2] ... AI_p05 [bar] AI_p06 [bar]
0 0.00000 -0.119239 ... 1.480193 1.022401 1 0.00005 -0.124917 ... 1.480193 1.037661 2 0.00010 -0.068136 ... 1.464933 1.022401 3 0.00015 -0.096527 ... 1.419154 0.976622 4 0.00020 -0.238478 ... 1.464933 0.976622 ... ... ... ... ... ... 112791 5.63955 0.198731 ... 1.525972 1.052921 112792 5.63960 0.102205 ... 1.525972 1.068180 112793 5.63965 0.193053 ... 1.449673 1.083440 112794 5.63970 0.227122 ... 1.434414 1.052921 112795 5.63975 0.153307 ... 1.480193 1.068180
[112796 rows x 13 columns]
but is it possible to change the time column value as date & time as below:- 17-02-2023 22:04:22
To get the full timestamp and a comma delimiter you might consider using:
import IMCtermite
import pandas
import datetime
def add_trigger_time(trigger_time, add_time) :
trgts = datetime.datetime.strptime(trigger_time,'%Y-%m-%dT%H:%M:%S')
dt = datetime.timedelta(seconds=add_time)
return (trgts + dt).strftime('%Y-%m-%dT%H:%M:%S:%f')
if __name__ == "__main__" :
# read file and extract data
imctm = IMCtermite.imctermite(b"Measurement.raw")
chns = imctm.get_channels(True)
# prepare abscissa
xcol = "timestamp ['%Y-%m-%dT%H:%M:%S:%f']"
xsts = [add_trigger_time(chns[0]['trigger-time'],tm) for tm in chns[0]['xdata']]
# sort channels
chnnms = sorted([chn['name'] for chn in chns], reverse=False)
chnsdict = {}
for chn in chns :
chnsdict[chn['name']] = chn
# construct dataframe
df = pandas.DataFrame()
df[xcol] = pandas.Series(xsts)
for chnnm in chnnms :
chn = chnsdict[chnnm]
ycol = chn['yname']+" ["+chn['yunit']+"]"
df[ycol] = pandas.Series(chn['ydata'])
# show entire dataframe and write file
print(df)
df.to_csv("Measurement.csv",header=True,sep=',',index=False)
resulting in
timestamp ['%Y-%m-%dT%H:%M:%S:%f'],AI_a01_x [m/s^2],AI_a01_y [m/s^2],AI_a01_z [m/s^2],AI_a02_x [m/s^2],AI_a02_y [m/s^2],AI_a02_z [m/s^2],AI_p01 [bar],AI_p02 [bar],AI_p03 [bar],AI_p04 [bar],AI_p05 [bar],AI_p06 [bar]
2022-08-30T11:40:21:000000,-0.119238804,-0.034912617,-0.070025491,-0.006025557,0.014546924,0.0,0.0,-0.152597204,0.137337484,0.0,1.480192883,1.02240127
2022-08-30T11:40:21:000050,-0.124916842,-0.058187695,-0.075860948,-0.003012778,0.014546924,0.002928929,-0.030519441,-0.076298602,0.167856925,0.0,1.480192883,1.03766099
2022-08-30T11:40:21:000100,-0.06813646,-0.069825235,0.0,-0.003012778,0.014546924,0.0,0.0,-0.01525972,0.167856925,0.01525972,1.464933162,1.02240127
2022-08-30T11:40:21:000150,-0.096526651,-0.069825235,-0.040848203,-0.006025557,0.014546924,0.0,-0.01525972,-0.030519441,0.106818043,0.030519441,1.419154001,0.976622108
2022-08-30T11:40:21:000200,-0.238477608,0.011637539,-0.029177288,-0.003012778,0.014546924,0.0,-0.01525972,0.01525972,0.091558323,-0.030519441,1.464933162,0.976622108
Simply adjust the time format string '%Y-%m-%dT%H:%M:%S:%f' according to your needs.
@Priyanka260895
@mario-fink Hi, could you please help me to this issues it would be great help. https://github.com/blackary/st_pages/issues/27#issue-1630263816