UnROOT.jl
UnROOT.jl copied to clipboard
JuliaHEP
TH1_3 not defined
I use for the first time and I try to read a TH1F histogram produced with GEANT4 But I got the following issue:
julia> f = ROOTFile("/Users/lebrun/Desktop/dedx2COMET.root") ROOTFile with 1 entry and 53 streamers. /Users/lebrun/Desktop/dedx2COMET.root └─ h1 (TH1D)
julia> f["h1"] ERROR: UndefVarError: TH1_3 not defined
Did I make a trivial error ?
https://github.com/JuliaHEP/UnROOT.jl/blob/7d3712202b0c9bc975937dac5258e20321c3659d/src/bootstrap.jl#L79-L85
I think we just don't have bootstrap for that type here
idk how this works, looks like it does literally nothing, but if you want, try adding these two lines
struct TH1_3 <: TH1 end
function readfields!(io, fields, T::Type{TH1_3}) end
I added the two lines from Moelf in bootstrap.jl which fixed this issue but I need to do the same sort of thing for many other types (TAxis_ , TAttAxis_ TNamed_ , ...) and at the end I got the following error: ERROR: Object 'UnROOT.TAttAxis' has 40 bytes; expected 3180234308
(geant4-v11.0.1 , OS: Ubuntu 20.04.4 LTS)
Hi Jerry,
Here is this very simple root file having only one TH1D histogram which I can’t read with UnROOT
Le 11 mai 2022 à 17:33, Jerry Ling @.***> a écrit :
can you send an example file?
— Reply to this email directly, view it on GitHub https://github.com/JuliaHEP/UnROOT.jl/issues/168#issuecomment-1123936618, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKNFOYO5JTEIRLMGXPFAQP3VJPHL5ANCNFSM5VSACI5A. You are receiving this because you authored the thread.
What do you mean by you can’t see the file ? I attached it again in this email.
Le 11 mai 2022 à 19:50, Jerry Ling @.***> a écrit :
sorry I can't see your file
— Reply to this email directly, view it on GitHub https://github.com/JuliaHEP/UnROOT.jl/issues/168#issuecomment-1124082763, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKNFOYNCWXTOLJS3AGG6NXTVJPXNJANCNFSM5VSACI5A. You are receiving this because you authored the thread.
you're not sending email to me directly, see: https://github.com/JuliaHEP/UnROOT.jl/issues/168
Oups sorry. I just send a .gz file
Le 11 mai 2022 à 20:03, Jerry Ling @.***> a écrit :
you're not sending email to me directly, see: #168 https://github.com/JuliaHEP/UnROOT.jl/issues/168 — Reply to this email directly, view it on GitHub https://github.com/JuliaHEP/UnROOT.jl/issues/168#issuecomment-1124105385, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKNFOYIZKBWUTD66RTGCEYDVJPY6DANCNFSM5VSACI5A. You are receiving this because you authored the thread.
looks like this is the difference between
# this issue
In [3]: f = up.open("./dedx2COMET.root")
In [4]: f.keys()
Out[4]: ['h1;1']
In [5]: f["h1"]
Out[5]: <TH1D (version 1) at 0x7f07b281ce20>
In [6]: f["h1"]
Out[6]: <TH1D (version 1) at 0x7f07b281ce20>
# what UnROOT.jl knows how to handle
In [4]: f["myTH1D"]
Out[4]: <TH1D (version 2) at 0x7fa9a22892b0>
I don't have experience in hunting down different internal version of ROOT object, asking @tamasgal
In case it helps, you could try to copy the current structs and modify them based on these differences
root [27] TH1::Class()->GetStreamerInfo(3)->ls()
StreamerInfo for class: TH1, version=3, checksum=0x5668b3d7
TNamed BASE offset= 0 type=67 The basis for a named object (name, title)
TAttLine BASE offset= 64 type= 0 Line attributes
TAttFill BASE offset= 80 type= 0 Fill area attributes
TAttMarker BASE offset= 96 type= 0 Marker attributes
Int_t fNcells offset=112 type= 3 number of bins(1D), cells (2D) +U/Overflows
TAxis fXaxis offset=120 type=61 X axis descriptor
TAxis fYaxis offset=336 type=61 Y axis descriptor
TAxis fZaxis offset=552 type=61 Z axis descriptor
Short_t fBarOffset offset=768 type= 2 (1000*offset) for bar charts or legos
Short_t fBarWidth offset=770 type= 2 (1000*width) for bar charts or legos
Stat_t fEntries offset=776 type= 8 Number of entries
Stat_t fTsumw offset=784 type= 8 Total Sum of weights
Stat_t fTsumw2 offset=792 type= 8 Total Sum of squares of weights
Stat_t fTsumwx offset=800 type= 8 Total Sum of weight*X
Stat_t fTsumwx2 offset=808 type= 8 Total Sum of weight*X*X
Double_t fMaximum offset=816 type= 8 Maximum value for plotting
Double_t fMinimum offset=824 type= 8 Minimum value for plotting
Double_t fNormFactor offset=832 type= 8 Normalization factor
TArrayD fContour offset=840 type=62 Array to display contour levels
TArrayD fSumw2 offset=864 type=62 Array of sum of squares of weights
TString fOption offset=888 type=65 histogram options
TList* fFunctions offset=912 type=63 ->Pointer to list of functions (fits and user)
i= 0, TNamed type= 67, offset= 0, len=1, method=0
i= 1, TAttLine type= 0, offset= 64, len=1, method=0
i= 2, TAttFill type= 0, offset= 80, len=1, method=0
i= 3, TAttMarker type= 0, offset= 96, len=1, method=0
i= 4, fNcells type= 3, offset=112, len=1, method=0
i= 5, fXaxis type= 61, offset=120, len=1, method=0
i= 6, fYaxis type= 61, offset=336, len=1, method=0
i= 7, fZaxis type= 61, offset=552, len=1, method=0
i= 8, fBarOffset type= 22, offset=768, len=2, method=0 [optimized]
i= 9, fEntries type= 28, offset=776, len=8, method=0 [optimized]
i=10, fContour type= 62, offset=840, len=1, method=0
i=11, fSumw2 type= 62, offset=864, len=1, method=0
i=12, fOption type= 65, offset=888, len=1, method=0
i=13, fFunctions type= 63, offset=912, len=1, method=0
root [28] TH1::Class()->GetStreamerInfo(8)->ls()
StreamerInfo for class: TH1, version=8, checksum=0x1c3740c4
TNamed BASE offset= 0 type=67 The basis for a named object (name, title)
TAttLine BASE offset= 64 type= 0 Line attributes
TAttFill BASE offset= 80 type= 0 Fill area attributes
TAttMarker BASE offset= 96 type= 0 Marker attributes
int fNcells offset=112 type= 3 number of bins(1D), cells (2D) +U/Overflows
TAxis fXaxis offset=120 type=61 X axis descriptor
TAxis fYaxis offset=336 type=61 Y axis descriptor
TAxis fZaxis offset=552 type=61 Z axis descriptor
short fBarOffset offset=768 type= 2 (1000*offset) for bar charts or legos
short fBarWidth offset=770 type= 2 (1000*width) for bar charts or legos
double fEntries offset=776 type= 8 Number of entries
double fTsumw offset=784 type= 8 Total Sum of weights
double fTsumw2 offset=792 type= 8 Total Sum of squares of weights
double fTsumwx offset=800 type= 8 Total Sum of weight*X
double fTsumwx2 offset=808 type= 8 Total Sum of weight*X*X
double fMaximum offset=816 type= 8 Maximum value for plotting
double fMinimum offset=824 type= 8 Minimum value for plotting
double fNormFactor offset=832 type= 8 Normalization factor
TArrayD fContour offset=840 type=62 Array to display contour levels
TArrayD fSumw2 offset=864 type=62 Array of sum of squares of weights
TString fOption offset=888 type=65 histogram options
TList* fFunctions offset=912 type=63 ->Pointer to list of functions (fits and user)
int fBufferSize offset=920 type= 6 fBuffer size
double* fBuffer offset=928 type=48 [fBufferSize] entry buffer
TH1::EBinErrorOpt fBinStatErrOpt offset=968 type= 3 option for bin statistical errors
TH1::EStatOverflows fStatOverflows offset=972 type= 3 per object flag to use under/overflows in statistics
i= 0, TNamed type= 67, offset= 0, len=1, method=0
i= 1, TAttLine type= 0, offset= 64, len=1, method=0
i= 2, TAttFill type= 0, offset= 80, len=1, method=0
i= 3, TAttMarker type= 0, offset= 96, len=1, method=0
i= 4, fNcells type= 3, offset=112, len=1, method=0
i= 5, fXaxis type= 61, offset=120, len=1, method=0
i= 6, fYaxis type= 61, offset=336, len=1, method=0
i= 7, fZaxis type= 61, offset=552, len=1, method=0
i= 8, fBarOffset type= 22, offset=768, len=2, method=0 [optimized]
i= 9, fEntries type= 28, offset=776, len=8, method=0 [optimized]
i=10, fContour type= 62, offset=840, len=1, method=0
i=11, fSumw2 type= 62, offset=864, len=1, method=0
i=12, fOption type= 65, offset=888, len=1, method=0
i=13, fFunctions type= 63, offset=912, len=1, method=0
i=14, fBufferSize type= 6, offset=920, len=1, method=84435080
i=15, fBuffer type= 48, offset=928, len=1, method=920
i=16, fBinStatErrOpt type= 23, offset=968, len=2, method=0 [optimized]
and similar for TAxis 6 vs 10. I didn't get further than that because of an EOF error :(
I'm currently experiencing the same situation with TH1_6 not defined and wanted to check if there has been any progress / new input regarding this. As above just copy pasting the struct for TAxis gives me an EOF error as well, I've attached the .root file as a .zip just in case. DataSample_search.zip
Can you easily check how old those root files are? I feel like we must haven't experience this because we have been testing with new files
They are at least from 2013 since they are part of the supplementary material of a statistics book we are working with that got published in that year I can ask if there's a more exact date but I doubt it, so your guess is likely correct since you mentionted most tests were made with NanoAOD files.
Sorry for the very late response, I was on holidays and then had a lot of teaching to do.
As @aminnj wrote, the GetStreamerInfo() is very helpful but I also used uproot in the past successfully to create the parser logic and compare between versions. That being said, I'd propose to check how uproot handles it. There are still some unresolved mysteries regarding old files, so maybe we are on hitting some higher barriers there ;) I will try to free up some time and look at your file.
Thanks for responding anyway and taking the time to look at it, since I'm currently working with a workaround it's not urgent but I appreciate it! Tamas Gal @.***> schrieb am Di., 21. Juni 2022, 09:53:
Sorry for the very late response, I was on holidays and then had a lot of teaching to do.
As @aminnj https://github.com/aminnj wrote, the GetStreamerInfo() is very helpful but I also used uproot in the past successfully to create the parser logic and compare between versions. That being said, I'd propose to check how uproot handles it. There are still some unresolved mysteries regarding old files, so maybe we are on hitting some higher barriers there ;) I will try to free up some time and look at your file.
— Reply to this email directly, view it on GitHub https://github.com/JuliaHEP/UnROOT.jl/issues/168#issuecomment-1161390537, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALY43GIYNLBQX6PTXXOVBBLVQFYI7ANCNFSM5VSACI5A . You are receiving this because you commented.Message ID: <JuliaHEP/UnROOT. @.***>
Thanks Jan. We really need a bit more contributors 🙈
I'll try to look at this tomorrow (I have another issue as draft for a week or so...)
Thanks! I'd be interested in the workaround that @Momox357 has mentioned. At the moment I'm a bit stuck.
You might be a bit disappointed in that case! Since I only had a few histograms that had this problem I wrote a ROOT script that exported the relevant data (bin content and error) to a .txt and imported that into a new histogram that I could continue working with. "Workaround" might have been a generous term for that.
I just had a look at the file @Momox357 has provided and I even bump into a parsing error in uproot:
>>> f = uproot.open("dedx2COMET.root")
>>> h1 = f["h1"]
>>> h1.to_hist()
---------------------------------------------------------------------------
KeyInFileError Traceback (most recent call last)
Cell In [18], line 1
----> 1 h1.to_hist()
File ~/Dev/UnROOT.jl/venv/lib/python3.10/site-packages/uproot/behaviors/TH1.py:206, in Histogram.to_hist(self, metadata, axis_metadata)
195 def to_hist(self, metadata=boost_metadata, axis_metadata=boost_axis_metadata):
196 """
197 Args:
198 metadata (dict of str \u2192 str): Metadata to collect (keys) and
(...)
203 Converts the histogram into a ``hist`` object.
204 """
205 return uproot.extras.hist().Hist(
--> 206 self.to_boost(metadata=boost_metadata, axis_metadata=boost_axis_metadata)
207 )
File ~/Dev/UnROOT.jl/venv/lib/python3.10/site-packages/uproot/behaviors/TH1.py:312, in TH1.to_boost(self, metadata, axis_metadata)
309 else:
310 storage = boost_histogram.storage.Double()
--> 312 xaxis = _boost_axis(self.member("fXaxis"), axis_metadata)
313 out = boost_histogram.Histogram(xaxis, storage=storage)
314 for k, v in metadata.items():
File ~/Dev/UnROOT.jl/venv/lib/python3.10/site-packages/uproot/behaviors/TH1.py:23, in _boost_axis(axis, metadata)
20 fNbins = axis.member("fNbins")
21 fXbins = axis.member("fXbins", none_if_missing=True)
---> 23 if axis.member("fLabels") is not None:
24 out = boost_histogram.axis.StrCategory([str(x) for x in axis.member("fLabels")])
26 elif fXbins is None or len(fXbins) != fNbins + 1:
File ~/Dev/UnROOT.jl/venv/lib/python3.10/site-packages/uproot/model.py:561, in Model.member(self, name, all, none_if_missing)
559 return None
560 else:
--> 561 raise uproot.KeyInFileError(
562 name,
563 because="""{}.{} has only the following members:
564
565 {}
566 """.format(
567 type(self).__module__,
568 type(self).__name__,
569 ", ".join(repr(x) for x in self.all_members),
570 ),
571 file_path=getattr(self._file, "file_path", None),
572 )
KeyInFileError: not found: 'fLabels' because uproot.dynamic.Model_TAxis_v6 has only the following members:
'@fUniqueID', '@fBits', 'fName', 'fTitle', 'fNdivisions', 'fAxisColor', 'fLabelColor', 'fLabelFont', 'fLabelOffset', 'fLabelSize', 'fTickLength', 'fTitleOffset', 'fTitleSize', 'fTitleColor', 'fTitleFont', 'fNbins', 'fXmin', 'fXmax', 'fXbins', 'fFirst', 'fLast', 'fTimeDisplay', 'fTimeFormat'
in file dedx2COMET.root
I think that also uproot tries to simply parse data with possibly incompatible streamer versions and either it works -- which means it should be OK since there are several sanity checks in the ROOT format itself -- or produces some errors which indicate that some versions are not compatible.
I tried to teach UnROOT that TH1_3 is TH1_8 but I think the problem is much more deeper.
After working myself through all the streamer versions I encountered (like TAxis_1, TAxis_6, TAttMarker_1, TAttMarker_6 and so on), which is anyways very weird: why would a file use different versions of streamers? Was it opened for writing with different ROOT version? I finally hit this:
julia> f["h1"]
ERROR: UndefVarError: TAttAxis_8321 not defined
Which can only be some parsing error, unless the file is from the post-alien era 😉
OK I have a bit more information.
I am now working with the file @jstrube provided in https://github.com/JuliaHEP/UnROOT.jl/issues/168#issuecomment-1306108550
When I add TAttMarker_1 as an alias to TAttMarker_2 I get a bit further and the parsing eventually chokes at the fIOFeatures step, which is simply missing since the preamble version is <20. I need to figure out how to parse earlier versions:
julia> using UnROOT
julia> f = ROOTFile("t.root");
julia> ENV["JULIA_DEBUG"] = "UnROOT";
julia> f["Primaries/px1"]
┌ Debug: Splitting path 'Primaries/px1' and getting items recursively
└ @ UnROOT ~/Dev/UnROOT.jl/src/root.jl:152
┌ Debug: Retrieving Primaries ('TTree')
└ @ UnROOT ~/Dev/UnROOT.jl/src/root.jl:157
┌ Debug: TTree: UnROOT.TKey32(2965, 2, 22758, 0x00000000, 73, 1, 5381164, 64, "TTree", "Primaries", "Step-by-step energy deposition")
└ @ UnROOT ~/Dev/UnROOT.jl/src/bootstrap.jl:899
┌ Debug: Compression type: ZL
└ @ UnROOT ~/Dev/UnROOT.jl/src/types.jl:154
┌ Debug: Compressed/uncompressed size in bytes: 2883 / 22758
└ @ UnROOT ~/Dev/UnROOT.jl/src/types.jl:155
┌ Debug: fIOFeatures is missing
└ @ UnROOT ~/Dev/UnROOT.jl/src/bootstrap.jl:955
┌ Debug: Reading branches
└ @ UnROOT ~/Dev/UnROOT.jl/src/bootstrap.jl:959
ERROR: EOFError: read end of file
So I guess this is where to search first: https://github.com/JuliaHEP/UnROOT.jl/blob/57e49d3ac52a77174094d1e212b215268a768e8b/src/bootstrap.jl#L944
but this might not be the end of the story ;)
Well, this looks like a different issue to me, somehow a recursion error when inside the TTree the same TTree is parsed again. Below you see the output with a bit more debug.
Maybe the problem is that these branches are sitting at the top of the ROOT file, without any TDirectory or so, but I am now guessing (I am close to 15 hours of work today with just a few tiny interruptions 🙈)
julia> f["pfates"]
┌ Debug: Retrieving pfates ('TTree')
└ @ UnROOT ~/Dev/UnROOT.jl/src/root.jl:157
┌ Debug: TTree: UnROOT.TKey32(575673, 2, 1258332, 0x00000000, 70, 1, 5844079, 64, "TTree", "pfates", "Step-by-step energy deposition")
└ @ UnROOT ~/Dev/UnROOT.jl/src/bootstrap.jl:899
┌ Debug: Compression type: ZL
└ @ UnROOT ~/Dev/UnROOT.jl/src/types.jl:154
┌ Debug: Compressed/uncompressed size in bytes: 575594 / 1258332
└ @ UnROOT ~/Dev/UnROOT.jl/src/types.jl:155
┌ Debug: fIOFeatures is missing
└ @ UnROOT ~/Dev/UnROOT.jl/src/bootstrap.jl:955
┌ Debug: Reading branches
└ @ UnROOT ~/Dev/UnROOT.jl/src/bootstrap.jl:959
┌ Debug: Unpacking: UnROOT.TKey32(575673, 2, 1258332, 0x00000000, 70, 1, 5844079, 64, "TTree", "pfates", "Step-by-step energy deposition")
└ @ UnROOT ~/Dev/UnROOT.jl/src/streamers.jl:312
ERROR: EOFError: read end of file
Stacktrace:
Hi, is there maybe any update on this topic?
I have a (rather new) .root file and get the error UndefVarError: TAttMarker_1 not defined, which seems to be connected to this issue here.
(Wanted to show people how much nicer it would be to use Julia than ROOT/PyROOT. But this didn't work out since already reading the file fails 😖)
Hi @Cornelius-G ! I have not made any progress but if you could provide a small sample file, that would already increase the sample size greatly ;)
hi @Cornelius-G , I think we're mainly much more ergonomic when dealing with TTree or RNTuple.
Btw, you can try using uproot via PythonCall.jl or PyCall.jl for reading histograms, since they're juts one-time reading of small data, doesn't involve looping.
oh and it would be nice to have a sample .root file with TAttMarker_1
Unfortunately, the original file is quite large and should be looped over.
I attached a smaller sample of the file that also gives the TAttMarker_1 error when running:
f = ROOTFile("data.root")
f["proton"]
I'm a bit confused what is _1:
https://github.com/root-project/root/blob/4ef94f4432a39ef5542cc4f00d46f079115b263d/core/base/inc/TAttMarker.h#LL50C9-L50C9
https://github.com/scikit-hep/uproot5/blob/d6d311aef4e22cc73ae056665dbb41cc18b2d6ac/src/uproot/models/TAtt.py#L328
as far as I can tell the only known version is _2 although, of course, Uproot doesn't have trouble opening this file so there must be something we can do as well