hist icon indicating copy to clipboard operation
hist copied to clipboard

[FEATURE] Read and write Yoda (and HEPData) ASCII file formats

Open jpivarski opened this issue 4 years ago • 3 comments

This can be an easy way to enable interoperability with software in the HEP-theory/reinterpreation world and also provides an already-standard human readable serialization format for some histogram types. YODA histograms have been in continuous use, mainly by the theory community, since 2008:

https://yoda.hepforge.org/

One of the features its users enjoy is the ability to read histogram data manually as an ASCII file. There are two formats (discounting the ancient AIDA format): YODA and FLAT, both of which are space-delimited text with some headings. The following script (which, I believe, includes all of the analysis object types):

import yoda
import random

random.seed(42)

h1d = yoda.Histo1D(10, -5, 5, "/some_h1d", "title")
h1d.setAnnotation("some", "stuff")
for _ in range(1000):
    h1d.fill(random.gauss(0, 1))

random.seed(42)

h2d = yoda.Histo2D(3, -4, 4, 5, -6, 6, "/some_h2d", "title")
h2d.setAnnotation("some", "stuff")
for _ in range(1000):
    h2d.fill(random.gauss(0, 1), random.uniform(-5, 5))

random.seed(42)

p1d = yoda.Profile1D(10, -5, 5, "/some_p1d", "title")
p1d.setAnnotation("some", "stuff")
for _ in range(1000):
    p1d.fill(random.gauss(0, 1), random.uniform(-5, 5))

random.seed(42)

p2d = yoda.Profile2D(3, -4, 4, 5, -6, 6, "/some_p2d", "title")
p2d.setAnnotation("some", "stuff")
for _ in range(1000):
    p2d.fill(random.gauss(0, 1), random.uniform(-5, 5), random.uniform(-5, 5))

random.seed(42)

s2d = yoda.Scatter2D([
    (random.gauss(0, 1), random.uniform(-5, 5)) for _ in range(10)
], "/some_s2d", "title")
s2d.setAnnotation("some", "stuff")

random.seed(42)

s3d = yoda.Scatter2D([
    (random.gauss(0, 1), random.uniform(-5, 5), random.gauss(1, 1))
    for _ in range(10)
], "/some_s3d", "title")
s3d.setAnnotation("some", "stuff")

yoda.writeYODA([h1d, h2d, p1d, p2d, s2d, s3d], "data.yoda")
yoda.writeFLAT([h1d, h2d, p1d, p2d, s2d, s3d], "data.flat")

produces the following YODA file:

BEGIN YODA_HISTO1D_V2 /some_h1d
Path: /some_h1d
Title: title
Type: Histo1D
some: stuff
---
# Mean: -1.826931e-02
# Area: 1.000000e+03
# ID	 ID	 sumw	 sumw2	 sumwx	 sumwx2	 numEntries
Total   	Total   	1.000000e+03	1.000000e+03	-1.826931e+01	1.038888e+03	1.000000e+03
Underflow	Underflow	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00
Overflow	Overflow	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00
# xlow	 xhigh	 sumw	 sumw2	 sumwx	 sumwx2	 numEntries
-5.000000e+00	-4.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00
-4.000000e+00	-3.000000e+00	1.000000e+00	1.000000e+00	-3.154543e+00	9.951142e+00	1.000000e+00
-3.000000e+00	-2.000000e+00	2.600000e+01	2.600000e+01	-5.880533e+01	1.343327e+02	2.600000e+01
-2.000000e+00	-1.000000e+00	1.440000e+02	1.440000e+02	-1.983116e+02	2.839175e+02	1.440000e+02
-1.000000e+00	0.000000e+00	3.380000e+02	3.380000e+02	-1.537316e+02	9.678831e+01	3.380000e+02
0.000000e+00	1.000000e+00	3.380000e+02	3.380000e+02	1.536739e+02	9.607486e+01	3.380000e+02
1.000000e+00	2.000000e+00	1.270000e+02	1.270000e+02	1.803327e+02	2.663863e+02	1.270000e+02
2.000000e+00	3.000000e+00	2.200000e+01	2.200000e+01	4.844410e+01	1.070365e+02	2.200000e+01
3.000000e+00	4.000000e+00	4.000000e+00	4.000000e+00	1.328298e+01	4.440106e+01	4.000000e+00
4.000000e+00	5.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00
END YODA_HISTO1D_V2


BEGIN YODA_HISTO2D_V2 /some_h2d
Path: /some_h2d
Title: title
Type: Histo2D
some: stuff
---
# Mean: (-9.993362e-03, 1.477861e-01)
# Volume: 1.000000e+03
# ID	 ID	 sumw	 sumw2	 sumwx	 sumwx2	 sumwy	 sumwy2	 sumwxy	 numEntries
Total   	Total   	1.000000e+03	1.000000e+03	-9.993362e+00	9.959904e+02	1.477861e+02	8.079625e+03	-6.349343e+01	1.000000e+03
# 2D outflow persistency not currently supported until API is stable
# xlow	 xhigh	 ylow	 yhigh	 sumw	 sumw2	 sumwx	 sumwx2	 sumwy	 sumwy2	 sumwxy	 numEntries
-4.000000e+00	-1.333333e+00	-6.000000e+00	-3.600000e+00	1.200000e+01	1.200000e+01	-2.053749e+01	3.618178e+01	-5.147380e+01	2.219949e+02	8.806086e+01	1.200000e+01
-4.000000e+00	-1.333333e+00	-3.600000e+00	-1.200000e+00	2.400000e+01	2.400000e+01	-4.279648e+01	8.135725e+01	-6.352660e+01	1.778829e+02	1.106406e+02	2.400000e+01
-4.000000e+00	-1.333333e+00	-1.200000e+00	1.200000e+00	2.100000e+01	2.100000e+01	-3.787008e+01	7.044087e+01	-8.233215e-01	1.002409e+01	8.671186e-01	2.100000e+01
-4.000000e+00	-1.333333e+00	1.200000e+00	3.600000e+00	2.400000e+01	2.400000e+01	-4.197474e+01	7.661144e+01	5.678960e+01	1.453823e+02	-1.007723e+02	2.400000e+01
-4.000000e+00	-1.333333e+00	3.600000e+00	6.000000e+00	1.500000e+01	1.500000e+01	-2.561424e+01	4.576702e+01	6.289810e+01	2.664811e+02	-1.071839e+02	1.500000e+01
-1.333333e+00	1.333333e+00	-6.000000e+00	-3.600000e+00	1.060000e+02	1.060000e+02	1.259149e+01	5.026012e+01	-4.501826e+02	1.929574e+03	-5.467934e+01	1.060000e+02
-1.333333e+00	1.333333e+00	-3.600000e+00	-1.200000e+00	1.830000e+02	1.830000e+02	-1.884376e-01	8.649397e+01	-4.348038e+02	1.112851e+03	1.024296e+01	1.830000e+02
-1.333333e+00	1.333333e+00	-1.200000e+00	1.200000e+00	1.940000e+02	1.940000e+02	-1.017305e+01	9.403036e+01	1.116095e+01	8.609351e+01	6.896002e+00	1.940000e+02
-1.333333e+00	1.333333e+00	1.200000e+00	3.600000e+00	2.110000e+02	2.110000e+02	-4.754021e+00	9.867403e+01	5.006934e+02	1.280025e+03	-6.344411e+00	2.110000e+02
-1.333333e+00	1.333333e+00	3.600000e+00	6.000000e+00	1.220000e+02	1.220000e+02	6.594140e-01	4.569665e+01	5.273801e+02	2.303101e+03	2.122680e+00	1.220000e+02
1.333333e+00	4.000000e+00	-6.000000e+00	-3.600000e+00	1.100000e+01	1.100000e+01	2.024581e+01	3.995801e+01	-4.512890e+01	1.862778e+02	-8.442214e+01	1.100000e+01
1.333333e+00	4.000000e+00	-3.600000e+00	-1.200000e+00	2.100000e+01	2.100000e+01	3.557867e+01	6.210986e+01	-4.206759e+01	9.183817e+01	-7.170194e+01	2.100000e+01
1.333333e+00	4.000000e+00	-1.200000e+00	1.200000e+00	2.800000e+01	2.800000e+01	5.274324e+01	1.059118e+02	-4.278838e+00	1.319808e+01	-7.736072e+00	2.800000e+01
1.333333e+00	4.000000e+00	1.200000e+00	3.600000e+00	2.300000e+01	2.300000e+01	4.250192e+01	8.346248e+01	6.089813e+01	1.723366e+02	1.117016e+02	2.300000e+01
1.333333e+00	4.000000e+00	3.600000e+00	6.000000e+00	5.000000e+00	5.000000e+00	9.594602e+00	1.903482e+01	2.025123e+01	8.256478e+01	3.881484e+01	5.000000e+00
END YODA_HISTO2D_V2


BEGIN YODA_PROFILE1D_V2 /some_p1d
Path: /some_p1d
Title: title
Type: Profile1D
some: stuff
---
# ID	 ID	 sumw	 sumw2	 sumwx	 sumwx2	 sumwy	 sumwy2	 numEntries
Total   	Total   	1.000000e+03	1.000000e+03	-9.993362e+00	9.959904e+02	1.477861e+02	8.079625e+03	1.000000e+03
Underflow	Underflow	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00
Overflow	Overflow	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00
# xlow	 xhigh	 sumw	 sumw2	 sumwx	 sumwx2	 sumwy	 sumwy2	 numEntries
-5.000000e+00	-4.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00
-4.000000e+00	-3.000000e+00	1.000000e+00	1.000000e+00	-3.162596e+00	1.000201e+01	-2.144231e+00	4.597728e+00	1.000000e+00
-3.000000e+00	-2.000000e+00	2.800000e+01	2.800000e+01	-6.175264e+01	1.370437e+02	1.108863e+01	2.310019e+02	2.800000e+01
-2.000000e+00	-1.000000e+00	1.350000e+02	1.350000e+02	-1.818558e+02	2.532532e+02	-4.786425e+00	1.046848e+03	1.350000e+02
-1.000000e+00	0.000000e+00	3.410000e+02	3.410000e+02	-1.540762e+02	9.764065e+01	1.012868e+02	2.849965e+03	3.410000e+02
0.000000e+00	1.000000e+00	3.390000e+02	3.390000e+02	1.499775e+02	9.238634e+01	8.722722e+01	2.837567e+03	3.390000e+02
1.000000e+00	2.000000e+00	1.360000e+02	1.360000e+02	1.914041e+02	2.794030e+02	-5.243371e+01	9.763814e+02	1.360000e+02
2.000000e+00	3.000000e+00	1.800000e+01	1.800000e+01	4.250578e+01	1.018512e+02	1.054310e+01	1.080355e+02	1.800000e+01
3.000000e+00	4.000000e+00	2.000000e+00	2.000000e+00	6.966450e+00	2.441025e+01	-2.995309e+00	2.522879e+01	2.000000e+00
4.000000e+00	5.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00
END YODA_PROFILE1D_V2


BEGIN YODA_PROFILE2D_V2 /some_p2d
Path: /some_p2d
Title: title
Type: Profile2D
some: stuff
---
# sumw	 sumw2	 sumwx	 sumwx2	 sumwy	 sumwy2	 sumwz	 sumwz2	 sumwxy	 numEntries
Total   	Total   	1.000000e+03	1.000000e+03	2.992019e+01	9.641736e+02	9.285495e+00	8.216848e+03	1.242772e+02	8.265476e+03	6.070245e+01	1.000000e+03
# 2D outflow persistency not currently supported until API is stable
# xlow	 xhigh	 ylow	 yhigh	 sumw	 sumw2	 sumwx	 sumwx2	 sumwy	 sumwy2	 sumwz	 sumwz2	 sumwxy	 numEntries
-4.000000e+00	-1.333333e+00	-6.000000e+00	-3.600000e+00	1.200000e+01	1.200000e+01	-2.022418e+01	3.550942e+01	-5.150942e+01	2.221012e+02	-2.496849e+00	8.482240e+01	8.685983e+01	1.200000e+01
-4.000000e+00	-1.333333e+00	-3.600000e+00	-1.200000e+00	2.600000e+01	2.600000e+01	-4.531300e+01	8.250807e+01	-5.795812e+01	1.418539e+02	1.271776e+00	2.632191e+02	9.978354e+01	2.600000e+01
-4.000000e+00	-1.333333e+00	-1.200000e+00	1.200000e+00	1.600000e+01	1.600000e+01	-2.649263e+01	4.517885e+01	3.019582e+00	7.917567e+00	4.801062e+00	1.302075e+02	-6.565159e+00	1.600000e+01
-4.000000e+00	-1.333333e+00	1.200000e+00	3.600000e+00	2.100000e+01	2.100000e+01	-3.595198e+01	6.384077e+01	4.644698e+01	1.114367e+02	-1.498958e+01	2.313941e+02	-8.176495e+01	2.100000e+01
-4.000000e+00	-1.333333e+00	3.600000e+00	6.000000e+00	9.000000e+00	9.000000e+00	-1.788020e+01	3.704986e+01	3.819655e+01	1.634458e+02	-2.133972e+01	8.345975e+01	-7.594331e+01	9.000000e+00
-1.333333e+00	1.333333e+00	-6.000000e+00	-3.600000e+00	1.210000e+02	1.210000e+02	7.321768e+00	6.021643e+01	-5.175785e+02	2.233933e+03	-2.443569e+01	9.877241e+02	-2.656332e+01	1.210000e+02
-1.333333e+00	1.333333e+00	-3.600000e+00	-1.200000e+00	1.890000e+02	1.890000e+02	1.865504e+00	8.566063e+01	-4.458311e+02	1.140151e+03	5.817582e+01	1.450121e+03	4.774833e+00	1.890000e+02
-1.333333e+00	1.333333e+00	-1.200000e+00	1.200000e+00	1.970000e+02	1.970000e+02	3.350589e+00	9.017797e+01	-3.607306e+00	8.798352e+01	1.641253e+01	1.503640e+03	8.033808e+00	1.970000e+02
-1.333333e+00	1.333333e+00	1.200000e+00	3.600000e+00	2.180000e+02	2.180000e+02	4.771243e-01	9.666391e+01	5.208674e+02	1.349799e+03	2.175810e+01	1.756886e+03	6.722870e+00	2.180000e+02
-1.333333e+00	1.333333e+00	3.600000e+00	6.000000e+00	1.030000e+02	1.030000e+02	-1.605816e+00	4.372516e+01	4.511406e+02	1.994447e+03	-8.586562e+00	9.017917e+02	-1.079378e+01	1.030000e+02
1.333333e+00	4.000000e+00	-6.000000e+00	-3.600000e+00	1.100000e+01	1.100000e+01	1.987418e+01	3.849468e+01	-4.553647e+01	1.901479e+02	7.290011e+00	1.199587e+02	-8.308561e+01	1.100000e+01
1.333333e+00	4.000000e+00	-3.600000e+00	-1.200000e+00	2.200000e+01	2.200000e+01	4.132088e+01	8.032287e+01	-5.142289e+01	1.313103e+02	1.151509e+01	1.521926e+02	-9.595620e+01	2.200000e+01
1.333333e+00	4.000000e+00	-1.200000e+00	1.200000e+00	2.000000e+01	2.000000e+01	3.676646e+01	7.078044e+01	5.410910e+00	9.484978e+00	2.237384e+01	2.092782e+02	1.107969e+01	2.000000e+01
1.333333e+00	4.000000e+00	1.200000e+00	3.600000e+00	1.900000e+01	1.900000e+01	3.589762e+01	7.305764e+01	4.915609e+01	1.380157e+02	1.639107e+01	1.854429e+02	9.371743e+01	1.900000e+01
1.333333e+00	4.000000e+00	3.600000e+00	6.000000e+00	1.600000e+01	1.600000e+01	3.051387e+01	6.098689e+01	6.849116e+01	2.948216e+02	3.613633e+01	2.053375e+02	1.304028e+02	1.600000e+01
END YODA_PROFILE2D_V2


BEGIN YODA_SCATTER2D_V2 /some_s2d
ErrorBreakdown: {0: {"": {up: 0, dn: 0}}, 1: {"": {up: 0, dn: 0}}, 2: {"": {up: 0, dn: 0}}, 3: {"": {up: 0, dn: 0}}, 4: {"": {up: 0, dn: 0}}, 5: {"": {up: 0, dn: 0}}, 6: {"": {up: 0, dn: 0}}, 7: {"": {up: 0, dn: 0}}, 8: {"": {up: 0, dn: 0}}, 9: {"": {up: 0, dn: 0}}}
Path: /some_s2d
Title: title
Type: Scatter2D
some: stuff
---
# xval	 xerr-	 xerr+	 yval	 yerr-	 yerr+	
-1.497353e+00	0.000000e+00	0.000000e+00	-4.130612e+00	0.000000e+00	0.000000e+00
-2.169587e-01	0.000000e+00	0.000000e+00	-2.813620e+00	0.000000e+00	0.000000e+00
-1.729036e-01	0.000000e+00	0.000000e+00	-2.767893e+00	0.000000e+00	0.000000e+00
-1.440903e-01	0.000000e+00	0.000000e+00	-2.249707e+00	0.000000e+00	0.000000e+00
-1.275883e-01	0.000000e+00	0.000000e+00	3.921796e+00	0.000000e+00	0.000000e+00
1.105072e-01	0.000000e+00	0.000000e+00	4.494148e-01	0.000000e+00	0.000000e+00
1.158848e-01	0.000000e+00	0.000000e+00	5.355288e-02	0.000000e+00	0.000000e+00
2.463422e-01	0.000000e+00	0.000000e+00	3.094305e+00	0.000000e+00	0.000000e+00
6.566365e-01	0.000000e+00	0.000000e+00	1.498844e+00	0.000000e+00	0.000000e+00
1.311081e+00	0.000000e+00	0.000000e+00	-4.935012e+00	0.000000e+00	0.000000e+00
END YODA_SCATTER2D_V2


BEGIN YODA_SCATTER2D_V2 /some_s3d
ErrorBreakdown: {0: {"": {up: 0, dn: 0}}, 1: {"": {up: 0, dn: 0}}, 2: {"": {up: 0, dn: 0}}, 3: {"": {up: 0, dn: 0}}, 4: {"": {up: 0, dn: 0}}, 5: {"": {up: 0, dn: 0}}, 6: {"": {up: 0, dn: 0}}, 7: {"": {up: 0, dn: 0}}, 8: {"": {up: 0, dn: 0}}, 9: {"": {up: 0, dn: 0}}}
Path: /some_s3d
Title: title
Type: Scatter2D
some: stuff
---
# xval	 xerr-	 xerr+	 yval	 yerr-	 yerr+	
-6.777791e-01	8.033572e-01	8.033572e-01	8.926568e-01	0.000000e+00	0.000000e+00
-2.919520e-01	1.359663e-01	1.359663e-01	-3.445205e+00	0.000000e+00	0.000000e+00
-1.440903e-01	8.270964e-01	8.270964e-01	-2.249707e+00	0.000000e+00	0.000000e+00
4.165686e-02	8.936767e-01	8.936767e-01	3.058193e+00	0.000000e+00	0.000000e+00
2.735990e-01	2.610071e+00	2.610071e+00	1.766995e+00	0.000000e+00	0.000000e+00
3.323183e-01	7.326625e-01	7.326625e-01	-7.807818e-01	0.000000e+00	0.000000e+00
6.566365e-01	1.110507e+00	1.110507e+00	1.498844e+00	0.000000e+00	0.000000e+00
6.901730e-01	1.130746e+00	1.130746e+00	5.355288e-02	0.000000e+00	0.000000e+00
8.734044e-01	7.593703e-01	7.593703e-01	-4.072542e+00	0.000000e+00	0.000000e+00
1.592156e+00	2.107315e+00	2.107315e+00	1.037260e+00	0.000000e+00	0.000000e+00
END YODA_SCATTER2D_V2

and the following FLAT file:

# BEGIN HISTO1D /some_h1d
Path=/some_h1d
Title=title
some=stuff
# xlow	 xhigh	 val	 errminus	 errplus
-5.000000e+00	-4.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00
-4.000000e+00	-3.000000e+00	1.000000e+00	1.000000e+00	1.000000e+00
-3.000000e+00	-2.000000e+00	2.600000e+01	5.099020e+00	5.099020e+00
-2.000000e+00	-1.000000e+00	1.440000e+02	1.200000e+01	1.200000e+01
-1.000000e+00	0.000000e+00	3.380000e+02	1.838478e+01	1.838478e+01
0.000000e+00	1.000000e+00	3.380000e+02	1.838478e+01	1.838478e+01
1.000000e+00	2.000000e+00	1.270000e+02	1.126943e+01	1.126943e+01
2.000000e+00	3.000000e+00	2.200000e+01	4.690416e+00	4.690416e+00
3.000000e+00	4.000000e+00	4.000000e+00	2.000000e+00	2.000000e+00
4.000000e+00	5.000000e+00	0.000000e+00	0.000000e+00	0.000000e+00
# END HISTO1D


# BEGIN HISTO2D /some_h2d
Path=/some_h2d
Title=title
some=stuff
# xlow	 xhigh	 ylow	 yhigh	 val	 errminus	 errplus
-4.000000e+00	-1.333333e+00	-6.000000e+00	-3.600000e+00	1.875000e+00	5.412659e-01	5.412659e-01
-4.000000e+00	-1.333333e+00	-3.600000e+00	-1.200000e+00	3.750000e+00	7.654655e-01	7.654655e-01
-4.000000e+00	-1.333333e+00	-1.200000e+00	1.200000e+00	3.281250e+00	7.160275e-01	7.160275e-01
-4.000000e+00	-1.333333e+00	1.200000e+00	3.600000e+00	3.750000e+00	7.654655e-01	7.654655e-01
-4.000000e+00	-1.333333e+00	3.600000e+00	6.000000e+00	2.343750e+00	6.051536e-01	6.051536e-01
-1.333333e+00	1.333333e+00	-6.000000e+00	-3.600000e+00	1.656250e+01	1.608692e+00	1.608692e+00
-1.333333e+00	1.333333e+00	-3.600000e+00	-1.200000e+00	2.859375e+01	2.113711e+00	2.113711e+00
-1.333333e+00	1.333333e+00	-1.200000e+00	1.200000e+00	3.031250e+01	2.176311e+00	2.176311e+00
-1.333333e+00	1.333333e+00	1.200000e+00	3.600000e+00	3.296875e+01	2.269662e+00	2.269662e+00
-1.333333e+00	1.333333e+00	3.600000e+00	6.000000e+00	1.906250e+01	1.725838e+00	1.725838e+00
1.333333e+00	4.000000e+00	-6.000000e+00	-3.600000e+00	1.718750e+00	5.182226e-01	5.182226e-01
1.333333e+00	4.000000e+00	-3.600000e+00	-1.200000e+00	3.281250e+00	7.160275e-01	7.160275e-01
1.333333e+00	4.000000e+00	-1.200000e+00	1.200000e+00	4.375000e+00	8.267973e-01	8.267973e-01
1.333333e+00	4.000000e+00	1.200000e+00	3.600000e+00	3.593750e+00	7.493487e-01	7.493487e-01
1.333333e+00	4.000000e+00	3.600000e+00	6.000000e+00	7.812500e-01	3.493856e-01	3.493856e-01
# END HISTO2D


# BEGIN HISTO1D /some_p1d
Path=/some_p1d
Title=title
some=stuff
# xlow	 xhigh	 val	 errminus	 errplus
-5.000000e+00	-4.000000e+00	nan	nan	nan
-4.000000e+00	-3.000000e+00	-2.144231e+00	nan	nan
-3.000000e+00	-2.000000e+00	3.960226e-01	5.474938e-01	5.474938e-01
-2.000000e+00	-1.000000e+00	-3.545500e-02	2.405400e-01	2.405400e-01
-1.000000e+00	0.000000e+00	2.970288e-01	1.559548e-01	1.559548e-01
0.000000e+00	1.000000e+00	2.573074e-01	1.567438e-01	1.567438e-01
1.000000e+00	2.000000e+00	-3.855420e-01	2.282077e-01	2.282077e-01
2.000000e+00	3.000000e+00	5.857279e-01	5.769541e-01	5.769541e-01
3.000000e+00	4.000000e+00	-1.497654e+00	3.220470e+00	3.220470e+00
4.000000e+00	5.000000e+00	nan	nan	nan
# END HISTO1D


# BEGIN HISTO2D /some_p2d
Path=/some_p2d
Title=title
some=stuff
# xlow	 xhigh	 ylow	 yhigh	 val	 errminus	 errplus
-4.000000e+00	-1.333333e+00	-6.000000e+00	-3.600000e+00	-2.080708e-01	7.991609e-01	7.991609e-01
-4.000000e+00	-1.333333e+00	-3.600000e+00	-1.200000e+00	4.891445e-02	6.362835e-01	6.362835e-01
-4.000000e+00	-1.333333e+00	-1.200000e+00	1.200000e+00	3.000664e-01	7.324812e-01	7.324812e-01
-4.000000e+00	-1.333333e+00	1.200000e+00	3.600000e+00	-7.137894e-01	7.248887e-01	7.248887e-01
-4.000000e+00	-1.333333e+00	3.600000e+00	6.000000e+00	-2.371080e+00	6.755819e-01	6.755819e-01
-1.333333e+00	1.333333e+00	-6.000000e+00	-3.600000e+00	-2.019479e-01	2.601638e-01	2.601638e-01
-1.333333e+00	1.333333e+00	-3.600000e+00	-1.200000e+00	3.078086e-01	2.007678e-01	2.007678e-01
-1.333333e+00	1.333333e+00	-1.200000e+00	1.200000e+00	8.331232e-02	1.972483e-01	1.972483e-01
-1.333333e+00	1.333333e+00	1.200000e+00	3.600000e+00	9.980780e-02	1.925950e-01	1.925950e-01
-1.333333e+00	1.333333e+00	3.600000e+00	6.000000e+00	-8.336468e-02	2.928613e-01	2.928613e-01
1.333333e+00	4.000000e+00	-6.000000e+00	-3.600000e+00	6.627283e-01	1.023041e+00	1.023041e+00
1.333333e+00	4.000000e+00	-3.600000e+00	-1.200000e+00	5.234130e-01	5.624725e-01	5.624725e-01
1.333333e+00	4.000000e+00	-1.200000e+00	1.200000e+00	1.118692e+00	6.963226e-01	6.963226e-01
1.333333e+00	4.000000e+00	1.200000e+00	3.600000e+00	8.626877e-01	7.077319e-01	7.077319e-01
1.333333e+00	4.000000e+00	3.600000e+00	6.000000e+00	2.258521e+00	7.179914e-01	7.179914e-01
# END HISTO2D


# BEGIN HISTO1D /some_s2d
Path=/some_s2d
Title=title
some=stuff
# xlow	 xhigh	 val	 errminus	 errplus
-1.497353e+00	-1.497353e+00	-4.130612e+00	0.000000e+00	0.000000e+00
-2.169587e-01	-2.169587e-01	-2.813620e+00	0.000000e+00	0.000000e+00
-1.729036e-01	-1.729036e-01	-2.767893e+00	0.000000e+00	0.000000e+00
-1.440903e-01	-1.440903e-01	-2.249707e+00	0.000000e+00	0.000000e+00
-1.275883e-01	-1.275883e-01	3.921796e+00	0.000000e+00	0.000000e+00
1.105072e-01	1.105072e-01	4.494148e-01	0.000000e+00	0.000000e+00
1.158848e-01	1.158848e-01	5.355288e-02	0.000000e+00	0.000000e+00
2.463422e-01	2.463422e-01	3.094305e+00	0.000000e+00	0.000000e+00
6.566365e-01	6.566365e-01	1.498844e+00	0.000000e+00	0.000000e+00
1.311081e+00	1.311081e+00	-4.935012e+00	0.000000e+00	0.000000e+00
# END HISTO1D


# BEGIN HISTO1D /some_s3d
Path=/some_s3d
Title=title
some=stuff
# xlow	 xhigh	 val	 errminus	 errplus
-1.481136e+00	1.255781e-01	8.926568e-01	0.000000e+00	0.000000e+00
-4.279184e-01	-1.559857e-01	-3.445205e+00	0.000000e+00	0.000000e+00
-9.711867e-01	6.830061e-01	-2.249707e+00	0.000000e+00	0.000000e+00
-8.520198e-01	9.353336e-01	3.058193e+00	0.000000e+00	0.000000e+00
-2.336472e+00	2.883670e+00	1.766995e+00	0.000000e+00	0.000000e+00
-4.003442e-01	1.064981e+00	-7.807818e-01	0.000000e+00	0.000000e+00
-4.538707e-01	1.767144e+00	1.498844e+00	0.000000e+00	0.000000e+00
-4.405734e-01	1.820919e+00	5.355288e-02	0.000000e+00	0.000000e+00
1.140341e-01	1.632775e+00	-4.072542e+00	0.000000e+00	0.000000e+00
-5.151589e-01	3.699471e+00	1.037260e+00	0.000000e+00	0.000000e+00
# END HISTO1D

These data types can't encode all hist/boost-histogram types, so a writer of these formats would have to raise exceptions if a hist object doesn't conform to these types, but a completely general reader is possible. Moreover, the Python code needed to do it would be straightforward, and there's a big advantage to recognizing a widely-used format.

Even without programmatic access to the YODA Python bindings (which would be hard to test, since YODA is distributed as a configure-make-install package), it would already be advantageous to interact with YODA (and YODA-compatible software) at the level of file interchange.


Perhaps this should be a separate issue, but HEPData YAML files are also a recognized interchange format, also entirely text-based:

https://hepdata-submission.readthedocs.io/en/latest/data_yaml.html

jpivarski avatar May 10 '21 20:05 jpivarski

Even without programmatic access to the YODA Python bindings (which would be hard to test, since YODA is distributed as a configure-make-install package), it would already be advantageous to interact with YODA (and YODA-compatible software) at the level of file interchange.

In case it changes your perspective on this, I can look at adding YODA to conda-forge in the next week. (I expect to be using it during summer so it's already on my radar.)

chrisburr avatar May 11 '21 06:05 chrisburr

That is interesting, and it would make things much easier! (I didn't manage to make install it for some configure reason—to do my example above, I set paths explicitly.)

In that case, Uproot should have a to_yoda along with to_numpy, to_boost, and to_hist, as well as the other direction for writing files (and to_pyroot).

But even with all of that, it would be beneficial for hist to directly read and (if the histogram has only one or two dimensions) write these text files because they're a standard.

jpivarski avatar May 11 '21 11:05 jpivarski

Yoda is now available on conda-forge

chrisburr avatar May 13 '21 06:05 chrisburr