dig-etl-engine icon indicating copy to clipboard operation
dig-etl-engine copied to clipboard

Timeseries annotation: extractSpreadsheet very slow in col-based sheets

Open majidghgol opened this issue 7 years ago • 0 comments

The sample file for Earthquakes data (issue 175) has about 8000 rows and 20 columns. I introduced a very simple mapping spec to extract only the magnitude of the earthquakes (one column). However, I noticed every iteration in parse_col_ts takes 1s, which makes it need more than 2 hours to finish. I am not sure if it is specific to my machine, but as far as I know pyexcel is much more efficient in row by row reading of excel file. So, it might be due the fact that the spreadsheet is read col by col in this function.

The mapping spec I am using:

[
	{
		"TimeSeriesRegions":
		[
			{
				"orientation": "col",
				"locs": "[2:8498]",
				"cols": "[E:E]",
				"metadata": [
					{
						"orientation": "row",
						"name": "data_label",
						"loc": 1
					}
				],
				"times": {
					"locs": "[A]"
				}
			}
		],
		"Metadata":
		[

		]
	}
]

majidghgol avatar Jan 12 '18 23:01 majidghgol