data_set.get_parameter_data gets slower with increasing values of start/end

[thibaudruelle]

Our dataset comprises 4 dependent parameters, which depend on 1 or 2 of the 2 independent parameters. interdeps.dependencies returns:

{ParamSpecBase('PDt_gain', 'numeric', 'Gain', ''): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),),
 ParamSpecBase('PDt', 'array', 'PDt', 'V'): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),
  ParamSpecBase('Time', 'array', 'Time', 's')),
 ParamSpecBase('PDr', 'array', 'PDr', 'V'): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),
  ParamSpecBase('Time', 'array', 'Time', 's')),
 ParamSpecBase('AWG', 'array', 'AWG', 'V'): (ParamSpecBase('N_stepd', 'numeric', 'number of stepper stepd steps', ''),
  ParamSpecBase('Time', 'array', 'Time', 's'))}

The number of results contained in the dataset is typically on the order of 1000 per dependent parameters (ds.number_of_results returns 4000). The dependent parameters of type array typically contain 200000 values (len(ds.get_parameter_data('PDt', start=0, end=0)['PDt']['PDt'][0]) returns 200000). We analyze these datasets row by row, and it is not practical to load the whole dataset at once in RAM. We thus use the start and end parameters of ds.get_parameter_data.

We observe a significant increase in the time taken to process ds.get_parameter_data('PDt', start=n, end=n)['PDt'] for increasing values of n.

%%timeit
ds.get_parameter_data('PDt', start=0, end=0)['PDt']

returns 11.6 ms ± 137 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

%%timeit
ds.get_parameter_data('PDt', start=600, end=600)['PDt']

returns 929 ms ± 11.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

System

Windows 10

fork of qcodes master branch, last common ancestor 48287481f16400189034e9b433a823e4aeae9cc