Extend the `select` operation to allow more fine grained sweep selection

[t-b]

The select operation 1 has the following signature:

select([array channels, array sweeps[, string mode[, string clampMode]]])

now we would like to add more control over sweep selection. Not only should it be able to select via displayed and clampmode but also via stimulus set and if it passed or not.

Sweep passing state is a LBN entry, see PSQ_FMT_LBN_SWEEP_PASS 2 as input for CreateAnaFuncLBNKey.

Not all analysis functions have that entry, only DA, SP, RA, CR, PB, SE, VM, AR so from the complete list

PSQ_PipetteInBath PB
PSQ_SealEvaluation SE
PSQ_AccessResistanceSmoke AR
PSQ_TrueRestingMembranePotential VM
PSQ_DaScale (sub threshold) DA (sub)
PSQ_SquarePulse (long pulse) SP
PSQ_Rheobase (long pulse) RB
PSQ_DaScale (supra threshold) DA (supra)
PSQ_SquarePulse (short pulse) SP
PSQ_Rheobase (short pulse) RB
PSQ_Ramp RA
PSQ_Chirp CR

and that means PSQ_Rheobase needs to be covered differently (see AD_FillWaves line 344, but note #2017). Idea:

select([array channels, array sweeps, selectXXX(), ...])

where all selectXXX operations would be and-ed.

selectOptVis([all | displayed]) # default: displayed
selectOptCM([all | ic | vc | izero]) # default: all
selectOptStimset([name1, name2]) # name supports wildcards
selectOptIVSCCSweepQC(passed | failed)
selectOptIVSCCSetQC(passed | failed)

As users would also like to select multiple options OR'ed (e.g. passing sweeps from stimset A and failing sweeps from stimset B) we propose to allow an array of select() for all operations which accept a select statement (tp, data, psxKernel, psxStats, epochs, labnotebook).

[t-b]

All operations will accept an array of select statements. Operations which accept a range will drop that argument. tp will warn when range is set in the select statement.

select will then return multiple datasets, one or multiple ranges (Nx2) and the sweepNo-channelType-channelNo tripplet (Nx3).

1. Unify arguments

$so1  = selectVis(displayed)
$so2  = selectCM(ic, vc)
$so3  = selectStimset(name1, name2)
$so4  = channels(AD6)
$so5  = sweeps()
$so6  = selectRange() # accepts what formerly data accepted in the first argument
$so7  = selectIVSCCSweepQC(passed | failed)
$so8  = selectIVSCCSetQC(passed | failed)

// select will show all displayed sweeps from channel 6 with ic/vc clampMode from stimsets name1/name2
select($so1, $so2, $so3, $so4, $so5, ...)

Approaches not choosen

2. Combined array

$so1 = selectOptVis()
$so2 = selectOptCM(ic)
select(channels(AD), sweeps(), [$so1, $so2])

3. Key-Value pairs

$so1 = selectOpt("vis=all", "clampMode=vc", "stimset=setA*")
$so2 = selectOpt("vis=all", "clampMode=ic", "stimset=setB*")

select(channels(AD), sweeps(), selectJoin($so1, $so2)) # or with []

4. Ordered arguments

$so = selectOpt(vis, all, clampMode,[vc,ic],stimset,setA*)
select(channels(AD), sweeps(), $so)
</details>

[t-b]

Option 1 it is.

[timjarsky]

@t-b I'd also like to add an option to select all baseline or all pulse epochs, for example. We could also give the user a chance to apply some QC. e.g., check for spikes in baseline data?

[t-b]

And it would be nice if we could recursively do selections:

sel = select(sweeps())
sel1 = select($sel, selectRange(E1))

This helps in disecting a bunch of sweeps (an RAC?) into two selections, one for each SCI for example.

[MichaelHuth]

Thought regarding selectRange(E1):

select itself can not determine if ranges are good as this is depending on sweep / channelNumber / channelType. Thus, it makes no sense to think of any AND way to combine ranges when multiple are specified.

e.g. select(selectRange("E*", "U*"), selectRange("*1", "*2"))

Currently the implementation just gathers all ranges specified. Another option would be to allow only a single selectRange per select.

As select does not evaluate the ranges itself this information must be carried over to e.g. data. The idea is to make a composite return with two datasets, where the first dataset is the classic Nx3 select wave and the second dataset the range specification (as returned from SFH_EvaluateRange). The whole composite would have the SELECT data type.

Regarding other entries appearing multiple times in select: For selectStimset, sweeps, channels the SetIntersection is calculated. String inputs such as selectStimset and channels are evaluated as is for the intersection. Any wildcard patterns are not evaluated. This also applies to channels in a sense, as {NaN, NaN} is a channels wildcard.

[MichaelHuth]

behavior for select:

• if no select argument is set, then non-appearing filter arguments are default.
• if select+ argument is set, then non-appearing filter arguments are ignored/off.
• select returns always only the ranges that are specified in this select operation

[MichaelHuth]

• allow only a single selectRange specification per select call. This solves the 1 range vs exactly N ranges, we should not have any other number of ranges
• operations that have a select argument: use a generic function that parses that argument as either single select or array of selects, return waveref wave of selectComp
• SFH_GetSweepsForFormula: iterates independently over each selectComp of the waveref input, handle selects and ranges as before per selectComp