incorporate waveform variability/drift

[nsteinme]

Of some kind...

[czuba]

tl;dr @marius10p

Is drift [of the flavor in the attached example] largely a solved problem in Kilosort2? (i.e. high SNR, but time-varying amplitude spanning multiple channels)

---

We've been working to incorporate kilosort into our standard neural data processing pipeline. Kilosort certainly achieves an impressive feat of [gpu] processing efficiency, and I think the overall framework it introduces for allowing clusters to span multiple channels & vary over time is clearly 'the way forward' from classic channel-based manual sorting techniques. However, when sorting linear array recordings, [even gradual] drift in spike waveform over time leads to largely unusable clustering results.

The attached example is from a set of consecutively recorded files ('_bcdef') from a non-chronically implanted linear array (Plexon 32ch Uprobe, stereo configuration: 50 um within stereo pairs, 100 um between) with the following penetration procedure:

• probe is slowly advanced to target depth (first 4 mm at 50 um/s, 1.3 mm at 20 um/s)
• brief initial settling period (~5 min)
• slight retraction to aid tissue relaxation (-100 um at 10 um/s)
• extended settling before start of recording (~1 hr)
• no further probe movements throughout duration of recordings
• no gaps between recorded files (~3hr total) (...parameters & approaches obviously vary between labs, recording depths, etc, but just laying it all out here for clarity.)

If Kilosort is run on a small subset of the total recording duration (e.g. 10-15 minute file spanning mid-session RF mapping & tuning measurements), the automated* outputs generally capture the majority of well isolatable units with a modest—but manageable—amount of manual curation. (* after manual artifact removal via common average referencing during raw .dat file creation, and fine tuning of kilosort config & post-hoc merge parameters for our hardware & data specs)

However, when concatenating files from the full session (no gaps between files, concatenated in PlexUtil, then run through identical kilosort code pipeline), automated clustering outputs are fragmented to the point of being largely unusable. Rather than capturing the gradual shift in amplitude across channels over time, a single isolatable units are split into multiple clusters across the duration of recording. Merging of clusters in this best-case-scenario example seems manageable enough, but manual trimming becomes increasingly esoteric/subjective when clearly distinct components become intermixed (e.g. blue & yellow cluster components shown).

Presumably this introduces not only manual curation issues, but also degrades the quality of spike detection & template refinement when sub-optimal templates are subtracted from the continuous voltage trace.

The question at hand is, have features of Kilosort2 (as described in supplementary methods of pre-prints) already been developed to address these issues? Or should we be investing time in refinements to kilosort parameterization & merging operations (if so, of what form?), developing in-house pre/post processing solutions outside of Kilosort, or dividing longer sessions into separately sorted epochs that can be manually merged after clustering (method tbd)?

For the broader community:

I'm curious what methods other Kilosort users have developed to address drifting in linear array recordings. ...it seems there is long-standing recognition of the issue, and that there should be plenty of temporal & structural (xyzk channel map) information across channels/clusters to make reasonable assumptions about timing & direction of drift in a rigid linear array device, but there's been little implementation/explanation/development of relevant code in the repository.

In any case, thanks to all involved & contributing to the knowledge base here.

[marius10p]

Yes, this is what Kilosort2 was developed for. Yours look like slow drift, but we can also deal with moderately fast drift (few seconds). I am ironing out the kinks and working towards a full release, hopefully this month. Here is a typical example of one unit tracked by KS2 over 1.5 hours:

[czuba]

Ah great! That was a particularly stable session & well behaved subject, so slow drift was the primary culprit. One or more abrupt shifts are not uncommon over the course of a 2-3hr recording session, so the ability to track both fast & slow drift will go a long way in real-world applications. (...where lab==world, anyway)

Thanks for the quick reply and continued development. ...good luck on the final push.