remove mtrain reliance

[djkapner]

We want to remove mtrain reliance, so that there aren't potential data inconsistencies out of our control, for instance, between mtrain, pkl files, and LIMS.

Tasks

• [x] survey allensdk for mtrain usage. Document what information is extracted from mtrain.
• [x] specify alternative sources for this same information. If pkl files, is the access fast enough? If not, add columns to LIMS.
• [ ] If columns are added to LIMS, work with LIMS team on pkl ingest to add these columns automatically.
• [ ] replace mtrain calls with LIMS or pkl reads.
• [ ] delete mtrain from AllenSDK (as much as possible).

Validation

• [ ] x
• [ ] y
• [ ] z

[djkapner]

survey AllenSDK for mtrain reliance:

• AllenSDK/allensdk/brain_observatory/behavior/session_apis/data_io/behavior_ophys_lims_api.py credentials imported, it doesn't seem to be used (in this module).
• AllenSDK/allensdk/brain_observatory/behavior/session_apis/data_io/behavior_lims_api.py credentials imported, it doesn't seem to be used (in this module).
• AllenSDK/allensdk/brain_observatory/behavior/project_apis/data_io/behavior_project_lims_api.py - mtrain used to query session_type via behavior_session -> foraging_id. Replacement requires adding session_type (stimulus_name) to LIMS behavior_sessions table. (pkl reads too slow).
• AllenSDK/allensdk/brain_observatory/behavior/behavior_project_cache.py not used, just passed on to API.
• https://github.com/AllenInstitute/AllenSDK/blob/9474bafaf0ebd1eb0e61bb69dd9423f924e4b310/allensdk/brain_observatory/behavior/schemas.py#L72 - if needed, could replace with foraging id. should we add foraging_id to behavior_sessions table also?
• AllenSDK/allensdk/brain_observatory/nwb/ndx-aibs-behavior-ophys.extension.yaml - shows up twice here.
• a few corresponding tests.

[djkapner]

Framework for LIMS team request:

• [ ] add column named stimulus_name to behavior_sessions table. type: TEXT, allow nulls. NOTE: do we like stimulus_name ? aliases are session_type (behavior cache name) and stage_name (pkl file name)
• [ ] update behavior sessions by querying existing pkl files:

TBD SQL + python for finding path and extracting correct field.

• [ ] implement additional step in <name of LIMS strategy that ingests pkl files> to populate this field going forward.

[wbwakeman]

http://jira.corp.alleninstitute.org/browse/PBS-2771 created for Platform team to add the column and populate with the information from mtrain