Harsh Sikka

v0 Dataset and Benchmark Specification

4

Context: @snat-s has done great work w/ the analysis of various data that may be relevant to Neko We should now, w/ the input of the team, finalize our proposed...

Dataset Reconcilliation

1

Context: @snat-s has made significant progress in reviewing and updating the planned multimodal dataset (combination of many datasets) for the NEKO model. there are numerous older issues that we want...

Investigate Potential Use of Pretrained Models in Generalist /Massively Multimodal Setting

2

Rather than training from scratch, we might use pretrained weights to serve as a basis for our model. We want to understand models might serve as the basis for our...

building on #40, capture: - original analysis document - updated analysis of storage and compute requirements - process across manifold for managing resources (can be project driven or org driven)

Context: We want to profile existing multimodal models we picked in #62 and evaluate them on the benchmark we are proposing. Output: Reported model performance on benchmark

Outline Control Modality Next Steps across Data, Training

3

@jsjung00 is catching up to context from control modality work w/ help from @daniellawson9999. We want to rapidly understand what next datasets are needed, as we are finalizing the v0...

Outline Language & Vision Modality next steps

2

Similar to #59 we want to understand next steps (both brief context on small level issues (i.e. bugs etc) and next major steps