NEKO
NEKO copied to clipboard
ManifoldRG
In Progress Implementation of GATO style Generalist Multimodal model capable of image, text, RL and Robotics tasks
Notebooks don't show up pretty in the Github diff viewer. You can see the pretty notebook here: https://github.com/eihli/NEKO/blob/vqa-eval-exploration/dev_notebooks/vqa_playground.ipynb Explores the difference between training and evaluation runs. Trying to find why...
Experiment with https://github.com/lucidrains/vit-pytorch and replace the current ImageEmbedding with such ViT to see whether it can improve the NEKO performance
VQA loss is not decreasing during training, we have tried several ways to reduce it, this is one of the tests. So far we have been following GATO paper to...
TODO: Add details. This is just a rough draft of something Henry and I were talking about over a screen share. ``` def pad(predicted, target): torch.tensor() if len(target) > len(predicted):...
Added debug logging as per issue 73 for more verbose error messages. I added to all files except for `trajectory_gpt2.py` as this already had logging imported from `transformers.utils`. I kept...
Aside from the print statements and wandb output, it might be nice for development and debugging to have the option to turn on verbose debug logs. I've used something like...
Refer to https://github.com/ManifoldRG/NEKO/pull/30 about the PR to add caption and vqa task. Refer to https://github.com/ManifoldRG/NEKO/pull/77 about the PR to merge the code for caption task and vqa task (all of...
Metadata
Owner
Metadata
In Progress Implementation of GATO style Generalist Multimodal model capable of image, text, RL and Robotics tasks