NEKO
NEKO copied to clipboard
ManifoldRG
In Progress Implementation of GATO style Generalist Multimodal model capable of image, text, RL and Robotics tasks
We need more compute & storage than is individually available to us via local GPUs to train the MVM outlined on our Roadmap: To engage with any potential compute &...
I noticed [I was getting 4 wandb runs when I trained with `accelerate launch`](https://wandb.ai/eihli/gato-control/runs/8cm71tzr/overview?workspace=user-eihli) on a machine with 4 GPUs. All 4 of the wandb runs include system statistics, like...
After adding text modality to the NEKO project on top of the continuous control, discrete control tasks etc., we are adding this task of image-caption, it is self-explanatory - train...
We have a guide on doing distributed training w/ Vast here: https://docs.google.com/document/d/1W_dN3qarCOcLRDdEZ75LBtkLGiwUziWWDtVTjd43Ad4/edit?usp=sharing . However, we have not performed full distributed training runs. This issue does not specify specific issues, but...
gato_policy. py, the following lines pad_len = self.context_len - seq_len if pad_len > 0: assumes seq_len is always smaller than context_len, but it could be the other way also. Need...
This is one of the series of issues found during the code review under the branch "add_text_modality" (some issues are inherited from the master branch), we will keep track of...
We want fast model inference for quickly evaluating the performance of our checkpoints during training or after training. Evaluation is called in trainer.py, looping over each task: https://github.com/ManifoldRG/gato-control/blob/93009abfaa1e0a9efcfcb8eba1435352dfdbcd4b/gato/training/trainer.py#L77-L84 task.evaluate() a...
This issue relates to how we embed each (16x16) patch. Additionally, we discuss the positional encodings we add to each patch's embedding. # Patch Embedding Let's review, we split the...
Metadata
Owner
Metadata
In Progress Implementation of GATO style Generalist Multimodal model capable of image, text, RL and Robotics tasks