Joseph Bloom

Make it possible to track the preferences of the PPO in the app.

1

https://docs.google.com/document/d/1N1lVOXS5bLKYiXfoEeQoxxtI_0EfROi-JXcs-eYTCSA/edit?usp=sharing I think this could be very valuable form the perspective of measuring the agent-simulators proclivity for modelling different agents in it's training distribution.

SVD Decomp / Explore ways to use dimensionality reduction to quickly understand what heads are doing.

1

This [post](https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight ) is awesome. I think the value from using this method comes from both understanding the method better, understanding our models better and the editing could be cool...

ask me for details

Add static interpretability visualizations to wandb dashboard. Seems like a cool idea I just had. Some stuff: QK/OV circuit viz, different attribution embeddings, time embedding, L2 Norms of different components.

https://pytorch.org/docs/stable/onnx.html#example-alexnet-from-pytorch-to-onnx Not sure how much of a priority this is but it looks cool.

The BOWEmbedding from BabyAI has very large vectors on init. I wonder if that's really terrible and was slowing down my training of demo generating models. Need to investigate at...

From Gato paper: "Regularization: We train with an AdamW weight decay parameter of 0.1. Additionally, we use stochastic depth (Huang et al., 2016) during pretraining, where each of the transformer...

Write a utility for merging sampled rollouts into a single file

2

We can sample rollouts using `sample from agents`. However, it would be good to be able to merge trajectory datasets. In order to do this, it might be worth cleaning...

The collect demonstrations utility is responsible for collecting example trajectories from a trained agent (one of the 3 ppo agent architectures supported, only two work). It provides a few different...