Arthur Conmy
Arthur Conmy
**Describe the bug** The `n_params` counts calculated [here](https://github.com/neelnanda-io/TransformerLens/blob/f5a7d455546a88cfdfb26e781d5bd6447e8243eb/transformer_lens/HookedTransformerConfig.py#L242) are wrong. For example, LLAMA uses SwiGLU so the 2x factor in the [linked code ](https://github.com/neelnanda-io/TransformerLens/blob/f5a7d455546a88cfdfb26e781d5bd6447e8243eb/transformer_lens/HookedTransformerConfig.py#L242) is wrong. Further this just ignores...
Addressing in https://github.com/neelnanda-io/TransformerLens/pull/352 [x] Implemented Llama-2-7B and Llama-2-13B [ ] Implement Llama-2-70B architecture (add Grouped-Query Attention)
We should be able to use the python script here: https://github.com/nojvek/vscode-ipynb-py-converter to i) write notebooks as `.py` files with `#%%` ii) have these automatically converted to `.ipynb` files on push...
**Background**: When I implemented `use_split_qkv_input` [here](https://github.com/neelnanda-io/TransformerLens/pull/158), I changed the attention module to take three inputs (to query, key and value). Even when this feature is not enabled, **we compute the...
### Proposal Add a feature to provide detailed print-outs of currently attached hooks to a model and a HookPoint. ### Motivation Sometimes I want to look at what hooks I've...
## 📚 Documentation The formatting https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html of `Tensor.register_hook` is strange, the link after this is broken, and the code example should specify `gradient=` to the backward call, as this is...
```python for k in exp.global_cache.corrupted_cache.keys(): print(k, exp.global_cache.corrupted_cache[k].shape, k in exp.global_cache.online_cache) ``` returns lots of unnecessary things: ``` blocks.0.ln1.hook_scale torch.Size([40, 41, 8, 1]) False blocks.0.ln1.hook_normalized torch.Size([40, 41, 8, 512]) False ......
[This](https://github.com/callummcdougall/ARENA_2.0/blob/4cda66b64b48dbabdc6b0bd6f5d7a86eea375507/chapter1_transformers/exercises/part3_indirect_object_identification/ioi_dataset.py#L532) file has several nicer methods for easy IOI usage
Will close https://github.com/ArthurConmy/Automatic-Circuit-Discovery/issues/73 However, relies on a PR to TransformerLens here https://github.com/neelnanda-io/TransformerLens/pull/336 (the poetry is updated in the ACDC PR to install this version of TransformerLens)
`neato -s1 -Tpdf -ogreaterthan_mlp.pdf greaterthan_mlp.gv` generates much nicer plots: [greaterthan_mlp.pdf](https://github.com/ArthurConmy/Automatic-Circuit-Discovery/files/11782518/greaterthan_mlp.pdf)