Alex Spies

icon indicating a website link afspies.com icon indicating an email [email protected]

icon company @ImperialCollegeLondon icon location London, UK icon location Mechanistic Interpretability & Neurosymbolic AI | PhD Candidate in CS @ImperialCollegeLondon | Research Lead @understanding-search |

Results 1 comments of Alex Spies

[Bug Report] `load_and_process_state_dict` handles LayerNorm folding poorly

I would have expected the behaviour of `fold_ln=True` to be that the result is a `HookedTransformer` with properly folded layernorms, regardless of whether the `state_dict` being loaded already had these...