Easy-Transformer icon indicating copy to clipboard operation
Easy-Transformer copied to clipboard

add better model properties table to docs

Open mivanit opened this issue 1 year ago • 0 comments

[WIP]

Description

This PR ports features from my transformerlens-model-table repo to TransformerLens, implementing many of the features requested in #97. I still need some feedback on this, and presumably building docs will fail for one reason or another once I make the PR.

Features:

The static table has a few more fields added to it, but the primary focus is the interactive table. This provides:

  • information on parallel attn/mlps, positional embeddings, and other config elements
  • filtering and searching on any column (i.e. sort by parameter count and only show standard positional embeddings)
  • links back to the huggingface model page, where applicable (extracted from the "official model name")
  • tokenizer information, including vocab hash (need feedback on if there is a better way to do this)
  • full config in title text or new window
  • organized view of dimensions of all tensors in state dict and activation cache (via setting device to meta, doesn't require actually loading models)

Adds dependencies

under group docs:

  • tiktoken for dealing with certain tokenizers
  • muutils for pretty-printed data on tensor shapes

Type of change

  • [x] This change requires a documentation update

Screenshots

Before:

Original model properties table

image

After (static):

You can see what the generated data looks like here

image

After (interactive):

See demo

tl-new

image

Checklist:

(currently draft PR, testing incomplete)

  • [x] I have commented my code, particularly in hard-to-understand areas
  • [x] I have made corresponding changes to the documentation
  • [ ] My changes generate no new warnings
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] New and existing unit tests pass locally with my changes
  • [ ] I have not rewritten tests relating to key interfaces which would affect backward compatibility

mivanit avatar Jun 17 '24 10:06 mivanit