contextual-corr-analysis

contextual-corr-analysis copied to clipboard

This repository accompanies the paper ([[https://arxiv.org/abs/2005.01172][arXiv]]). Accepted to ACL 2020.

tl;dr read intro and one of the usage headings.

• intro Tools to understand neural representations, and application to contextualizers. A "contexualizer" is a model producing a context-dependent word embedding.

Concretely, similarity measures, eg.

• [[https://arxiv.org/abs/1905.00414][CKA]]
• [[https://arxiv.org/abs/1706.05806][SVCCA]] applied to SOTA contextualizers, eg.
• [[https://arxiv.org/abs/1810.04805][BERT]]
• [[https://arxiv.org/abs/1802.05365][ELMo]]

All similarity measures can be found in =corr_methods.py=.

We also experimented with novel attention-based similarity measures in =attention_corr_methods.py=.

This repository should be on your Python path. #+BEGIN_SRC sh export PYTHONPATH="${PWD}:${PYTHONPATH}" #+END_SRC

• usage (script) The main script is =main.py=.

#+BEGIN_SRC sh main.py [--methods [METHODS ...]] REPRESENTATION_FILES OUTPUT_FILE #+END_SRC

For examples, see =slurm= (eg. =mk_resultsN.sh= and =mk_resultsN-helper.sh=). To see all options, run ~python main.py --help~. Note that =REPRESENTATION_FILES= is a file containing an input file on each line. =OUTPUT_FILE= is a pickle dump.

=main_attn.py= is analogous.

• usage (python) You can also call the correlation methods directly from python. See =ex.ipynb=.
• dir

• =var.py=. Stuff you might want to change if you use this. eg, it has function =fname2mname= (filename to model name) that transforms =/data/sls/temp/belinkov/contextual-corr-analysis/contextualizers/bert_large_cased/ptb_pos_dev.hdf5= to =bert_large_cased-ptb_pos_dev.hdf5=.

• analysis. Data analysis. The results that will be presented. analysis-n analyzes the result of experiment n.

• hnb. "Helper notebook." Files in this directory are to

  • help me code
  • help the reader understand the resulting .py files.

  These are files containing a copy of the function with loops and co. destructured (run once with an arbitrary value, to help debugging).

  It may help you understand a function.

• slurm. [[https://slurm.schedmd.com/documentation.html][SLURM]] scripts.

  Run directly as ~SCRIPTNAME~.

• other. Everything else. Lots of junk.

• workflow Our pipeline is:

1. Generate representations (hdf5 files)
2. Run =main.py= on them
   1. Loads the representations (=load_representations= in =corr_methods.py=)
   2. Compute the correlations using the given methods
   3. Writes them to =OUTPUT_FILE=.
3. Analyze results (the OUTPUT_FILE above) in the =analysis= directory

• modifying New correlation methods should extend =corr_methods.Method=.
• gallery SVCCA similarities (=Method= =CCA= in =corr_methods.py=): #+BEGIN_CENTER [[file:assets/svcca-biglabels-lowres.png]] #+END_CENTER

CKA attention similarity (=Method= =AttnLinCKA= in =attention_corr_methods.py=): #+BEGIN_CENTER [[file:assets/attn-cka-biglabels-lowres.png]] #+END_CENTER