yoyodyne icon indicating copy to clipboard operation
yoyodyne copied to clipboard

WIP: single embeddings matrix design

Open Adamits opened this issue 1 year ago • 46 comments

Throwing this WIP up to store all vocabs in a single embeddings matrix shared between source, target, and features. This will fix the current pointer-generator issues when we have disjoint vocabularies. I will get back to implementing it later this week.

  • Computes a single vocab_map
  • Retains target_vocabulary and source_vocabulary for logging.
  • Moves embedding initialization onto the model
  • Passes a single embedding layer to each module to share.

TODO

  • I did not do anything with features. I still need to merge these into the single vocab, being careful not to break anything that requires separate features.
  • This does not handle the case that someone may want to have unmerged embeddings, such that the same unicode codepoint on the source and target side should be considered different. We will need to update the index to mark those characters with special symbols to implement this.

Adamits avatar Oct 02 '23 19:10 Adamits