keras-nlp

keras-nlp copied to clipboard

Add a keras.io guide for pretraining a transformer with keras-nlp

3

Working on a keras.io guide for pretraining a keras-nlp transformer model from scratch, using word piece tokenizer, transformer encoder, embedding layers, and our MLM layer helpers. Will link a draft...

mattdangerw

Adding the ELECTRA Model

11

While looking around, I found [this paper on the ELECTRA model](https://arxiv.org/abs/2003.10555), it shows that replacing MLM with RTD gave them better GLUE scores than BERT. Might be worth taking note...

Stealth-py

Add TPU and evaluation saving for GLUE finetuning

1

chenmoneygithub

Decoding Functions Not Working when `jit_compile = True`

2

When `jit_compile` is set to True, the decoding functions do not work. We wish to add support for the same in the future. Error: https://p.ip.fi/2TNt

abheesht17

Adding `embed_dim` argument to `TransformerEncoder` and `TransformerDecoder` layers

4

**Is your feature request related to a problem? Please describe.** Currently, both `TransformerEncoder` and `TransformerDecoder` accept an `intermediate_dim` argument, allowing users to specify the latent space dimensionality, but inherit the...

DavidLandup0

`TokenAndPositionEmbedding`, `TransformerEncoder` and `TransformerDecoder` can be saved, but prevent the model from being loaded

2

**Describe the bug** Some of the new custom layers correctly override the `get_config()` method, so as to be able to be saved, but when a model is saved in `h5`...

DavidLandup0

Add a byte pair encoding (BPE) tokenizer layer

14

We would like to add a BPE tokenizer (used by gpt-2, roberta and others). This ideally should be configurable to be compatible with the actual tokenization used by gpt-2 and...

mattdangerw

Demo BLEU in the spanish-english translation example

2

Once we land the BLEU metric, let's add BLEU to the mix of metrics we are showing off here: https://keras.io/examples/nlp/neural_machine_translation_with_keras_nlp/#evaluating-our-model-quantitative-analysis

mattdangerw

Add a model proto training utility for sentence piece

2

Along with https://github.com/keras-team/keras-nlp/issues/248, we should add a utility to train a model proto for sentence piece. This can leverage the sentencepiece pip package to run their trainer. We should not...

mattdangerw

Add a vocab training utility for word piece

1

We would like to add a vocab training utility for wordpiece. This can leverage the utilities for doing this in tensorflow text. Note that we do not have to cover...

mattdangerw