keras-nlp

keras-nlp copied to clipboard

abheesht17

Integrate ELECTRA Pretrained Model into Keras_NLP Ecosystem

6

I would like to add ELECTRA pretrained model in the keras_nlp ecosystem. I have went through the `CONTRIBUTINT_MODELS.md` and making an issue is the first task to add the model....

pranavvp16

Machine Translation With Transformers

3

can't run this example on jax or pytorch backend it just works on tensorflow backend https://keras.io/examples/nlp/neural_machine_translation_with_keras_nlp/ also inferencing is significantly slower than a similar implementation in pytorch, like 8 times...

alerem18

Sharding for model loading

2

As we are progressing towards welcoming Large models in library, we need a sharding mechanism for loading large checkpoints in model.

kanpuriyanawab

Signature: ```python def compile( self, optimizer="keras_nlp>BertOptimizer", loss="sparse_categorical_crossentropy", metrics="sparse_categorical_accuracy", jit_compile=True, **kwargs, ): ``` Usage: ```python classifier = keras_nlp.models.BertClassifier.from_preset( "bert_base_en_uncased", num_classes=5, ) # Default compilation. classifier.fit(dataset) # Custom learning rate. classifier.compile( optimizer=keras_nlp.models.BertOptimizer(...

mattdangerw

As discussed on https://github.com/keras-team/keras-nlp/issues/1270, we might want to add a small integration test that builds a transformer from scratch using our blocks and runs a small amount of training on...

mattdangerw

Add support for soft prompts.

1

**Is your feature request related to a problem? Please describe.** An alternative to fine tuning a whole model, or only some layers, is to fine tuning an ad-hoc prompt with...

arivero

Parameter Tying

4

**Is your feature request related to a problem? Please describe.** A lot of models, including GPT, use the same weights matrix for the embedding of the input and, transposing it,...

arivero

Add support for learnable relative position encoding

2

Relative postion is useful for text of arbitrary length. Our DeBERTa model now has a relative postional encoding, but it now only returns the repeated embedding matrix: [code link](https://github.com/keras-team/keras-nlp/blob/340a5cc7370d0f91bd1acff5b25bf60a73aa6e38/keras_nlp/models/deberta_v3/relative_embedding.py#L73) I...

chenmoneygithub

Add nightly testing for extra_large tests

2

We have some testing that is very resource intensive to run marked with an "extra large" annotation. This includes our fenced docstring tests, and testing for most of our presets....

mattdangerw