keras-nlp

keras-nlp copied to clipboard

Validate bert pretraining on new code changes

2

We have made a number of changes to our bert modeling code of late. Primary changes are https://github.com/keras-team/keras-nlp/pull/288 and https://github.com/keras-team/keras-nlp/pull/360 We should re-run an entire pretraining job for the base...

mattdangerw

Checkpoint conversion scripts for RoBERTa/XLMRoBERTa

6

Unmaintained examples of how to do checkpoint conversions between pytorch and tensorflow for RoBERTa and XLM-R models, also includes benchmarking.

jessechancy

Publish an Example Showing Intermediate Output from Pretrained Models

1

The last layer's hidden state is not always the best representation of text. In literature, output from intermediate layers is leveraged as well, to improve predictive performance. Here is a...

abheesht17

Allow instantiating a BERT model in mixed precision from pre-trained weights

2

Our BERT models should support mixed precision as described in https://keras.io/api/mixed_precision/. The models should follow the global policy `keras.mixed_precision.set_global_policy()` when set. The models should probably also also expose a dtype...

mattdangerw

We should be able to rework the input data for bert pretraining, to use `model.fit()` directly without a custom `train_step`. - We will need to rework our input data into...

mattdangerw

Rework examples/bert pretraining to use KerasNLP preprocessing

1

This will be a little investigatory, as it is unclear what the precise solution could look like. Right now, we have a [large preprocessing script](https://github.com/keras-team/keras-nlp/blob/master/examples/bert/bert_preprocess.py) that we inherited from the...

mattdangerw

[Tracking] Performance optimizations for library

1

### Discussed in https://github.com/keras-team/keras-nlp/discussions/322 * [`mixed_precision`](https://www.tensorflow.org/guide/mixed_precision) * [`xla`](https://www.tensorflow.org/xla/tutorials/jit_compile) * Documentation for performance optimizations

jbischof

Requested in #299 for consistency with `keras-cv`. Currently blocked on finalizing our preprocessing API.

jbischof

We would like to be able to annotate `keras_nlp.layers.MLMHead` with `@keras.utils.register_keras_serializable(package="keras_nlp")`. Which will allow the python object for the layer to restore after saving. However doing this naively does not...

mattdangerw

I recently found out that hard way that `tensorflow-text` is incompatible with Conda (see [issue](https://github.com/tensorflow/text/issues/644)). Sitting downstream of this library makes `keras-nlp` also incompatible with Conda. We should add a...

jbischof