Albert model? I will be so pleased to test the ALBert model in tf2.

[YankeeMarco]

Please go to Stack Overflow for help and support:

http://stackoverflow.com/questions/tagged/tensorflow

Also, please understand that many of the models included in this repository are experimental and research-style code. If you open a GitHub issue, here is our policy:

1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
2. The form below must be filled out.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

---

System information

• What is the top-level directory of the model you are using:
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
• TensorFlow installed from (source or binary):
• TensorFlow version (use command below):
• Bazel version (if compiling from source):
• CUDA/cuDNN version:
• GPU model and memory:
• Exact command to reproduce:

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

[saberkun]

Hi, Albert is straightforward to implement. We are releasing a newer version soon including Albert variants. Assigning the issue to me. Will update here. Thanks!

[saberkun]

We are working in progress. Expect to release checkpoints and models soon.

[stefan-it]

Hi guys,

is there any update about an ALBERT pretraining with Model Garden?

Many thanks!

[laxmareddyp]

Hi @stefan-it ,

Could you please check this Albert pretrained_models.md from the Model Garden will help you.

[stefan-it]

Hi @laxmareddyp,

thanks for your quick reply! So my aim is to pretrain an ALBERT model from scratch. Thanks to TRC I could sucessfully pretrain BERT, BERT with Token Dropping and TEAMS models with this great Model Garden library here. E.g. this is pretraining command for BERT with Token Dropping:

python3 train.py \
    --experiment=token_drop_bert/pretraining \
    --config_file=gwlms_base_pretrain_sequence_pack.yaml \
    --config_file=gwlms_base_token_drop.yaml \
    --params_override="runtime.distribution_strategy=tpu" \
    --tpu=gwlms \
    --model_dir=gs://gwlms/models/bert-base-td-german-wikipedia-v1-dupe5-cased \
    --mode=train

As far as I can see, there's no experiment definition like albert/pretraining. Technically, the generation of pretraining data should work for ALBERT, as the create_pretraining_data.py script supports SPM models, so I think "only" the experiment definition part for ALBERT is missing here :thinking: