tomohideshibata

Pre-training using GPUs is strange

4

I am trying pre-training from scratch using GPUs in Japanese, but the pre-training seems strange. In the following log, `masked_lm_accuracy` and `sentence_order_accuracy` suddenly dropped. ``` .. I1211 00:37:45.981178 139995264022336 model_training_utils.py:346]...

[ALBERT] --random_next_sentence option in create_pretraining_data.py

1

Thanks for releasing ALBERT codes. The default for `--random_next_sentence` option in `create_pretraining_data.py` is "True", but to enable the sentence order prediction (SOP) task, this option is set to be "False",...

| S-ID | Type | Wrong | Right | Note ---- | ---- | ---- | ---- | ---- w201106-0000928415-2 | BasicPhrase | - | add "NE:LOCATION:日本" | 日本 w201106-0000928415-2...

| S-ID | Type | Wrong | Right | Note ---- | ---- | ---- | ---- | ---- w201106-0000463299-1 | BasicPhrase | - | add "NE:LOCATION:日本" | 日本 w201106-0000463299-1...

| S-ID | Type | Wrong | Right | Note ---- | ---- | ---- | ---- | ---- w201106-0002100016-1-02 | BasicPhrase | NE:PERSON:謝霆鋒 | - | 霆 w201106-0002100016-1-02 |...

| S-ID | Type | Wrong | Right | Note ---- | ---- | ---- | ---- | ---- w201106-0000116058-1 | BasicPhrase | NE:LOCATION:広島県呉市焼山三ツ石町 | - | 焼山 w201106-0000116058-1 |...

| S-ID | Type | Wrong | Right | Note ---- | ---- | ---- | ---- | ---- w201106-0001761264-1 | BasicPhrase | NE:LOCATION:平和通り | - | 平和 w201106-0001761264-1 |...

| S-ID | Type | Wrong | Right | Note ---- | ---- | ---- | ---- | ---- w201106-0001513550-3 | BasicPhrase | NE:LOCATION:「大井町」駅 | - | 「 大井 町...

| S-ID | Type | Wrong | Right | Note ---- | ---- | ---- | ---- | ---- w201106-0000981708-2 | BasicPhrase | NE:ORGANIZATION:関西大学 | - | 関西 w201106-0000981708-2 |...

| S-ID | Type | Wrong | Right | Note ---- | ---- | ---- | ---- | ---- w201106-0002300153-1 | BasicPhrase | NE:OPTIONAL:アネモネ**S** | NE:OPTIONAL:アネモネ**Ｓ** | Ｓ は