Bert-large model not attaining ~65% accuracy even after training till 52k timesteps!

[karthikj11]

We are using p100 and 25 gb ram to train the bert large model. But when we tried to run the default code with bs=6 and num_batch_accumulated=4, we got cuda out of memory error. Thus we changed it to bs=2 and num_batch_accumulated=8 as you said anything between 16...24 would perform similarly. But now after training till 52000 timesteps, the maximum accuracy we got is ~59.6% at 44000th timestep. Is it taking more time because we changed the batch_size? Or is there anything else we are missing out?

RESULT at 48000 and 52000 timestep:

Loading model from logdir/bert_run/bs=2,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1/model_checkpoint-00048000 DB connections: 100% 166/166 [02:31<00:00, 1.10it/s] 100% 1034/1034 [05:45<00:00, 2.99it/s] DB connections: 100% 166/166 [00:00<00:00, 448.81it/s] Wrote eval results to logdir/bert_run/bs=2,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1/ie_dirs/bert_run_true_1-step48000.eval 48000 0.5638297872340425

Loading model from logdir/bert_run/bs=2,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1/model_checkpoint-00052000 DB connections: 100% 166/166 [00:00<00:00, 443.91it/s] 100% 1034/1034 [05:31<00:00, 3.12it/s] DB connections: 100% 166/166 [00:00<00:00, 467.06it/s] Wrote eval results to logdir/bert_run/bs=2,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1/ie_dirs/bert_run_true_1-step52000.eval 52000 0.586073500967118

[berlino]

We tuned the bert model very carefully with a very small lr (3e-6), but the price to pay is that it has to be trained for much longer. I guess you can still expect the performance to be >60% if you keep training. Another option is that you could try a larger bert lr (like 1e-5).

[DevanshChoubey]

Hi @berlino,

can you please tell me when can we expect those reparsed trees that are corrupt and is that the reason also for lower accuracy.

[DevanshChoubey]

Hi @karthikj11

is there a possibility that you can share your trained model?? with me???

[berlino]

Hi, @DevanshChoubey I don't think that's the reason.

Sorry for the confusion. There seems to be only one or two sqls in Spider that is not well-structured. The patch we do is actually available at Richard's repo https://github.com/rshin/seq2struct/blob/master/data/spider-20190205/train_spider.json.patch .

[DevanshChoubey]

thanks @berlino kudos to you and @alexpolozov

just one more question you guys trained on a v100 with 16gb mem with the same default code with bs=6 and num_batch_accumulated=4 without any out of memory errors ????

can you guys please share the optimal trained model if possible???

[senthurRam33]

@berlino While evaluating dev dataset did you guys further cleaned it?? Because I trained the model and tested it for accuracy and it gives me only around 60% at 70000 timesteps.

[alexpolozov]

@DevanshChoubey We might be able to share our trained checkpoints later but that would require a separate release review process, unfortunately. Will take a few weeks to go through.

@berlino Richard actually re-parses every SQL into an AST. Many ASTs (more than two) in the original Spider release were broken (https://github.com/taoyds/spider/issues/3). The authors fixed their SQL parser later but did not re-generate the ASTs. The patch file is actually orthogonal, it fixes the SQL string in a couple instances, not the AST. I thought that this would be updated in the latest release of Spider, but – after checking with them – apparently it wasn't. So we need to include the re-parsing code in the preprocessing of this release. This is a big bug in our release and is probably responsible for many people's drop in accuracy. This is only a release bug. We have run this re-parsing in the very beginning internally, so our ASTs are correct and thus RAT-SQL can fit a more reasonable model.

[berlino]

@alexpolozov I see. I guess we could add the script https://github.com/rshin/seq2struct/blob/master/data/spider-20190205/generate.sh and the patch file to this repo, as Richard did. I could help submit a PR to fix this.

[alexpolozov]

Update: upon our request, there should be a new release of Spider with the fixed ASTs over the next few days. I think this is a better solution in the long term. Once it happens, we can close this issue.

[senthurRam33]

I have trained the BERT model and plotted the loss of my model against the loss of @alexpolozov. Both the loss curves overlaps one another. But the model achieves only 60% accuracy at 80000 steps in spider dev set. The result at 80000 timestep is as follows.

Loading model from logdir/bert_run/bs=2,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1/model_checkpoint-00080000 DB connections: 100% 166/166 [00:00<00:00, 365.57it/s] 100% 1034/1034 [05:31<00:00, 3.12it/s] DB connections: 100% 166/166 [00:00<00:00, 366.52it/s] Wrote eval results to logdir/bert_run/bs=2,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1/ie_dirs/bert_run_true_1-step80000.eval 80000 0.6015473887814313

So you guys think this could be the highest accuracy that the BERT based model can achieve in the current spider dataset or there any ways to improve the accuracy?

[karthikj11]

@alexpolozov @berlino Thanks for your quick response. Is there any way to re-parse the ASTs in the spider dataset by ourselves? Or should we wait until the new release of Spider dataset with fixed ASTs?

It would be very helpful if you can guide me to re-parse the ASTs in the current spider dataset.

[slamandar]

We tuned the bert model very carefully with a very small lr (3e-6), but the price to pay is that it has to be trained for much longer. I guess you can still expect the performance to be >60% if you keep training. Another option is that you could try a larger bert lr (like 1e-5).

Hi, have you tried larger bert lr like 1e-5 or even 5e-5? Will these lrs lead to non-convergence? thanks.

[DevanshChoubey]

@berlino @alexpolozov

Hi, i was able to reparse all the AST from @rshin generate.sh it did improve something but not much still there is a lot of difference in loss from the ideal log.txt posted by @alexpolozov and this

anything else you guys did for, that we are missing???

update-- spider released their update after reparsing the AST, still the same loss not much diff...

[senthurRam33]

@DevanshChoubey Even I had the same loss initially. Try running the model for about minimum 10000 steps so that the train loss will settle around 2.0 and val loss will settle around 6.0

[DevanshChoubey]

@senthurRam33,

yeah got that after training for 6000 steps...

anyway how much accuracy did you get???on EVAL??

[senthurRam33]

It was around 60%. But didn't check the model on the newly released spider dataset

[dorajam]

@senthurRam33 do you mind sharing which hyperparams you used? Specifically the batch size and LR? I trained for 80K steps and the performance dropped from 56% to around 51% after around the first 50K steps. It never even approached 60%. I used a batch size of 4x4 (bs x num_batch_accumulated). Thanks in advance!

[senthurRam33]

@dorajam I have only changed batch size because of memory issues. Remaining hyperparameters have been used as they are. Changed batch size 2x8 (bs x num_batch_accumulated). Try retraining the model with new spider dataset for improved accuracy.

[senthurRam33]

@alexpolozov @rshin https://github.com/microsoft/rat-sql/issues/7#issuecomment-662271122 In this issue you have added your log file. And the loss in training has been down to 0 but the val loss has stayed around 6. Perhaps if we apply the flooding technique, is there a chance that the val loss will get reduced?

[ygan]

I run the BERT model for 81K steps with the original parameters but get only 64% accuracy. The double descent doesn't appear, because it reaches 64% accuracy since 30K step. The version is 648fc87f25feeb6740679b5a3cb61bbc0465c5c7 on 15 Aug.

[senthurRam33]

@ygan Did you train the model with the new Spider dataset??

[zhangyuchen584]

Hi @berlino,

Do you use current default hyper parameters to get 69% accuracy? I noticed that in the code, the encoder dropout is set to Null(0.2 in your paper), d_x, d_z in the attention layer are set to 128(256 in the paper), decoder dropout is set to 0.2068(0.21 in the paper).

I run Bert model for 78k with default parameters, only get 56% accuracy, Wrote eval results to logdir/bert_run/bs=6,lr=7.4e-04,bert_lr=3.0e-06,end_lr=0e0,att=1/ie_dirs/bert_run_true_1-step77100.eval 77100 0.5647969052224371

and some time the experiments just converge at around 35k, then the loss score become bigger and bigger, but I can got 60% accuracy at 35k which more than training 78k.

The best accuracy I can get is 60% currently, could you tell me how to fine-tune? Many Thanks!

[berlino]

hi, @zhangyuchen584

In the paper we use dropout 0.21 ~=0.2068, encoder dropout is only used for non-bert model (encoderbert does not have this option). Which attention are you referring to?

[berlino]

hi, all

I've attached my original model config below which is used for evaluation (V3) on Spider leaderboard. After careful comparison with my original config, I think I might missed one config to commit (really sorry for this if it really matters!).

The missing config is "loss_type", which I set to "label_smooth" whereas the default is "softmax". My initial experiments show that this config does not matter if max_steps is 40k. I somehow used 'label_smooth" for training for 81k steps, but didn't have the time to test the difference with "softmax" for this 81k setting then. If you tried to activate label_smooth, please let me know if it really matters. Thanks

{ "model": { "decoder": { "desc_attn": "mha", "dropout": 0.20687225956012834, "enc_recurrent_size": 1024, "enumerate_order": false, "loss_type": "label_smooth", "name": "NL2Code", "recurrent_size": 512, "use_align_loss": true, "use_align_mat": true }, "decoder_preproc": { "grammar": { "clause_order": null, "end_with_from": true, "factorize_sketch": 2, "include_literals": false, "infer_from_conditions": true, "name": "spider", "output_from": true, "use_table_pointer": true }, "save_path": "preproc/", "use_seq_elem_rules": true }, "encoder": { "bert_token_type": true, "bert_version": "bert-large-uncased-whole-word-masking", "name": "spider-bert", "summarize_header": "avg", "update_config": { "cv_link": true, "name": "relational_transformer", "num_heads": 8, "num_layers": 8, "sc_link": true }, "use_column_type": false }, "encoder_preproc": { "bert_version": "bert-large-uncased-whole-word-masking", "compute_cv_link": true, "compute_sc_link": true, "db_path": "database/", "fix_issue_16_primary_keys": true, "include_table_name_in_column": false, "save_path": "preproc/" }, "name": "EncDec" }, }

[zhangyuchen584]

Hi @berlino,

I used default 'desc_attn' = 'mha' . in the models/attention.py file, MultiHeadedAttention class, around 105 line, self.d_k = value_size // h which is equal 128(I run the experiments using default parameters). In the paper, attention dx = dz = 256.

Am I misunderstanding? Thank you.

[mellahysf]

Hi all,

someone can share the best pretrained model ? with code to use it directly to generate an SQL request from text ?

Thanks.

[DevanshChoubey]

@berlino @alexpolozov

Finally after changing the loss to label_smooth, I got 65 percent accuracy at around 22000 steps, gonna keep training, let's see how much it can improve....

anyway , thanks again for all the help, you guys are the real heroes...

[Sea-Snell]

I was having trouble with this for a while, getting 59% and stuff, but then I tried using label_smooth and Bert LR 1e-5, and got 68.8% after 42k steps. So I recommend trying that

[DevanshChoubey]

Hi @Sea-Snell,

yeah, exactly even I am at 67% at 26000 steps, still training going on...but will same LR 3e-6

latest update

got 69.2 accuracy at 41800

seems label_smooth was the deal breaker at least for me

[sanjayss34]

@DevanshChoubey could you please paste your config here? I think I'm using the default settings except with label_smooth but I'm only getting 60.9% accuracy at 23100 steps, rather than 65% like you said. Thanks!