different runs on same data produce slightly different outputs

[Enumaris]

I'm doing a classification task where I have multiple segments (more than 2) and my data is prepared into a format like [PAD,...,PAD,A,SEP,B,SEP,C,SEP,...,SEP,CLS]

To do this, I used the xlnet.py file as per the suggestion of the "Custom Usage" section. I pretty much just copied what the code in that section said to do, but since I have more than 2 segments for my problem, I had to write my own data input functions.

I grab summary = xlnet_model.get_pooled_out(summary_type="last") (just getting the CLS embedding) and then just put a single dense layer on top of that summary. I define my placeholders for those inputs, define my loss and an Adam optimizer and train the model using a tf.Session() and then save the overall model using a tf.save_model.simple_save().

When I reload the model and run it on the exact same input data multiple times, I get slightly differing outputs each time. The difference is slight, but significant enough to affect my average metrics by 3-5%. I am having a hard time understanding how a model loaded from memory can have non-deterministic behavior. I have double checked the inputs (input_ids, seg_ids, input_mask), and they are exactly the same across runs (to confirm that the tokenizer wasn't sampling different tokenizations for some unknown reason). I am pretty baffled by this behavior, can anyone shed some light? The difference seems to be too large to be some sort of floating-point rounding error, and the combinatorical shuffling of the inputs done to pre-train XLNET is shut off at finetuning time right? Is it possible the shuffling is still happening without my knowledge or something like that?

[Enumaris]

After thinking through things a bit more, I realized I just never turned off the training mode so the discrepancies are probably brought about by the dropout layers or other layers that behave randomly during training. What would be the best suggested way to do that (turn off training mode at run time) on XLNET on a saved model? Should I call "create_run_config" again after training, before saving my model, with training set to false?

[alexpnt]

You can turn it off mannuay:

    xlnet_config.dropout = 0.0
    xlnet_config.dropatt = 0.0

[Syrup274]

I truned off the dropout, set is_training=False and is_finetune=True, but get different results for seq_out. Did you encounter the same problem?