speeding up inference nmt chatbot nlp

[b7amine]

Hello , as discribed in title i'm trying to speed up inference in raspi , I reduced num_units from 512 to 256 and reduced beam width to 1 , noticed significant difference but still 6 seconds will make my humanoid robot sound stupid , could someone help , i'm trying to build an understanding like a map that represent the full model cause it sounds complicated since it contains many levels , I already know how to manipulate a simple neural network like the one on tensorflow tuto for classifying images , i'm studying A i , NLP through this project at the same time so consider that i'm still. a beginner please , thank you. github repository: https://github.com/daniel-kukiela/nmt-chatbot#introduction hparams:hparams = { 'attention': 'scaled_luong', 'src': 'from', 'tgt': 'to', 'vocab_prefix': os.path.join(train_dir, "vocab"), 'train_prefix': os.path.join(train_dir, "train"), 'dev_prefix': os.path.join(train_dir, "tst2012"), 'test_prefix': os.path.join(train_dir, "tst2013"), 'out_dir': out_dir, 'num_train_steps': 500000, 'num_layers': 2, 'num_units': 256, 'override_loaded_hparams': True, 'learning_rate':0.001,

'decay_factor': 0.99998,

'decay_steps': 1,

'residual': True,

'start_decay_step': 1,
'beam_width': 1,
'length_penalty_weight': 1.0,
'optimizer': 'adam',
'encoder_type': 'bi',
'num_translations_per_input': 30

}