tensor2tensor
tensor2tensor copied to clipboard
Not able to do machine translation using tensor2tensor transformer model in python. It is returning nothing for translation.
Description
Not able to do machine translation using tensor2tensor transformer model in python. It is returning nothing for translation. Below we can find the code that I am trying. Please correct me if I am doing anything wrong. I am running the model on CPU. ...
Environment information
OS: <Windows 10>
$ pip freeze | grep tensor
# your output here
$ python -V
# your output here
Python 3.8.3 Tensorflow 2.2.0 Tensor2Tensor 1.15.7
For bugs: reproduction and error logs
# Steps to reproduce:
...
-- coding: utf-8 --
""" Created on Thu Jul 23 14:41:01 2020
@author: nb185041 """
See what problems, models, and hyperparameter sets are available.
You can easily swap between them (and add new ones).
import tensorflow as tf from tensor2tensor.utils import registry from tensor2tensor import models #registry.list_models() from tensor2tensor import problems import json
from tensor2tensor.utils.trainer_lib import create_hparams from tensor2tensor.utils.trainer_lib import create_run_config, create_experiment import numpy as np
#PROBLEM='translate_ende_wmt32k' PROBLEM='translate_ende_wmt8k' MODEL='transformer' #HPARAMS=transformer_base_single_gpu
DATA_DIR='C:/MachineLearning/Machine_Translation/tensor2tensor/t2t_data' TMP_DIR='/tmp/t2t_datagen' TRAIN_DIR='C:/MachineLearning/Machine_Translation/tensor2tensor/t2t_data/t2t_train/translate_ende_wmt8k/transformer'
#mkdir -p $DATA_DIR $TMP_DIR $TRAIN_DIR
t2t_problem = problems.problem(PROBLEM) t2t_problem.generate_data(DATA_DIR, TMP_DIR) HPARAMS = 'transformer_base'
Init Hparams object from T2T Problem
hparams = create_hparams(HPARAMS)
Make Chngaes to Hparams
hparams.batch_size = 1024 hparams.learning_rate_warmup_steps = 45000 hparams.learning_rate = .4
Can see all Hparams with code below
print(json.loads(hparams.to_json()))
Initi Run COnfig for Model Training
RUN_CONFIG = create_run_config(model_name = MODEL, model_dir=TRAIN_DIR) # Location of where model file is store
More Params here in this fucntion for controling how noften to tave checkpoints and more.
Create Tensorflow Experiment Object
tensorflow_exp_fn = create_experiment( run_config=RUN_CONFIG, hparams=hparams, model_name=MODEL, problem_name=PROBLEM, data_dir=DATA_DIR, train_steps=50000, # Total number of train steps for all Epochs. Example 400000 eval_steps=100 # Number of steps to perform for each evaluation. example 100 )
Kick off Training
tensorflow_exp_fn.train_and_evaluate()
def encode(input_txt, encoders): """List of Strings to features dict, ready for inference""" encoded_inputs = [encoders["inputs"].encode(x) + [1] for x in input_txt]
# pad each input so is they are the same length
biggest_seq = len(max(encoded_inputs, key=len))
for i, text_input in enumerate(encoded_inputs):
encoded_inputs[i] = text_input + [0 for x in range(biggest_seq - len(text_input))]
# Format Input Data For Model
batched_inputs = tf.reshape(encoded_inputs, [len(encoded_inputs), -1, 1])
return {"inputs": batched_inputs}
def decode(integers, encoders): """Decode list of ints to list of strings"""
# Turn to list to remove EOF mark
to_decode = list(np.squeeze(integers))
if isinstance(to_decode[0], np.ndarray):
to_decode = map(lambda x: list(np.squeeze(x)), to_decode)
else:
to_decode = [to_decode]
# remove <EOF> Tag before decoding
to_decode = map(lambda x: x[:x.index(1)], filter(lambda x: 1 in x, to_decode))
# Decode and return Translated text
return [encoders["inputs"].decode(np.squeeze(x)) for x in to_decode]
INPUT_TEXT_TO_TRANSLATE = 'Translate this sentence into German'
Set Tensor2Tensor Arguments
MODEL_DIR_PATH = '~/En_to_De_translator' MODEL = 'transformer' #HPARAMS = 'transformer_big_single_gpu' HPARAMS = 'transformer_base' #T2T_PROBLEM = 'translate_ende_wmt32k' T2T_PROBLEM = 'translate_ende_wmt8k' model_dir = TRAIN_DIR hparams = create_hparams(HPARAMS, data_dir=model_dir, problem_name=T2T_PROBLEM)
Make any changes to default Hparams for model architechture used during training
hparams.batch_size = 1024 hparams.hidden_size = 780 hparams.filter_size = 780*4 hparams.num_heads = 8
Load model into Memory
T2T_MODEL = registry.model(MODEL)(hparams, tf.estimator.ModeKeys.PREDICT)
Init T2T Token Encoder/ Decoders
DATA_ENCODERS = problems.problem(T2T_PROBLEM).feature_encoders(model_dir)
START USING MODELS
encoded_inputs= encode(INPUT_TEXT_TO_TRANSLATE, DATA_ENCODERS) model_output = T2T_MODEL.infer(encoded_inputs, beam_size=2)["outputs"] translated_text_in_german = decode(model_output, DATA_ENCODERS)
print(translated_text_in_german)
# Error logs:
...