Not able to do machine translation using tensor2tensor transformer model in python. It is returning nothing for translation.

[nag0811]

Description

Not able to do machine translation using tensor2tensor transformer model in python. It is returning nothing for translation. Below we can find the code that I am trying. Please correct me if I am doing anything wrong. I am running the model on CPU. ...

Environment information

OS: <Windows 10>

$ pip freeze | grep tensor
# your output here

$ python -V
# your output here

Python 3.8.3 Tensorflow 2.2.0 Tensor2Tensor 1.15.7

For bugs: reproduction and error logs

# Steps to reproduce:
...

-- coding: utf-8 --

""" Created on Thu Jul 23 14:41:01 2020

@author: nb185041 """

See what problems, models, and hyperparameter sets are available.

You can easily swap between them (and add new ones).

import tensorflow as tf from tensor2tensor.utils import registry from tensor2tensor import models #registry.list_models() from tensor2tensor import problems import json

from tensor2tensor.utils.trainer_lib import create_hparams from tensor2tensor.utils.trainer_lib import create_run_config, create_experiment import numpy as np

#PROBLEM='translate_ende_wmt32k' PROBLEM='translate_ende_wmt8k' MODEL='transformer' #HPARAMS=transformer_base_single_gpu

DATA_DIR='C:/MachineLearning/Machine_Translation/tensor2tensor/t2t_data' TMP_DIR='/tmp/t2t_datagen' TRAIN_DIR='C:/MachineLearning/Machine_Translation/tensor2tensor/t2t_data/t2t_train/translate_ende_wmt8k/transformer'

#mkdir -p $DATA_DIR $TMP_DIR $TRAIN_DIR

t2t_problem = problems.problem(PROBLEM) t2t_problem.generate_data(DATA_DIR, TMP_DIR) HPARAMS = 'transformer_base'

Init Hparams object from T2T Problem

hparams = create_hparams(HPARAMS)

Make Chngaes to Hparams

hparams.batch_size = 1024 hparams.learning_rate_warmup_steps = 45000 hparams.learning_rate = .4

Can see all Hparams with code below

print(json.loads(hparams.to_json()))

Initi Run COnfig for Model Training

RUN_CONFIG = create_run_config(model_name = MODEL, model_dir=TRAIN_DIR) # Location of where model file is store

More Params here in this fucntion for controling how noften to tave checkpoints and more.

Create Tensorflow Experiment Object

tensorflow_exp_fn = create_experiment( run_config=RUN_CONFIG, hparams=hparams, model_name=MODEL, problem_name=PROBLEM, data_dir=DATA_DIR, train_steps=50000, # Total number of train steps for all Epochs. Example 400000 eval_steps=100 # Number of steps to perform for each evaluation. example 100 )

Kick off Training

tensorflow_exp_fn.train_and_evaluate()

def encode(input_txt, encoders): """List of Strings to features dict, ready for inference""" encoded_inputs = [encoders["inputs"].encode(x) + [1] for x in input_txt]

# pad each input so is they are the same length
biggest_seq = len(max(encoded_inputs, key=len))
for i, text_input in enumerate(encoded_inputs):
    encoded_inputs[i] = text_input + [0 for x in range(biggest_seq - len(text_input))]

# Format Input Data For Model
batched_inputs = tf.reshape(encoded_inputs, [len(encoded_inputs), -1, 1])
return {"inputs": batched_inputs}

def decode(integers, encoders): """Decode list of ints to list of strings"""

# Turn to list to remove EOF mark
to_decode = list(np.squeeze(integers))
if isinstance(to_decode[0], np.ndarray):
    to_decode = map(lambda x: list(np.squeeze(x)), to_decode)
else:
    to_decode = [to_decode]

# remove <EOF> Tag before decoding
to_decode = map(lambda x: x[:x.index(1)], filter(lambda x: 1 in x, to_decode))

# Decode and return Translated text
return [encoders["inputs"].decode(np.squeeze(x)) for x in to_decode]

INPUT_TEXT_TO_TRANSLATE = 'Translate this sentence into German'

Set Tensor2Tensor Arguments

MODEL_DIR_PATH = '~/En_to_De_translator' MODEL = 'transformer' #HPARAMS = 'transformer_big_single_gpu' HPARAMS = 'transformer_base' #T2T_PROBLEM = 'translate_ende_wmt32k' T2T_PROBLEM = 'translate_ende_wmt8k' model_dir = TRAIN_DIR hparams = create_hparams(HPARAMS, data_dir=model_dir, problem_name=T2T_PROBLEM)

Make any changes to default Hparams for model architechture used during training

hparams.batch_size = 1024 hparams.hidden_size = 780 hparams.filter_size = 780*4 hparams.num_heads = 8

Load model into Memory

T2T_MODEL = registry.model(MODEL)(hparams, tf.estimator.ModeKeys.PREDICT)

Init T2T Token Encoder/ Decoders

DATA_ENCODERS = problems.problem(T2T_PROBLEM).feature_encoders(model_dir)

START USING MODELS

encoded_inputs= encode(INPUT_TEXT_TO_TRANSLATE, DATA_ENCODERS) model_output = T2T_MODEL.infer(encoded_inputs, beam_size=2)["outputs"] translated_text_in_german = decode(model_output, DATA_ENCODERS)

print(translated_text_in_german)

# Error logs:
...