aster icon indicating copy to clipboard operation
aster copied to clipboard

anyone train with tf >=1.4

Open pageedward opened this issue 5 years ago • 3 comments

have anyone train with tf>=1.4, company server cannot allow switch cuda8,

pageedward avatar Apr 30 '19 03:04 pageedward

It failed on Tensorflow 1.13 (works in 1.4). haven't tried any other versions

njferrier avatar May 03 '19 19:05 njferrier

It failed on Tensorflow 1.13 (works in 1.4). haven't tried any other versions

can you run c_ops/build.sh ? tf <1.4 should be installed from source?

ChChwang avatar May 13 '19 08:05 ChChwang

I was able to get the model to run in a virtual environment with a pre-compiled binary for TF 1.9 (so just a pip installed version).

What I had to do was the following:

  1. In aster/c_ops/CMakeLists.txt, on line 5 under compiler flags, change the flag -D_GLIBCXX_USE_CXX11_ABI=0 to -D_GLIBCXX_USE_CXX11_ABI=1
  2. Run make clean if you've already tried building this.
  3. Rerun build.sh. Make SURE you're in the virtual environment which contains the TF version you want to use, because this script fetches the TF include path for your current environment.
  4. Next, in aster/core/sync_attention_wrapper.py, I encountered an error with the outputs of _compute_attention() around line 50. I guess newer versions of TF have this function outputting 3 values, so I captured the last one, which is supposed to contain info about the next state.
  5. Finally, I had to pass this captured value to the call to seq2swq.AttentionWrapperState() around line 65 as an additional arg attention_state=original_next_state.

Doing this, I was able to run an evaluation script using a model I had trained using the older TF version. I didn't perform any tests to compare the equivalency of the model's performance, but the outputs seemed okay.

Hope this helps.

EDIT: It should also be noted that the environment that you're running the model in must have the same compiler and version as the environment you first created the .sos in. I was able to create new .sos and keep the model though.

dsandii avatar Jul 24 '19 12:07 dsandii