batch normalization and tfreader ops

[ViralBShah]

We are trying to port a resnet-50 implementation form python to Tensorflow.jl but are missing batch normalization and tfreader ops. Could you provide him some guidance on how he’d go about adding them?

Cc @tanmaykm @keno @MikeInnes

[ViralBShah]

Perhaps #319 is related?

[tanmaykm]

The python equivalents are:

• parsing ops: https://github.com/tensorflow/tensorflow/blob/r1.7/tensorflow/python/ops/parsing_ops.py
• batchnorm: https://github.com/tensorflow/tensorflow/blob/r1.7/tensorflow/python/layers/normalization.py

I could see some others plugged in at ops/*.jl files, but haven't quite understood the exact mechanics yet.

[oxinabox]

Please can you clarify what you mean by tfreader? That could be the name of a number of things. Can you link to the python docs for it?

[tanmaykm]

Oh those are basically the parse example methods in parsing_ops.py

[oxinabox]

Re: Batch norm. Will either of these do?

julia> import_op("FusedBatchNormV2")
fused_batch_norm_v2 (generic function with 1 method)

help?> Ops.fused_batch_norm_v2
  No documentation found.

  TensorFlow.Ops.fused_batch_norm_v2 is a Function.

  # 1 method for generic function "fused_batch_norm_v2":
  fused_batch_norm_v2(x_, scale_, offset_, mean_, variance_; name, U, epsilon, data_format, is_training) in TensorFlow.Ops

julia> import_op("BatchNormWithGlobalNormalization")
batch_norm_with_global_normalization (generic function with 1 method)

help?> Ops.batch_norm_with_global_normalization
  No documentation found.

  TensorFlow.Ops.batch_norm_with_global_normalization is a Function.

  # 1 method for generic function "batch_norm_with_global_normalization":
  batch_norm_with_global_normalization(t_, m_, v_, beta_, gamma_; name, variance_epsilon, scale_after_normalization) in TensorFlow.Ops

[oxinabox]

Re parse_examples You mean this bit? https://github.com/tensorflow/tensorflow/blob/r1.7/tensorflow/python/ops/parsing_ops.py#L363-L364 that is a 1.5 thousand line file.

It would be useful to have a like to the HTTP python docs for the operations you are looking for

[tanmaykm]

Sorry for not being clear. Maybe I should point you to the python implementation of what I am trying to run on TensorFlow.jl: https://github.com/tensorflow/tpu/tree/master/models/experimental/resnet_bfloat16

There's a call to parse_single_example here: https://github.com/tensorflow/tpu/blob/2f3c087a445dd017701025e895b1588a9a8f69d9/models/experimental/resnet_bfloat16/imagenet_input.py#L98.

And batchnorm is being used here: https://github.com/tensorflow/tpu/blob/2f3c087a445dd017701025e895b1588a9a8f69d9/models/experimental/resnet_bfloat16/resnet_model.py#L58. I am not sure if FusedBatchNormV2 or BatchNormWithGlobalNormalization can be used instead, maybe @Keno / @MikeInnes would know better?

[tanmaykm]

I see now that parse_single_example can be gotten with:

julia> import_op("ParseSingleExample")
parse_single_example (generic function with 1 method)

help?> Ops.parse_single_example
  No documentation found.

  TensorFlow.Ops.parse_single_example is a Function.

  # 1 method for generic function "parse_single_example":
  parse_single_example(serialized_, dense_defaults_; name, num_sparse, sparse_keys, dense_keys, sparse_types, Tdense, dense_shapes) in TensorFlow.Ops

But the python entrypoint has quite a bit of code before the op is actually called. Wondering where does that go or does it have to be in the Julia code.

[oxinabox]

For licencing reasons. I'm uncomfortable looking at python tensorflow source. It is Apache and this is MIT. Where as looking at the docs I am more comfortable with, aswe are clearly matching an API. Not porting code

[tanmaykm]

ah okay, I understand

[ViralBShah]

Can't we make this Apache too, if that makes things easier?