docs on converting to .trt format

[korejan]

Hi, in the docs about converting to .trt protobuf format, there are very little details. I think it should mention the fact that the --max_batch_size paramater should match the batch size used/expected for inference.

Initially when i used the same batch size in the documentation this increased the tensorrt excute/dispatch time significantly. When i re-created the .trt file using the same batch size as my program then the time matched using the original model.

Also example.gen_serialized_engine does not expose a GPU data type parameter which again has an effect if you're expecting to use say half floats instead of the default type. I've modifed the code in my local repo that exposes this parameter, don't know if something you guys are interested in me submitting, pretty trivial to add.

[ganler]

Hi @korejan , thanks for your advice. I will enhance the documents according to your feedbacks. When doing batching, we'll convert the images bytes into float type. And we did allow users to set the engine execution type. I didn't make the data type setting available in the gen_serialized.cpp for the reason that float32 is compatible in many GPU environment. (I mean some machines doesn't support float16/int8 and will just throw unexpected errors) I think we enhance the different tensorrt data type support in the future. And you're welcome to join us.

Feel free to have a PR, so that we can discuss the pros and cons in detail.