TensorFlow-Unreal-Examples
TensorFlow-Unreal-Examples copied to clipboard
how to communicate real time?
I have a question, I've trained a model in TensorFlow.Keras and now my python editor in the Unreal Engined 4.18.3, have all modules I need (i.e TensorFlow and Keras) but in the case of the driving simulator I need communication between python scripts and the Unreal Environment, how can I do that?
You can check out the old qlearning branch examples for in the loop communication: https://github.com/getnamo/tensorflow-ue4-examples/blob/qlearn/Content/Scripts/PongAI.py
The basics of it is call a json input with your ai input e.g. wheel location, images as float arrays, etc, then run your python code inside the function https://github.com/getnamo/tensorflow-ue4-examples/blob/qlearn/Content/Scripts/PongAI.py#L63 and return a result. On unreal side use the updated outputs to update your simulation (e.g. steering).
thanks for sharing, but you know, using the pip command in the same directory I could update the TensorFlow, ((pip install TensorFlow == 2.6.1)) and now the python editor in the Unreal Engine 4.18.3 recognize (Tensorflow. Keras) but another question has arisen is, how can I use the Unreal output as an Input for python script? for example, I want a way of communication between python and the Unreal engine. for example, my NN model takes a bunch of inputs and outputs desired steering degree. I require that the car in the Unreal give me its steering degree and compare it with my model output to produce an error.
Convert your input to json and reshape the inputs into the size/dimensions your network expects.
There's a handy struct to json string converter which can be used to feed the input, see this section of the documentation https://github.com/getnamo/tensorflow-ue4#basic-json-string and https://github.com/getnamo/tensorflow-ue4#basic-json-string
Keep in mind typically NNs expect float array inputs.