davisking
Add input_tensor input type
These changes add a new input type that enables feeding networks with batches of data that already reside in device memory.
For context, I am developing a robot simulator for batch reinforcement learning. A deep Q-network receives inputs generated from an OpenGL pipeline that renders a camera view representation of the world to a texture. This texture can be read using CUDA graphics interoperability. The experiences then accumulate in device memory. To avoid the round trip from host to device, I added this input layer.
I think the functionality could be useful beyond my application.
Thanks for the PR.
Why not just call .forward() with your tensor you have on hand though? Like instead of calling operator() you can call forward() and give that the tensor on device directly.
As I understand, .forward() only supports feeding a single tensor that is already assembled. I can concatenate the tensors from replay memory before hand, but is that not the responsibility of the network input?
There is a larger issue though with training. The dnn_trainer expects the input to properly support to_tensor() for the expected input_type. Therefore, if I do not define this input class, I cannot train the model like below
// "replay" is a container of tensors that were read from device memory
auto batch = sample(replay.begin(), replay.end(), batch_size, rng);
trainer.train_one_step(batch.begin(), batch.end(), target_values.begin());
I would have to develop my own trainer.
For inference, the same situation applies. I will be streaming video frames from a camera connected to a Jetson system. nvarguscamerasrc is used to capture the frames, and they reside in device memory.
Let me know if I am missing something, and thanks!
Ah, didn't realize you wanted to train with it. Yeah this is cool, makes sense :D
Can you add a short unit test to check that it works and then I'll merge it?