Protocol / integration documentation

[oparisy]

Hi,

my understanding is that this Blender addon is communicating with external engines using a network protocol.

Is this protocol documented? Is it designed with Panda3D in mind (since I reckon it is used as the basis of BlenderPanda), or do you feel it has a more general scope?

Regards, Olivier.

[BluePrintRandom]

What about using a jvm to call compiled libs both directions?

Pgi has some success integrating a engine running JS, Lua, and python all in 1 shot.

On Jan 29, 2017 9:59 AM, "oparisy" [email protected] wrote:

Hi,

my understanding is that this Blender addon is communicating with external engines using a network protocol.

Is this protocol documented? Is it designed with Panda3D in mind (since I reckon it is used as the basis of BlenderPanda https://github.com/Moguri/BlenderPanda), or do you feel it has a more general scope?

Regards, Olivier.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Kupoman/BlenderRealtimeEngineAddon/issues/10, or mute the thread https://github.com/notifications/unsubscribe-auth/AG25WZCcknA5sfofyDUWJYjx3uB2wwlwks5rXNOXgaJpZM4Lw2wD .

[Moguri]

@oparisy The protocol is not currently documented, but I will try to make documentation a higher priority now that someone else has expressed interest in the project. In the meantime, you can look at the Server class in BlenderPanda's processor_app.py as an example implementation: https://github.com/Moguri/BlenderPanda/blob/master/processor_app.py#L33. If your external process happens to be in Python, BlenderPanda is MIT licensed, so feel free to just copy that Server class.

While BlenderPanda is currently the only consumer of this project that I know of, brte was designed to work with any engine or language. If you run into issues with the design when trying to integrate your project, please feel free to create an issue.

@BluePrintRandom brte relies on a socket protocol for communicating to external engines. This means it is already language agnostic.

[oparisy]

@Moguri Thanks for your answer! Maybe I can share some of your documentation effort by drafting my understanding of the BlenderPanda class you pointed me to? I'll include some questions in italics and edit it with your precisions, if you are OK with this. Following your suggestion, I'll also insert some proposals for better platform/language independence (hopefully without breaking compatibility with the existing BlenderPanda client).

Protocol specification (edited following answers below):

• The Blender addon listens on the 5555 TCP port.
• Once a client is connected, the addon drives the dialog by sending it messages.
• All messages begin with an unsigned short describing their type.
• A type of 0 (MSG_DATA) describes JSON data sent from the addon to the client (a scene update in glTF format). It is followed by the data length (an unsigned int), then the data in ascii encoding (using UTF-8 is discussed at #11). The client then acknowledge by sending a single 0 byte.
• A type of 1 (MSG_UPDATE) describes a query for an image (a frame rendering) by the addon. It is followed by a float (the elapsed time since last frame). The client must then send the width and height of the image as two unsigned short, then send the image as an array of size width*height*3 (24bit RGB, or BGR if RealTimeEngine.use_bgr_texture is set. Image data is in the glTexImage2D format).

All data types used above are described here. They are interpreted in native byte order (fixing byte order is discussed ar #12).

Regards, Olivier.

[Moguri]

For reference, here is the other side of the socket communication.

• The socket is a TCP port with a default protocol of 5555. The port can be changed via an argument to the ExternalProcessor constructor.
• The overall architecture is a request/response setup with the client (brte) making requests to a server (external process). For the most part, yest, brte is driving things.
• The two defined message types are MSG_DATA=0 and MSG_UPDATE=1 (these are defined in external_processor.py.
• MSG_DATA is an scene update provided as glTF data provided by blendergltf. This is actually a delta instead of the full scene every time there is an update.
• MSG_UPDATE tells the engine to process a frame using the supplied dt (delta time, which is the time since the last frame/update). BlenderPanda does not actually use the supplied dt value since it just constantly renders. We may change this part of the API. The response to a MSG_UPDATE request is an image. The image data should be 24bit RGB (or BGR if RealTimeEngine.use_bgr_texture is set). The plan is to support more image formats in the future (especially compressed ones). This type of setting is set when the RenderEngine is created (see panda_engine.py in BlenderPanda for an example) instead of wasting bytes every image update. The image data should be compatible with glTexImage2D data parameter, which states:

The first element corresponds to the lower left corner of the texture image. Subsequent elements progress left-to-right through the remaining texels in the lowest row of the texture image, and then in successively higher rows of the texture image. The final element corresponds to the upper right corner of the texture image.

• I don't know if we really defined what to do with an unrecognized request. If the request was expecting more than one byte (e.g., the MSG_UPDATE request), then we'll probably hang waiting for the other three bytes.

I have created an issue for the byte order (#12) and using UTF8 (#11). Specifying a byte order seems like a no brainer. We're unlikely to run into issues since the server and client will likely run on the same machine, but there is no need to leave it undefined. As for UTF8, I overall like the idea, but we'll have to evaluate the effect on message size and network bandwidth. We have a lot of data to send and not much time to do it.

We currently have room for 64k request types (we might need to re-evaluate using an unsigned short), and we did plan to be able to add new requests. One such plan is to allow for user/engine defined requests if addons need them. We also haven't, yet, dealt with input data, which may require some new request types. Pause/resume requests may also be useful.

[BluePrintRandom]

This socket could be used with blender game, to send game models in and get blender meshes out?

(to run blender BPY operations inside the bge?)

if you could provide a simple example, it would make me quite happy.

On Mon, Jan 30, 2017 at 7:42 PM, Mitchell Stokes [email protected] wrote:

For reference, here https://github.com/Kupoman/BlenderRealtimeEngineAddon/blob/develop/brte/processors/external_processor.py is the other side of the socket communication.

• The socket is a TCP port with a default protocol of 5555. The port can be changed via an argument to the ExternalProcessor constructor.
• The overall architecture is a request/response setup with the client (brte) making requests to a server (external process). For the most part, yest, brte is driving things.
• The two defined message types are MSG_DATA=0 and MSG_UPDATE=1 (these are defined in external_processor.py https://github.com/Kupoman/BlenderRealtimeEngineAddon/blob/develop/brte/processors/external_processor.py .
• MSG_DATA is an scene update provided as glTF https://github.com/KhronosGroup/glTF data provided by blendergltf https://github.com/Kupoman/blendergltf. This is actually a delta instead of the full scene every time there is an update.
• MSG_UPDATE tells the engine to process a frame using the supplied dt (delta time, which is the time since the last frame/update). BlenderPanda does not actually use the supplied dt value since it just constantly renders. We may change this part of the API. The response to a MSG_UPDATE request is an image. The image data should be 24bit RGB (or BGR if RealTimeEngine.use_bgr_texture is set). The plan is to support more image formats in the future (especially compressed ones). This type of setting is set when the RenderEngine is created (see panda_engine.py https://github.com/Moguri/BlenderPanda/blob/master/panda_engine.py in BlenderPanda for an example) instead of wasting bytes every image update. The image data should be compatible with glTexImage2D https://www.opengl.org/sdk/docs/man/html/glTexImage2D.xhtml data parameter, which states:

The first element corresponds to the lower left corner of the texture image. Subsequent elements progress left-to-right through the remaining texels in the lowest row of the texture image, and then in successively higher rows of the texture image. The final element corresponds to the upper right corner of the texture image.

• I don't know if we really defined what to do with an unrecognized request. If the request was expecting more than one byte (e.g., the MSG_UPDATE request), then we'll probably hang waiting for the other three bytes.

I have created an issue for the byte order (#12 https://github.com/Kupoman/BlenderRealtimeEngineAddon/issues/12) and using UTF8 (#11 https://github.com/Kupoman/BlenderRealtimeEngineAddon/issues/11). Specifying a byte order seems like a no brainer. We're unlikely to run into issues since the server and client will likely run on the same machine, but there is no need to leave it undefined. As for UTF8, I overall like the idea, but we'll have to evaluate the effect on message size and network bandwidth. We have a lot of data to send and not much time to do it.

We currently have room for 64k request types (we might need to re-evaluate using an unsigned short), and we did plan to be able to add new requests. One such plan is to allow for user/engine defined requests if addons need them. We also haven't, yet, dealt with input data, which may require some new request types. Pause/resume requests may also be useful.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Kupoman/BlenderRealtimeEngineAddon/issues/10#issuecomment-276267193, or mute the thread https://github.com/notifications/unsubscribe-auth/AG25WTcEwgv25NeTIRDOVOKiu0cUuMLyks5rXq2SgaJpZM4Lw2wD .

[agoose77]

Long term, it might be good to look at separating ordered from unordered data [and write your own reliable & throttling protocol]. I expect that there are some operations such as object creation and destruction, which don't have any required ordering. Transformations obviously demand those features.

I mention this because at the moment, this is TCP based, but required to be practically realtime. Any network problems during transmission will stall the stream, which may be undesirable. However, for simplicity, it might be best if one could say that this is acceptable as a potential issue.

[oparisy]

@Moguri Thanks for those precisions. I edited my message accordingly to remove dangling questions.

Now that you described the protocol, I understand that it aims at sending Blender scene data to an external engine/renderer, which then returns back a frame to be displayed in a Blender viewport.

What I do not understand is how the frame width and height are negotiated. It seems like the external engine can choose an arbitrary frame size... while we may have expected the Blender addon to send its actual/preferred viewport size. In the current BlenderPanda implementation, how is frame size specified? What happens if Blender viewport is resized?

I suppose the existing protocol could also be used to send data to an engine for rapid prototyping purpose (update asset in Blender -> "realtime" update in the engine), in which case a "dummy" image could be sent back to Blender (should the external engine have its own window/viewport). This was the use case I originally had in mind when I encountered your addon.

Regards, Olivier.

[Moguri]

@BluePrintRandom A BGE game could use this socket API, but only to receive scene data from Blender and send image data back. If you really want to use bpy in the BGE, you can. It isn't available in the blenderplayer, but it can be added if make a small change in the source code. However, this is a different discussion than this issue is about.

@agoose77 For the moment TCP is working fine performance wise, and I prefer the reliability and ease of use. I do not foresee us dealing with many network problems since the main use case for this is over a loopback socket. If you're not running this over a loopback connection, the uncompressed image data will be a problem long before TCP issues since we require somewhere around 3 Gbps of bandwidth for a full HD viewport to run at 60fps.

@oparisy We send over viewport information (width, height, projection_matrix, view_matrix) as part of a view object in the glTF extras. BlenderPanda handles this here. If the viewport is resized, BlenderPanda resizes its buffer as well.

You could have your external application send back a dummy image. I am not certain that you can get the behavior you're describing with brte, but I would start by looking into overriding some RealTimeEngine methods in a subclass. You may have to use some different hooks from the RenderEngine API and call some things manually.