Running luamqtt through the Qt eventloop

[syyyr]

Hi,

I am embedding Lua in my Qt application. However, it seems to me that running luamqtt would require me to run it in a separate thread, because it uses its own eventloop. Is it possible to run luamqtt via a custom eventloop? The workflow would be:

1. Open a socket via Qt
2. Wait for data via the Qt event-loop
3. When something gets read, the C++ code runs a lua function, that handles the communication, possibly calling some sort of a callback that sends data over the socket back.
4. The connection would stay alive for the runtime of the C++ application

Is it possible to implement something like this? I can see that there is an iteration function, that would maybe work for this usecase? I'm not how many iterations would one operation take and also, I don't want the Lua code to block if the connection is disrupted (when an operation gets a timeout). Or maybe I can use a custom connector? From what I can read, the connector doesn't allow me to use a custom eventloop.

Thanks

[xHasKx]

Hi @syyyr,

Yes, luamqtt can be used in a dedicated event loop. Please take a look at https://github.com/xHasKx/luamqtt#connectors

You have to write a lua module with several functions - to establish and close a TCP connection, and to send/read packets into it. Then you use that module as a connector field when creating mqtt client instances from the lib. Example - https://github.com/xHasKx/luamqtt/blob/master/examples/copas.lua#L15

I suppose those functions have to deal with coroutines somehow to be paused and continued when your Qt's event loop performs that connect/close/send/read operations.

I would appreciate it if you will share your solution as an example so I can add it to this repo.

[xHasKx]

Here is a brief workflow that should work.

So in your Qt application, you create a Lua state (lua_newstate) and a Lua coroutine in it (lua_newthread).

Then you start that coroutine with lua_resume. That coroutine should run a Lua code to create an mqtt client instance with your connector module set, in sync mode - https://github.com/xHasKx/luamqtt/blob/master/examples/sync.lua. When the client calls connector.connect(), the control goes to the C function which starts to open a TCP connection and then pauses the coroutine with the lua_yield call, and lua_resume finally returns control back to your application.

When you got an event about opened connection, you can resume the Lua coroutine with the same lua_resume call. Thus the lua code will continue, and then will format an MQTT CONNECT packet for you to send through the connector.send() call, which can work in the same way as connect sequence I described above, with pause until you receive a data sent application event. And the same for other connector methods.

Documentation is here - https://www.lua.org/manual/5.3/manual.html#lua_yield

The only thing tricky here is sending PINGREQ MQTT packets periodically to maintain the MQTT connection open. I think you can call the Lua method client_mt:send_pingreq() by some timer in your application.

[syyyr]

Thanks for the quick reply.

In the end, I decided that it would be easier for me to just use Qt::Mqtt. However, I did understand your explanation and find it quite interesting. I'd like to learn more about coroutines in Lua, if I find some time, I'll try to make an example Qt app.

I'll leave the issue open until I can create the example, but for now, my problem is solved, so feel free to close this, if you want. :)

[Tieske]

I think this can be done, but it's quite hard. The library is very unsafe for concurrent use and will not handle partial data received or sent (and you cannot assume the data received/sent to be the full packet all at once, despite that it will mostly work).

I've made a fast number of changes to handle those situations (see https://github.com/xHasKx/luamqtt/pull/31 for my branch).

• reading the socket requires yield/resume on partial data, hence the 2 base classes for the connector (buffered /non-buffered, see https://github.com/Tieske/luamqtt/tree/keepalive/mqtt/connector/base) for blocking and non-blocking IO loops
• sending data cannot be assumed to be atomic, hence it needs a lock in an async environment (see https://github.com/Tieske/luamqtt/blob/keepalive/mqtt/connector/copas.lua#L81-L94 for example)
• timeouts should not be static. When waiting for data the timeout should be disabled (to wait forever), whilst when sending receiving packets the timeout should temporary be set to a reasonable short one, eg 10 seconds.

[xHasKx]

@syyyr ,

In the end, I decided that it would be easier for me to just use Qt::Mqtt

Of course, it will be the best solution if you have such an option available.

@Tieske ,

The library is very unsafe for concurrent use and will not handle partial data received or sent (and you cannot assume the data received/sent to be the full packet all at once, despite that it will mostly work).

Actually, connectors have to ensure they have sent/received the same amount of data as passed to their send()/receive() methods. Without breaking that rule the library should work stable in concurrent environment...

... and that was my thoughts before I saw this (my) code: https://github.com/xHasKx/luamqtt/blob/e3fa62c81251840c930d42b44e382187ab66b109/mqtt/client.lua#L1152-L1158

So connector usage has to be changed to a single send() call for the mqtt packet. I'll fix that.

By the way, I'm thinking about splitting library to two parts - separate mqtt protocol implementation from the transport layer (client logic, connectors, ioloop), maybe by several github repositories.

[Tieske]

that send logic is correct. LuaSocket will not guarantee that the whole packet was sent at once. I don't think there's anything you can do about it, except what I already did in my PR #31 . It has all those changes and has been running for months now, without issues.

That PR also separates the client from the runloop. So maybe best to continue with that PR first, and then separate the io-loop stuff out in a separate repo.

[xHasKx]

that send logic is correct. LuaSocket will not guarantee that the whole packet was sent at once.

I mean that the code with a while loop ensuring the whole packet was sent should be inside the connector.send() method, not in the client_mt:_send_packet() code