High CPU usage with eventlet

[hard2explain]

I'm using Flask on a project on an embedded system and I'm having performance issues. I'm running gunicorn with one eventlet worker by running:

gunicorn -b 0.0.0.0 --worker-class eventlet -w 1 'app:create_app()'

The problem I'm facing is that, when the MQTT messages start to pour with more cadence, the application starts to use almost all the CPU I have available. My initial thought was that I was handling the messages not ideally but, I even took out my handler, and just receive the messages, and the problem still persists.

I have another python application, running on the same platform, that subscribes to the same information with the paho client and this is not an issue, so I'm assuming I'm missing something on my Flask application and it's not the information itself.

My code is:

import eventlet

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_login import LoginManager, current_user
from flask_socketio import SocketIO
from flask_mqtt import Mqtt

eventlet.monkey_patch()

#USERS DB
db_alchemy = SQLAlchemy()

#socketIO
socketio = SocketIO(cors_allowed_origins="*", async_mode='eventlet')

# MQTT
mqtt_client = Mqtt()

'''
 APPLICATION CREATION
'''
def create_app():
    app = Flask(__name__)
    if app.config["ENV"] == "production":
        app.config.from_object("config.ProductionConfig")
    else:
        app.config.from_object("config.DevelopmentConfig")

    #USERS DB
    db_alchemy.init_app(app)

    #LoginManager
    login_manager = LoginManager()
    login_manager.login_view = "auth.login"
    login_manager.init_app(app)

    #SOCKETIO
    socketio.init_app(app)

    #FLASK-MQTT
    app.config['MQTT_BROKER_URL'] = 'localhost'  #
    app.config['MQTT_BROKER_PORT'] = 1883  
    app.config['MQTT_KEEPALIVE'] = 20  
    app.config['MQTT_TLS_ENABLED'] = False  
    mqtt_client.init_app(app)

    return app

#MQTT
@mqtt_client.on_connect()
def mqtt_on_connect():
    mqtt_client.subscribe('testTopic/#', 0)

@mqtt_client.on_disconnect()
def mqtt_on_disconnect():
    loggerMqtt.warning(' > Disconnected from broker')

@mqtt_client.on_subscribe()
def mqtt_on_subscribe(client, obj, mid, granted_qos):
    pass

@mqtt_client.on_message()
def mqtt_on_message(client, userdata, message):
    pass
    #mqtt_topicSplitter(client, userdata, message)

As you can see my handler mqtt_topicSplitter is commented but I'm still having performance issues. I've tried adding an sleep command [eventlet.sleep(0.1)] on the on_message handler which solved the CPU consumption problem but resulted on my application being constantly kicked from the broker.

I'm sorry if I wasn't clear, but I'm not an expert, please feel free to ask me any questions if needed.

Thanks in advance.

[bbartling]

You ever figure this out?

[hard2explain]

You ever figure this out?

No, I'm still stuck with this problem

[bbartling]

Can you use MQTT outside of Flask? Curious if that would make a CPU difference or not

[hard2explain]

If I don't connect to the broker it works perfectly, the MQTT is clearly the issue.

[umar7alabi]

From the Flask Mqtt documentation there is multi workers limitation so as you use many clients connecting in the same time it might be the reason. P.s. I am having the same problem and trying to find the solution

[hard2explain]

From the Flask Mqtt documentation there is multi workers limitation so as you use many clients connecting in the same time it might be the reason. P.s. I am having the same problem and trying to find the solution

This also happens with only one client connected