salt [BUG] Mysql returner raises decimal.ConversionSyntax / InvalidOperation

Description

While using event_return: [mysql], some requests are returning exception decimal.ConversionSyntax. ❗ This is reproducible with Python mysqlclient package, except with pymysql==1.1.0 which does work.

https://github.com/saltstack/salt/blob/v3006.3/salt/returners/mysql.py#L631-L634

https://github.com/saltstack/salt/blob/v3006.3/salt/returners/mysql.py#L376-L377

Setup

Using Docker image https://github.com/saltstack/salt/releases/tag/v3006.3 , customized to install dependencies:

Alpine mariadb-dev drivers to build python package
mysqlclient==2.2.0

📝 Logs

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/salt/utils/process.py", line 993, in wrapped_run_func
    return run_func()
  File "/usr/local/lib/python3.10/site-packages/salt/master.py", line 265, in run
    salt.daemons.masterapi.clean_old_jobs(self.opts)
  File "/usr/local/lib/python3.10/site-packages/salt/daemons/masterapi.py", line 161, in clean_old_jobs
    mminion.returners[fstr]()
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/returners/mysql.py", line 634, in clean_old_jobs
    cur.execute(sql)
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 332, in _query
    self._post_get_result()
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 366, in _post_get_result
    self._rows = self._fetch_row(0)
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 339, in _fetch_row
    return self._result.fetch_row(size, self._fetch_type)
decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]


[WARNING ] Returner unavailable: [<class 'decimal.ConversionSyntax'>]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/salt/client/__init__.py", line 1153, in get_iter_returns
    if not self.returns_for_job(jid):
  File "/usr/local/lib/python3.10/site-packages/salt/client/__init__.py", line 1115, in returns_for_job
    return self.returners["{}.get_load".format(self.opts["master_job_cache"])](jid)
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/salt/returners/mysql.py", line 377, in get_load
    cur.execute(sql, (jid,))
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 332, in _query
    self._post_get_result()
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 366, in _post_get_result
    self._rows = self._fetch_row(0)
  File "/usr/local/lib/python3.10/site-packages/MySQLdb/cursors.py", line 339, in _fetch_row
    return self._result.fetch_row(size, self._fetch_type)
decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]

Config

netapi_enable_clients:
    - local
    - local_async
    - runner
    - runner_async
    - wheel
event_return: [mysql]
master_job_cache: mysql
mysql.host: 'database-host'
mysql.user: 'alcali'
mysql.pass: 'alcali'
mysql.db: 'salt'
mysql.port: 3306
# alterning keep_jobs_seconds does execute a query causing errors.
#keep_jobs_seconds: 0
keep_acl_in_token: True

Additional context Attempting to use https://alcali.dev/ . In the end i could workaround with pymysql connector.

Oct 17 '23 11:10 duhow

Do older versions of mysqlclient work? In particular to check would be 1.4.6.

Oct 17 '23 11:10 OrangeDog

Tested with mysqlclient==1.4.6 and the issue is reproducible. 🐞 Note I'm using mariadb-dev package to build the python connector:

mariadb-10.5.19-r0 x86_64 {mariadb} (GPL-2.0-or-later)
mariadb-backup-10.5.19-r0 x86_64 {mariadb} (GPL-2.0-or-later)
mariadb-bench-10.5.19-r0 x86_64 {mariadb} (GPL-2.0-or-later)
mariadb-client-10.5.19-r0 x86_64 {mariadb} (GPL-2.0-or-later)
mariadb-common-10.5.19-r0 x86_64 {mariadb} (GPL-2.0-or-later) [installed]
mariadb-connector-c-3.1.13-r0 x86_64 {mariadb-connector-c} (LGPL-2.1-or-later) [installed]
mariadb-connector-c-dev-3.1.13-r0 x86_64 {mariadb-connector-c} (LGPL-2.1-or-later) [installed]
mariadb-dev-10.5.19-r0 x86_64 {mariadb} (GPL-2.0-or-later) [installed]

Database is mariadb from 11.1.2-MariaDB.

Custom image Dockerfile:

# syntax=docker/dockerfile:1.4

FROM scratch AS dependencies
COPY requirements.txt /app/
# ---
FROM saltstack/salt:3006.3
RUN --mount=type=bind,from=dependencies,source=/app,target=/app,rw \
    apk add \
      mariadb-dev \
    && \
    pip3 install --no-cache --no-warn-script-location -r /app/requirements.txt

Oct 17 '23 16:10 duhow

I tried to reproduce this issue in a minimal environment, but it didn't work.

$ docker run --name alpine --network host -it python:3.13-alpine3.22 /bin/sh
/ # apk add gcc g++ autoconf make libffi-dev openssl-dev mariadb-dev
...
(27/33) Installing mariadb-connector-c (3.3.10-r0)
(28/33) Installing mariadb-connector-c-dev (3.3.10-r0)
...
(33/33) Installing mariadb-dev (11.4.8-r0)
...
/ # python -m venv .venv
/ # source .venv/bin/activate
(.venv) / # pip install mysqlclient
Collecting mysqlclient
...
Successfully installed mysqlclient-2.2.7

(.venv) / # python
Python 3.13.9 (main, Oct 15 2025, 16:48:03) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>> con =  MySQLdb.connect(host='127.0.0.1', port=3306, user="root")
>>> cur=con.cursor()
>>> cur.execute("SELECT VERSION()")
1
>>> cur.fetchone()
('11.4.9-MariaDB-ubu2404',)
>>> cur.execute("select date_sub(now(), interval 3 second) as stamp;")
1
>>> cur.fetchone()
(datetime.datetime(2025, 11, 25, 1, 31, 52),)

Does this issue always occur when executing a specific query, or does it occur probabilistically?

Does this issue still occur when using the latest mariadb-connector-c?

Nov 25 '25 01:11 methane

I can reproduce on alpine 3.14-3.16 (mariadb-connector-c 3.1.13). I cannot reproduce on alpine 3.17 (mariadb-connector-c 3.3.3).

Nov 25 '25 09:11 methane