generate column doesn't show in DML event

[wk989898]

What did you do?

1. create a changefeed with Canal-JSON protocol
2. run sql

CREATE TABLE GENERATED_TABLE (
  id int PRIMARY KEY,
  A SMALLINT UNSIGNED,
  B SMALLINT UNSIGNED AS (2 * A),
  C SMALLINT UNSIGNED AS (3 * A) NOT NULL
);
INSERT INTO GENERATED_TABLE VALUES (1, 15, DEFAULT, DEFAULT);

3. decode messages from Kafka

What did you expect to see?

All column data

What did you see instead?

Column B and C are missing.

{
    "id": 0,
    "database": "test",
    "table": "GENERATED_TABLE",
    "pkNames": [
        "id"
    ],
    "isDdl": false,
    "type": "INSERT",
    "es": 1730255904223,
    "ts": 1730255936205,
    "sql": "",
    "sqlType": {
        "A": 5,
        "id": 4
    },
    "mysqlType": {
        "A": "smallint unsigned",
        "id": "int"
    },
    "data": [
        {
            "A": "15",
            "id": "1"
        }
    ],
    "old": null,
    "_tidb": {
        "commitTs": 453576203756634119
    }
}

Versions of the cluster

TiCDC version (execute cdc version):

master

[flowbehappy]

@BenMeadowcroft PTAL

[wk989898]

The root reason is that the message doesn't include the generated flag. If the generated column is sent downstream, the consumer can't distinguish generated columns, and raises an error like [CDC:ErrMySQLTxnError]MySQL txn error: Error 3105 (HY000): The value specified for generated column 'B' in table 'GENERATED_TABLE' is not allowed.

[fubinzh]

/severity major

[wk989898]

There are two kinds of generated columns: stored and virtual. A stored generated column is computed when it is written (inserted or updated) and occupies storage as if it were a normal column. A virtual generated column occupies no storage and is computed when it is read. Message carries stored generated column but not virtual generated column.