inlong
inlong copied to clipboard
[Bug][Sort] Sort audit cann't collect metric when load into iceberg
What happened
when I test inlong sort link and wanto to verify the correctness of audit data, but I find there is no audit data when I test sort link loading data into iceberg;
What you expected to happen
Audit data is normally stored when I test sort link loading data into iceberg;
How to reproduce
- prepare a mysql 5.7+ and flink 1.13 and inlong audit env
- create a mysql table and insert some data
use inlong;
create table if not exists `inlong_input13` (
`id` bigint,
`name` varchar(63),
PRIMARY KEY(id)
);
insert into inlong_input13 (id, name) values (1, 'haha');
insert into inlong_input13 (id, name) values (2, 'xixi');
- create a sort link in flink client
CREATE TABLE inlong_input13 (
`id` bigint,
`name` STRING,
PRIMARY KEY(id) NOT ENFORCED
) WITH (
'connector' = 'mysql-cdc-inlong',
'hostname' = 'localhost',
'port' = '3306',
'username' = 'root',
'password' = '123456',
'database-name' = 'inlong',
'table-name' = 'inlong_input13',
'scan.incremental.snapshot.enabled'='true');
CREATE TABLE inlong_iceberg13 (
id bigint,
name string,
PRIMARY KEY(id) NOT ENFORCED
) WITH (
'connector'='iceberg-inlong',
'catalog-name'='hive_prod',
'catalog-database'='inlong',
'catalog-table'='inlong_iceberg13',
'uri'='thrift://localhost:9083',
'warehouse'='hdfs://localhost:8020/hive/warehouse',
'sink.ignore.changelog' = 'true',
'inlong.group_stream_node'='group_1&stream_2&node_3', --验证metric的,可加可不加,加了那么指标就会发送给带groupId的指标就会发送给pushgateway
'metrics.audit.proxy.hosts'='localhost:10081'
);
- check there has any data in inlong audit storage database.
use apache_inlong_audit;
select * from audit_data;
the correct result is like :
+----+---------------+-----------+-----------+---------------------+-----------+---------------------+-----------------+------------------+----------+-------+------+-------+---------------------+
| id | ip | docker_id | thread_id | sdk_ts | packet_id | log_ts | inlong_group_id | inlong_stream_id | audit_id | count | size | delay | updateTime |
+----+---------------+-----------+-----------+---------------------+-----------+---------------------+-----------------+------------------+----------+-------+------+-------+---------------------+
| 1 | 172.21.78.175 | | 79 | 2022-09-16 17:38:02 | 1 | 2022-09-16 17:36:00 | group_1 | stream_2 | 8 | 8 | 97 | 0 | 2022-09-16 09:38:21 |
+----+---------------+-----------+-----------+---------------------+-----------+---------------------+-----------------+------------------+----------+-------+------+-------+---------------------+
Environment
No response
InLong version
master
InLong Component
InLong Sort
Are you willing to submit PR?
- [X] Yes, I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct