inlong icon indicating copy to clipboard operation
inlong copied to clipboard

[Bug][Sort] Sort audit cann't collect metric when load into iceberg

Open thesumery opened this issue 3 years ago • 0 comments

What happened

when I test inlong sort link and wanto to verify the correctness of audit data, but I find there is no audit data when I test sort link loading data into iceberg;

What you expected to happen

Audit data is normally stored when I test sort link loading data into iceberg;

How to reproduce

  1. prepare a mysql 5.7+ and flink 1.13 and inlong audit env
  2. create a mysql table and insert some data
use inlong;
create table if not exists `inlong_input13` (
	`id` bigint,
	`name` varchar(63),
	PRIMARY KEY(id)
);
insert into inlong_input13 (id, name) values (1, 'haha');
insert into inlong_input13 (id, name) values (2, 'xixi');
  1. create a sort link in flink client
CREATE TABLE inlong_input13 (
     `id` bigint,
     `name` STRING,
     PRIMARY KEY(id) NOT ENFORCED
 ) WITH (
     'connector' = 'mysql-cdc-inlong',
     'hostname' = 'localhost',
     'port' = '3306',
     'username' = 'root',
     'password' = '123456',
     'database-name' = 'inlong',
     'table-name' = 'inlong_input13',
     'scan.incremental.snapshot.enabled'='true');
     
CREATE TABLE inlong_iceberg13  (
    id bigint,
    name string,
    PRIMARY KEY(id) NOT ENFORCED
) WITH (
    'connector'='iceberg-inlong',
    'catalog-name'='hive_prod',
    'catalog-database'='inlong',
    'catalog-table'='inlong_iceberg13',
    'uri'='thrift://localhost:9083',
    'warehouse'='hdfs://localhost:8020/hive/warehouse',
    'sink.ignore.changelog' = 'true', 
    'inlong.group_stream_node'='group_1&stream_2&node_3', --验证metric的,可加可不加,加了那么指标就会发送给带groupId的指标就会发送给pushgateway
    'metrics.audit.proxy.hosts'='localhost:10081'
);
  1. check there has any data in inlong audit storage database.
use apache_inlong_audit;
select * from audit_data;

the correct result is like :

+----+---------------+-----------+-----------+---------------------+-----------+---------------------+-----------------+------------------+----------+-------+------+-------+---------------------+
| id | ip            | docker_id | thread_id | sdk_ts              | packet_id | log_ts              | inlong_group_id | inlong_stream_id | audit_id | count | size | delay | updateTime          |
+----+---------------+-----------+-----------+---------------------+-----------+---------------------+-----------------+------------------+----------+-------+------+-------+---------------------+
|  1 | 172.21.78.175 |           | 79        | 2022-09-16 17:38:02 |         1 | 2022-09-16 17:36:00 | group_1         | stream_2         | 8        |     8 |   97 |     0 | 2022-09-16 09:38:21 |
+----+---------------+-----------+-----------+---------------------+-----------+---------------------+-----------------+------------------+----------+-------+------+-------+---------------------+

Environment

No response

InLong version

master

InLong Component

InLong Sort

Are you willing to submit PR?

  • [X] Yes, I am willing to submit a PR!

Code of Conduct

thesumery avatar Sep 19 '22 02:09 thesumery