OpenMetadata
OpenMetadata copied to clipboard
Profiler - NumericValueOutOfRange Error Occur in Postgresql
Affected module Profiler Ingestion
Describe the bug
[2024-02-07704:53:57.629+0000] {server_mixin-py:66} INFO - OpenMetadata client running with Server version [1.2.5] and Client version [1.2.5.0]
[2024-02-07704:53:57.630+0000] {common.py:284} INFO - Sending status to Ingestion Pipeline dev.dev_profiler_my_schema
[2024-02-07704:53:57.671+0000] {standard_task_runner.py:104} ERROR - Failed to execute job 49 for task profiler_task
(('ProfilerProcessor reported errors', Status(source_start_time=1707280017.9226058, records=['Table [my_table]'], warnings=[], filtered=[l,
failures-[StackTraceError (name='dev.postgres.my_schema.my_table',
error='my_table.my_amount metric_type.value: (psycopg2.errors.NumericValueOutOfRange) bigint out of range\n\n
[SOL: /* {"app": "OpenMetadata", "version": "1.2.5.0"}
*/\nSELECT avg(my_amount) AS mean, count (my_amount) AS "valuesCount", count (DISTINCT my_amount) AS "distinctCount",
MIN(my_amount) AS min, MAX(my_amount) AS max, SUM(CAST(CASE WHEN (my_amount IS NULL) THEN %(param_1)s ELSE %(param_2)s END AS BIGINT)) AS "nullCount",
STDDEV_POP (my_amount) AS stddev, SUM(CAST(my_amount AS BIGINT)) AS sum \nFROM my_schema.my_table In LIMIT %(param_3)s]n[parameters:
(\'param_1\': 1, \'param_2\': 0, \'param_3\': 1)]
\n(Background on this error at: https://sqlalche.me/e/14/9h9h)*,
stack_trace='Traceback (most recent call last): \n File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py",
line 1910, in _execute_context\n self.dialect.do_execute(\n File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/default-py",
line 736, in do_executen cursor.execute(statement, parameters) \npsycopg2.errors .NumericValueOutOfRange: bigint out of rangeln\n\nThe above exception was the direct cause of the following
To Reproduce
- create table with decimal column in Postgresql table.(my_schema_my_table.my_amount above error)
- insert big number to decimal column. when error occur, sum of decimal column is -1141255742767639787836.114451487138355512129805902667630531896
- execute profiler to target table.
Expected behavior Profiler job execute successfully.
Version:
- OpenMetadata version: [1.2.5]
Additional context SQL error occur change to numeric, SQL execute successfully. before: SUM(CAST(my_amount AS BIGINT)) AS sum after: SUM(CAST(my_amount AS decimal)) AS sum