OpenMetadata icon indicating copy to clipboard operation
OpenMetadata copied to clipboard

Profiler - NumericValueOutOfRange Error Occur in Postgresql

Open shin1103 opened this issue 1 year ago • 0 comments

Affected module Profiler Ingestion

Describe the bug

[2024-02-07704:53:57.629+0000] {server_mixin-py:66} INFO - OpenMetadata client running with Server version [1.2.5] and Client version [1.2.5.0]
[2024-02-07704:53:57.630+0000] {common.py:284} INFO - Sending status to Ingestion Pipeline dev.dev_profiler_my_schema
[2024-02-07704:53:57.671+0000] {standard_task_runner.py:104} ERROR - Failed to execute job 49 for task profiler_task 
(('ProfilerProcessor reported errors', Status(source_start_time=1707280017.9226058, records=['Table [my_table]'], warnings=[], filtered=[l,
 failures-[StackTraceError (name='dev.postgres.my_schema.my_table',
error='my_table.my_amount metric_type.value: (psycopg2.errors.NumericValueOutOfRange) bigint out of range\n\n
[SOL: /* {"app": "OpenMetadata", "version": "1.2.5.0"}
*/\nSELECT avg(my_amount) AS mean, count (my_amount) AS "valuesCount", count (DISTINCT my_amount) AS "distinctCount",
MIN(my_amount) AS min, MAX(my_amount) AS max, SUM(CAST(CASE WHEN (my_amount IS NULL) THEN %(param_1)s ELSE %(param_2)s END AS BIGINT)) AS "nullCount", 
STDDEV_POP (my_amount) AS stddev, SUM(CAST(my_amount AS BIGINT)) AS sum \nFROM my_schema.my_table In LIMIT %(param_3)s]n[parameters:
(\'param_1\': 1, \'param_2\': 0, \'param_3\': 1)]
\n(Background on this error at: https://sqlalche.me/e/14/9h9h)*, 
stack_trace='Traceback (most recent call last): \n File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/base.py", 
line 1910, in _execute_context\n self.dialect.do_execute(\n File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/default-py",
line 736, in do_executen cursor.execute(statement, parameters) \npsycopg2.errors .NumericValueOutOfRange: bigint out of rangeln\n\nThe above exception was the direct cause of the following

To Reproduce

  1. create table with decimal column in Postgresql table.(my_schema_my_table.my_amount above error)
  2. insert big number to decimal column. when error occur, sum of decimal column is -1141255742767639787836.114451487138355512129805902667630531896
  3. execute profiler to target table.

Expected behavior Profiler job execute successfully.

Version:

  • OpenMetadata version: [1.2.5]

Additional context SQL error occur change to numeric, SQL execute successfully. before: SUM(CAST(my_amount AS BIGINT)) AS sum after: SUM(CAST(my_amount AS decimal)) AS sum

shin1103 avatar Feb 07 '24 05:02 shin1103