conductor icon indicating copy to clipboard operation
conductor copied to clipboard

Postgres indexing problem with task_type length

Open vinize opened this issue 1 year ago • 3 comments

Describe the bug When changing the indexing database from elastic to postgres, an error occurs related to the length of the task name.

Caused by: com.netflix.conductor.core.exception.NonTransientException: ERROR: value too long for type character varying(32)
	at com.netflix.conductor.postgres.dao.PostgresBaseDAO.getWithRetriedTransactions(PostgresBaseDAO.java:148) ~[conductor postgres-persistence.jar!/:?]
	at com.netflix.conductor.postgres.dao.PostgresBaseDAO.queryWithTransaction(PostgresBaseDAO.java:210) ~[conductor-postgres-persistence.jar!/:?]
	at com.netflix.conductor.postgres.dao.PostgresIndexDAO.indexTask(PostgresIndexDAO.java:151) ~[conductor-postgres-persistence.jar!/:?]
	at com.netflix.conductor.core.dal.ExecutionDAOFacade.updateTask(ExecutionDAOFacade.java:517) ~[conductor-core.jar!/:?]
	... 18 more

It turns out that there is a problem with the length of the task_type field. According to the database schema, it should be up to 32 characters long. However, in the case of SIMPLE tasks, the task_name values go there instead of task_type

Details Conductor version: 3.18 Persistence implementation: Postgres Queue implementation: Redis Lock: Redis Workflow definition:

{
  "accessPolicy": {},
  "name": "TestWorkflowWithLongTaskName",
  "description": "Test Workflow to Index long task name in postgres",
  "version": 1,
  "tasks": [
    {
      "name": "TaskWithLongNameInTestWorkflowWithLongTaskName",
      "taskReferenceName": "TaskWithLongNameInTestWorkflowWithLongTaskName",
      "inputParameters": {},
      "type": "SIMPLE",
      "startDelay": 0,
      "optional": false,
      "asyncComplete": false,
      "permissive": false
    }
  ],
  "inputParameters": [],
  "outputParameters": {},
  "schemaVersion": 2,
  "restartable": true,
  "workflowStatusListenerEnabled": false,
  "ownerEmail": "[email protected]",
  "timeoutPolicy": "ALERT_ONLY",
  "timeoutSeconds": 0,
  "variables": {},
  "inputTemplate": {}
}

To Reproduce Steps to reproduce the behavior:

  1. Use Postgres Index
  2. Create workflow with task which has name more then 32 symbols
  3. Run it and terminate (because lack of worker)

Expected behavior In case on simple tasks value 'SIMPLE' should be inserted to task_type column, or task_type column should be extended to 255 length (like task_name)

Additional context bug initially reported at: https://github.com/Netflix/conductor-community/issues/252

vinize avatar Mar 27 '24 08:03 vinize

I have faced exactly the same issue, had to revert back to ES indexing

Prebiusta avatar Mar 27 '24 14:03 Prebiusta

OK - should be pretty straightforward to change the column length in the database schema for this

bjpirt avatar Apr 02 '24 08:04 bjpirt

I think this issue can be closed

Prebiusta avatar Apr 17 '24 19:04 Prebiusta