feat: warehouse_jobs support for async warehouse jobs

[chaitanyapraveen]

trafficstars

Description

Support for Asynchronous warehouse jobs to be triggered by sources(for now). The design document is attached here

Warehouse_jobs package

1. Support for post,status, stop(incomplete) and get(incomplete)
2. Support for various kinds of future warehouse jobs
3. Modifications to pgNotifier to take tasks from warehouse_jobs runner
4. Provision for slaves to selectively run warehouse_jobs based on the payload

The details on the tests are pending. Will update them shortly

Notion Ticket

https://www.notion.so/rudderstacks/Warehouse-Generic-De-dup-Framework-8d22fced31454568a6278679a43e97a1

Security

• [x] The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov[bot]]

Codecov Report

Base: 45.30% // Head: 44.29% // Decreases project coverage by -1.00% :warning:

Coverage data is based on head (f9ba0b2) compared to base (60bc1be). Patch coverage: 26.02% of modified lines in pull request are covered.

:exclamation: Current head f9ba0b2 differs from pull request most recent head 47e4d61. Consider uploading reports for the commit 47e4d61 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2319      +/-   ##
==========================================
- Coverage   45.30%   44.29%   -1.01%     
==========================================
  Files         177      181       +4     
  Lines       36542    38350    +1808     
==========================================
+ Hits        16555    16987     +432     
- Misses      18911    20288    +1377     
+ Partials     1076     1075       -1     

Impacted Files	Coverage Δ
gateway/gateway.go	54.24% <0.00%> (-0.53%)	:arrow_down:
warehouse/admin.go	3.03% <0.00%> (ø)
warehouse/bigquery/bigquery.go	0.00% <0.00%> (ø)
warehouse/clickhouse/clickhouse.go	0.00% <0.00%> (ø)
warehouse/deltalake/deltalake.go	0.00% <0.00%> (ø)
warehouse/identities.go	1.06% <0.00%> (-0.46%)	:arrow_down:
warehouse/jobs/handlers.go	0.00% <0.00%> (ø)
warehouse/jobs/jobs.go	0.00% <0.00%> (ø)
warehouse/jobs/runner.go	0.00% <0.00%> (ø)
warehouse/mssql/mssql.go	0.44% <0.00%> (-0.02%)	:arrow_down:
... and 22 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[achettyiitr]

Let's not push rudder transformer changes in here.

[achettyiitr]

Let's try to separate out logging actions and SQL statements. We have seen in the past that adding too many SQL statements in logs unnecessarily make it not much searchable.

[achettyiitr]

Most of the comments are not following Go standards. Please follow this.

[achettyiitr]

Let's enrich the error where ever possible instead of just returning it back. It will help in debugging.

[achettyiitr]

Let's using context for all the warehouse jobsdb calls. It it can either application context or it can be runner context or it can be context.TODO().