ckanext-xloader
ckanext-xloader copied to clipboard
ckan master branch (test errors) on redis enqueue (due to rq 2.0.0 dropping redis below v4)
ckan master is now on rq==2.0.0
and 2.11 is on rq==1.16.2
https://github.com/rq/rq/compare/v1.16.2...v2.0
https://github.com/rq/rq/blob/master/CHANGES.md
RQ 2.0 (2024-10-28)
New Features:
Multiple job executions: a job can now have multiple executions running simultaneously. This will enable future support for long running scheduled jobs. Thanks @selwin!
Worker(default_worker_ttl=10) is deprecated in favor of Worker(worker_ttl=10). Thanks @stv8!
Added a cleanup parameter to registry.get_job_ids() and registry.get_job_count(). Thanks @anton-daneyko-ultramarin!
Added support for AWS Elasticache Serverless Redis. Thanks @bobbywatson3!
You can now specify TTL for deferred jobs. Thanks @hberntsen!
RQ's code base is now typed (mostly). Thanks @terencehonles!
Other minor fixes and improvements. Thanks @hongquan, @rbange, @jackkinsella, @terencehonles, @wckao, @sim6!
Breaking Changes:
Dropped support for Redis server < 4
RoundRobinWorker and RandomWorker are deprecated. Use --dequeue-strategy <round-robin/random> instead.
Job.__init__ requires both id and connection to be passed in.
Job.exists() requires connection argument to be passed in.
Queue.all() requires connection argument.
@job decorator now requires connection argument.
Built in Sentry integration has been removed. To use Sentry with RQ, please refer to [Sentry's docs](https://docs.sentry.io/platforms/python/integrations/rq/).
Bug Fixes:
Fixed an issue where abandoned jobs are sometimes not enqueued. Thanks @Marishka17!
Fixes an issue where Redis connection does not expose name attribute. Thanks @wckao!
job.get_status() will now always return JobStatus enum. Thanks @indepndnt!
Queue key should always be created even if jobs are deferred. Thanks @sim6!
RQ's pubsub thread will now attempt to reconnect on Redis connection errors. Thanks @fcharlier!
https://github.com/ckan/ckanext-xloader/actions/runs/13780190730/job/38536944549
https://github.com/ckan/ckanext-xloader/blob/2.1.0/ckanext/xloader/action.py#L174 https://github.com/ckan/ckan/blob/master/ckan/plugins/toolkit.py#L42 https://github.com/ckan/ckan/blob/master/ckan/lib/jobs.py#L155
https://github.com/ckan/ckanext-xloader/blob/2.1.0/.github/workflows/test.yml#L57
redis.exceptions.ResponseError: Command # 2 (HSET rq:job:a2fc19cf-4f7b-440d-921c-dd2e8f2f3af0 created_at 2025-03-11T04:13:55.956204Z
data ... success_callback_name failure_callback_name stopped_callback_name started_at ended_at last_heartbeat worker_name group_id origin ckan:test.ckan.net:default description
ckanext.xloader.jobs.xloader_data_into_datastore({'api_key': '*** enqueued_at 2025-03-11T04:13:55.956585Z timeout 3600 status JobStatus.QUEUED)
of pipeline caused error: wrong number of arguments for 'hset' command