Design for Database connection?

[kithokit]

Hi I am going to implement an Application using IronWorker. But I got something weird for using IronWorker.

My Background task is going to do update my database on some records, write some statistics data to DB. The whole task is not long ( >500ms or less than 2sec but long enough to be async task). This task is always be triggered as lots of requests come in my Web tier.

My Question is if using ironworker, i have to connect and disconnect the database connection everytime the task start. I am not sure if this would be good design as it seems trigger a lot of overhead to the database connection.

Or is there other way to do?

Thanks

Kit

[rkononov]

Hey Kit, each task is completely is launching in completely isolated environment and you should prepare connection in each task, but you could workaround this using 1) batch process - launch task only when you need to process more than 10 records 2) you could put payload to IronMQ and and process such records in one task (same as #1 but use IronMQ as persister)

2014-07-31 16:56 GMT+06:00 Kit Ho [email protected]:

Hi I am going to implement an Application using IronWorker. But I got something weird for using IronWorker.

My Background task is going to do update my database on some records, write some statistics data to DB. The whole task is not long ( >500ms or less than 2sec but long enough to be async task). This task is always be triggered as lots of requests come in my Web tier.

My Question is if using ironworker, i have to connect and disconnect the database connection everytime the task start. I am not sure if this would be good design as it seems trigger a lot of overhead to the database connection.

Or is there other way to do?

Thanks

Kit

— Reply to this email directly or view it on GitHub https://github.com/iron-io/iron_worker_rails_example/issues/7.

[kithokit]

Not quite understand the workaround. for my architecture Web Request --> Dispatch to IronWorker to do async task (Every request assume change one record in DB) for workaround 1) How can we know there is more than 10 records in Web Tier? for workaround 2) Are you talking about a Poll Queue approach? If yes, the background task will need to stand forever and keep polling the IronMQ?

I would like to know if IronWorker Is good fit for my use case? ( update DB for some rows, >500ms or less than 2sec but long enough to be async task).

[rkononov]

in this case only second approach will fit your needs - put them in MQ and process in scheduled task, or fire up task each time you need to process piece of data but i'm not sure that preparing connection and update db could take ~2 seconds (upto 5 in most cases, but you should test)

2014-07-31 17:07 GMT+06:00 Kit Ho [email protected]:

Not quite understand the workaround. for my architecture Web Request --> Dispatch to IronWorker to do async task (Every request assume change one record in DB) for workaround 1) How can we know there is more than 10 records in Web Tier? for workaround 2) Are you talking about a Poll Queue approach? If yes, the background task will need to stand forever and keep polling the IronMQ?

I would like to know if IronWorker Is good fit for my use case? ( update DB for some rows, >500ms or less than 2sec but long enough to be async task).

— Reply to this email directly or view it on GitHub https://github.com/iron-io/iron_worker_rails_example/issues/7#issuecomment-50745141 .

[carimura]

Hey Kit, so what'd you do is keep a couple workers running and have them read off IronMQ for "jobs". Your app woudl then just be responsible for putting jobs into IronMQ to be read off by workers. This way the IronWorker task will only open up a connection to the DB when there are avaialble jobs and process as many as it can. Maybe 10, 20, 100, and then die.

You can use push queues to kick up the IronWorker the first time, and it will read as many jobs as it can until queue is empty.

[kithokit]

Your proposal would be scheduled the iron worker tasks to be run in time interval (say 1min), then read as many as job in the IronMQ. Am I Correct?

[rkononov]

Yes that was one of the proposed solutions

2014-08-01 8:23 GMT+06:00 Kit Ho [email protected]:

Your proposal would be scheduled the iron worker tasks to be run in time interval (say 1min), then read as many as job in the IronMQ. Am I Correct?

— Reply to this email directly or view it on GitHub https://github.com/iron-io/iron_worker_rails_example/issues/7#issuecomment-50842714 .

[carimura]

And/or use our alerts feature to trigger the IronWorker to run.