runners & colliding databases

[avramd]

Hello,

I'm having trouble with Hydra hanging part way through a run. Sometimes gets as far as 20 specs, sometimes as few as 2. Then it just sits there, with no advancement after 10x as long as all my specs take to run serially.

I am only running locally (one worker), and things run fine when I specify only one runner. This is a rails app, and many specs use the same models as one another, so my best guess is that my database is getting frotzed by simultaneous access to the same objects. Sound like a good theory? After things go south, even running only 1 worker fails until I blow away my db and start from scratch.

I went through closed tickets, and found https://github.com/ngauthier/hydra/issues/43. This is a question about separate db's per runner, and the answer is "yes, but you have to do it yourself" - however the answer points to an init file for workers, not for runners. So...

How would you use a worker init file to get separate DB's per runner?

Also, would it be hard to port the worker init file feature to 0.20.0? If I'm seeing things correctly, it's not in there. Sadly we're on RSpec 1.x and Rails 2.x.

Thanks!

[ngauthier]

Hi @avramd,

Before we embark on the process of backporting a feature to setup multiple databases, let's make sure that's the problem. Check out how hydra logs deadlocks.

When a deadlock is detected the master will log that message. Try running hydra with verbose set to true (see Verbose Output) and see if that message comes out.

@ngauthier

[avramd]

Happily. Will hopefully get back to you later tonight. Thanks for the quick response.

On 26 Dec 11 , at 11:34 AM, Nick Gauthier wrote:

Hi @avramd,

Before we embark on the process of backporting a feature to setup multiple databases, let's make sure that's the problem. Check out how hydra logs deadlocks.

When a deadlock is detected the master will log that message. Try running hydra with verbose set to true (see Verbose Output) and see if that message comes out.

@ngauthier

---

Reply to this email directly or view it on GitHub: https://github.com/ngauthier/hydra/issues/55#issuecomment-3275438

[avramd]

Hello,

Yes, I'm getting lots of deadlocks. Note that I did try adding "ActiveRecord::TestCase.use_concurrent_connections" to spec_helper.rb. While I don't quite understand the story with that, I saw the suggestion in the FAQ so figured I should see if it helped before writing back to you. It didn't ;-)

-Avram

On 26 Dec 11 , at 11:34 AM, Nick Gauthier wrote:

Hi @avramd,

Before we embark on the process of backporting a feature to setup multiple databases, let's make sure that's the problem. Check out how hydra logs deadlocks.

When a deadlock is detected the master will log that message. Try running hydra with verbose set to true (see Verbose Output) and see if that message comes out.

@ngauthier

---

Reply to this email directly or view it on GitHub: https://github.com/ngauthier/hydra/issues/55#issuecomment-3275438

[ngauthier]

So, if you're interested in backporting the worker-init to a previous version, the way to make that work locally is instead of:

1 worker => 4 runners 4 workers => 1 runner (each)

and make all the workers local. Then they each run the worker-init. In the worker init you need to connect to a different db.

Good luck!

[avramd]

Ummm yeah... now that you say it, it's pretty obvious ;-)

Do you happen to have an example of a worker init file specifying a different database? I am not yet able to visualize how it fits into the process, and therefore what mechanism it would use to do this. I'm still going through the code but I figured I'd ask in case you had a half-way relevant example floating around.

Thanks for your help, Avram

On 27 Dec 11 , at 1:21 PM, Nick Gauthier wrote:

So, if you're interested in backporting the worker-init to a previous version, the way to make that work locally is instead of:

1 worker => 4 runners 4 workers => 1 runner (each)

and make all the workers local. Then they each run the worker-init. In the worker init you need to connect to a different db.

Good luck!

[ngauthier]

I do not, sorry.

Have you tried parallel_tests? It is a much more up-to-date and stable gem. If it works with your versions it'll probably be a better fit.

-Nick

On Tue, Dec 27, 2011 at 1:41 PM, avramd < [email protected]

wrote:

Ummm yeah... now that you say it, it's pretty obvious ;-)

Do you happen to have an example of a worker init file specifying a different database? I am not yet able to visualize how it fits into the process, and therefore what mechanism it would use to do this. I'm still going through the code but I figured I'd ask in case you had a half-way relevant example floating around.

Thanks for your help, Avram

On 27 Dec 11 , at 1:21 PM, Nick Gauthier wrote:

So, if you're interested in backporting the worker-init to a previous version, the way to make that work locally is instead of:

1 worker => 4 runners 4 workers => 1 runner (each)

and make all the workers local. Then they each run the worker-init. In the worker init you need to connect to a different db.

Good luck!

---

Reply to this email directly or view it on GitHub: https://github.com/ngauthier/hydra/issues/55#issuecomment-3284022

[avramd]

I am using parallel_tests. It does not have support for remote workers. At least not that I can see. But yes, it does handle the multiple databases very cleanly.

Thanks, Avram

On 27 Dec 11 , at 4:15 PM, Nick Gauthier wrote:

I do not, sorry.

Have you tried parallel_tests? It is a much more up-to-date and stable gem. If it works with your versions it'll probably be a better fit.

[scottdavis]

try using this fork of hydra that my co-worker and I are working on.

https://github.com/johnbintz/hydra

sorry nick, havent had time to do a pull request yet =(

[scottdavis]

actually im not sure if thats going to solve the issue i misread it as being a remote worker issue =( i need to overcome dyslexia!