Limit Concurrent Builds

[lsenft]

Is there a way to limit concurrent builds? We are currently on a resource constrained server and have run into some issues when a dev pushes several commits at once.

Also is there a better place for community support? Not sure this is ticket worthy, but I didn't see any forums, etc.

[dancryer]

The best solution here is to use the daemon mode, by running daemonise phpci:daemonise instead of running console phpci:run-builds on a cron job.

The daemon mode will only run one build at a time. :)

[tvbeek]

I'm not sure if I need to create a new ticket or use this one.

Is there a possibility to run one concurrent build for each project but allow to run more then one builds on the same time (say one build for project x and one build for project y but not 2 builds for project x)

[dancryer]

There isn't presently, no.

I'm open to adding this kind of logic, but it isn't there now.

[lsenft]

I'll check out daemonize. We were on 1.x, then yesterday i forked the project so I could write a plugin for cakephp unit testing. After seeing all the other improvements I upgraded our production ci server to the latest. I also bumped up our hardware specs, we were on the lowest rackspace.

I'm sure it will perform better. I do think it would be nice to have some control over builds. We could have handled 4-5 concurrent builds okay without running out of disk space. Since we are a small shop I think one build at a time will be fine, that would have crippled my last team that had around 10 devs.

[dancryer]

On the hosted service, we have multiple copies of the daemon running, so you get parallel builds, but each builder only runs one build at a time. That could work for you also...

[robertscherer]

Hi,

we have had some problems with the daemonised phpci, which resulted in huge error output with "MySQL server has gone away" and instances in which the daemon would not process new builds.

We have simply fixed this with a cron script running every minute:

phpci_cron.sh

#!/bin/bash
if ps -ef |grep -v grep |grep "phpci:run-builds"; then
        exit 0
else
    echo "starting phpci cron"
    /var/www/phpci/console phpci:run-builds >> /var/log/phpci.log
    exit 0
fi

This works like a charm. Thanks for this amazing tool!

[gumeniukcom]

For docker i'm using this script:

#! /bin/sh
while true; do

if ps -ef |grep -v grep |grep "phpci:run-builds"; then
        echo "nothing todo"
else
    echo "starting phpci cron"
    /phpci/console phpci:run-builds
fi
  sleep 15
done

Default daemon have problem with mysql after some hour working ( is https://github.com/Block8/PHPCI/issues/616 )

[junskihere]

hi all, @lsenft, just wanna ask how did you fork so that you can run cakephp test cases inside phpci? having sum trouble doing it.

[lsenft]

I didn't get it working 100% at the job where I had PHPCI implemented. Since then I've started working freelance and haven't had any ongoing projects where it would be worth getting it set up.

On Tue, Nov 11, 2014 at 12:53 AM, junskihere [email protected] wrote:

hi all, @lsenft https://github.com/lsenft, just wanna ask how did you fork so that you can run cakephp test cases inside phpci? having sum trouble doing it.

— Reply to this email directly or view it on GitHub https://github.com/Block8/PHPCI/issues/517#issuecomment-62513291.

[SamMousa]

I think this is the default behavior now: https://github.com/Block8/PHPCI/blob/master/PHPCI/Command/RunCommand.php#L104

[elboletaire]

@SamMousa It seems, but sometimes it fires multiple times at once.

Yesterday I was doing some tests using a local phpci and I tried to add multiple builds and then executed the phpci:run-builds multiple times at once. Every time I've tried to run the phpci:run-builds it showed me a notice like "There's a currently running build".

Today I've updated phpci in my office server and after some commits there has been one push that triggered four builds at a time. A part from the fact that the server slowed down too much, some of that builds started showing this error:

fatal: destination path '/var/www/phpci/PHPCI/build/50' already exists and is not an empty directory.

Failed to clone remote git repository.
Exception: Could not create a working copy.

Probably is another issue, but started showing with this huge build.

In my case I'm using the normal cron for phpci:run-builds instead of the daemon.

[elboletaire]

As said in my previous comment:

As you can see it executes the builds totally random :confused: .

[elboletaire]

(careful, I've made a lot of comments before ^)

I think that I can confirm that when it starts making such a lot of builds, builds coming later crash with the following error(s):

fatal: destination path '/var/www/phpci/PHPCI/build/70' already exists and is not an empty directory.

Failed to clone remote git repository.
Exception: Could not create a working copy.

This is common too:

Cloning into '/var/www/phpci/PHPCI/build/78'...
error: cannot run /var/www/phpci/PHPCI/build/78.sh: El fitxer o directori no existeix
fatal: unable to fork

Exception: Warning: unlink(/var/www/phpci/PHPCI/build/78.key): No such file or directory in /var/www/phpci/PHPCI/Model/Build/RemoteGitBuild.php line 108

The sentence El fitxer o directori no existeix means "The file or folder does not exist".

After crashing this way, if I trigger a manual build everything seems to work as expected.

[Adirelle]

Do not start several run-builds at all. In the current state of the code, run-builds fetches a list of builds to run at the very start of its execution and try to build them all without refreshing its data between each build. Several run-builds in parallel will stomp on each other toes, trying to build the same jobs at nearly the same time.

[elboletaire]

And that's why it crashes?

What's the solution then? Create a custom script like described by @robertscherer?

[Adirelle]

That's it.

[Adirelle]

@elboletaire: also, if the flock command is available, you can try this (much simpler):

flock --exclusive --nonblock /var/run/phpci.lock /var/www/phpci/console phpci:run-builds >> /var/log/phpci.log

This will exclusively lock the file /var/run/phpci.lock during the execution of phpci:run-builds, silently failing if the file is already locked (i.e. another instance of phpci:run-builds is running).

[elboletaire]

That's a little better.

[SamMousa]

Why not use a true task scheduler instead of trying to reinvent the wheel? https://github.com/illuminate/queue For example would allow multiple runners relatively easy.

Also, there could be some pruning in each build, for example:

1. Build task starts.
2. Build task checks if the commit it must be run against fits the requirements (like "there must not be a commit on this branch, after the current commit, that has a build finished or in queue")
3. Build task runs

Since step 1 and 2 are fast it can skip task 3 if 2 is not ok; this will keep it simple and makes it so very little actual queue management is required.