Nanobox server not running on Shippable CI

[mbrodala]

Currently launching Nanobox in a CI environment (thus using the CI bootstrap script) fails:

Error   : nanobox-server service start was successful but the service is not running
Context : 

Any idea how to get more output for this?

[lyondhill]

this happens when a ci does not start services the same way as a native linux machine would. shippable is a testable CI.

[mbrodala]

See this build log for details. And the repository source.

[lyondhill]

So the problem I have found so far is that on shippable instead of running inside a vm like other CI's on shippable runs inside a docker container. then in that container they clone the git repo inside /root/src/. This creates 2 problems: 1: nanobox is expecting a standard service manager (systemd, upstart, etc). But nanobox cannot start its server 2: Shippable grants access from inside the container to the docker sock file which is good. but nanobox trys to mount the code folder into the docker container ( -v :/app) the problem with this is that in shippable they just clone the nanobox code folder inside the container and it doesnt really exist on the host (technically it does exist but under the docker container id) so when the docker server on the host gets the request to mount /root/src/ that folder doesnt exist on the host.

[mbrodala]

I'm not that much into Docker yet but do you see a way to have Nanobox' containers directly access directories of Shippable's build container? Do you see any other way to solve this?

As for the startup issue I'm not sure if there is anything I can do. Can't the Nanobox server be started in background as regular binary?

[mbrodala]

@lyondhill I've created a Shippable support ticket, maybe you can watch this ticket and join the discussion once more detailled questions show up.

[mbrodala]

Do you have any other ideas how this could be solved given the options to customize the build?

[mbrodala]

@lyondhill Do you think issue 2 can be fixed as described here?

pre_ci_boot:
  options: "-v /app:/app"

This mounts the directory /app on the host to the /app directory on the container. By copying data/code to the /app directory in the container in a CI task, we make those files available in the /app directory on the host. The /app directory can be replaced by any other directory.

Then later cp -R . /app.

[mbrodala]

Is there any way to launch the Nanobox server without a service manager?

[danhunsaker]

Actually, yes, there is. As root: nanobox server

[mbrodala]

I just tried that and the process keeps hanging afterwards:

nanobox server --trace  

   0.00/10.72MB [                                          0.00%]

   ...

   10.72/10.72MB [****************************************  100.00%]
2017/06/13 07:12:52 <nil>

I guess there is no forking into background if the server is started this way? But I also cannot simply do a nanobox server & to start it in the background since then the following Nanobox commands wouldn't work as long as the server is not yet started (which takes a considerable amount of time). And waiting for the background process also wouldn't help since it never finishes (or shouldn't) unless it's an error what I'm seeing here.

Any idea how to get this working?

[danhunsaker]

Looks like it's actually trying to run nanobox-update first thing. Maybe run that manually ahead of time? Once that closes, you should be able to start the server in the background just fine, I believe.

[mbrodala]

That didn't work out well and yes, the issue is obvious:

nanobox-update --trace  
/root/a52197e6-7908-4ce6-a7ef-3f5c1ee0273e.sh: line 73: nanobox-update: command not found

I'll try to manually download this binary.

[mbrodala]

After loading the nanobox-update binary and making it executable the behavior is unchanged except that apparently the update is now performed twice. Again, nanobox server hangs at the end:

nanobox-update --trace 

   0.00/15.10MB [                                          0.00%]

   ...

   15.10/15.10MB [****************************************  100.00%]

nanobox server --trace

   0.00/10.72MB [                                          0.00%]

   ...

   10.72/10.72MB [****************************************  100.00%]
2017/06/13 10:36:15 <nil>

[danhunsaker]

Interesting size difference, there, too...

As to hanging at the end, that would be the server actually running. I'm not sure what it's actually downloading, at this point, though, or why. One of the staff members will have to weigh in to get much further.

[mbrodala]

Alright, I hope a staff member chimes in then. Thanks so far.

[mbrodala]

As mentioned here it should be possible to basically add another volume mount to the Shippable build container, copy everything into that directory and thus make it accessible on the host. If another container then volume mounts that directory, it should be able to access the code.

[mbrodala]

I've now asked Shippable if they could implement a feature to completely override the build container command (currently docker run). This way I should be able to use Nanobox to run the build itself.

[mbrodala]

Currently I'm trying to install Nanobox on the Shippable build host instead and mount it into the build container. However currently this fails:

# bash -c "$(curl -fsSL https://s3.amazonaws.com/tools.nanobox.io/bootstrap/ci.sh)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                     
            Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 15.0M  100 15.0M    0     0  60.7M      0 --:--:-- --:--:-- --:--:-- 60.8M
--------------------------------------------------------------------------------
+ ERROR:
+ Nanobox is designed to run as a standard user (non root)
+ Please run all nanobox commands as a non privilage user
--------------------------------------------------------------------------------
Nanobox is ready to go!

# nanobox version
Using nanobox with docker-machine requires tools that appear to not be available on your system.
vboxmanage
View these requirements at docs.nanobox.io/install/requirements/docker-machine/

There are various issues:

1. Running Nanobox as root should be allowed on CI systems.
2. The bootstrap script happily continues even if an error occurs in the middle, it should stop and fail hard instead.

[danhunsaker]

It should be working as root in CI mode. Wonder if the CI bootstrap sets CI mode first or not. It'll have to do that before anything else.

I think that's part of why it's trying to use VirtualBox, too.

[mbrodala]

Small update: the error is caused by the CI bootstrap script which runs nanobox config set provider native before nanobox config set ci-mode true. If the latter is the first invocation of Nanobox, that part works just fine.

[danhunsaker]

Indeed. This confirms my suspicion, above.

PR submitted to fix this in nanobox-io/bootstrap#27...

[mbrodala]

Thanks to that fix I was able to successfully install Nanobox on the build host machine:

build_pre_ci
  bash -c "$(curl -fsSL https://s3.amazonaws.com/tools.nanobox.io/bootstrap/ci.sh)"
  nanobox version
  Nanobox version 2.1.2 (eea68d66ac4e86987e8a15215bd2a06a)

However, currently I am unable to properly mount the Nanobox binary into the build container. Neither mounting it as single file (using -v ~/../usr/local/bin/nanobox:/usr/local/bin/nanobox) nor mounting the whole parent folder (using -v /usr/local/bin:/usr/local/bin/host) works. The Nanobox binary always appears as directory in the build container ...

[mbrodala]

Yesterday I tried the same though a simple test and it worked just fine:

$ docker run -it --rm -v $(which nanobox):/usr/local/bin/nanobox php bash
# ls -l /usr/local/bin/
total 38764
-rwxrwxr-x 1 root root       118 Jul  3 21:22 docker-php-entrypoint
-rwxrwxr-x 1 root root      1276 Jul  3 21:22 docker-php-ext-configure
-rwxrwxr-x 1 root root      2288 Jul  3 21:22 docker-php-ext-enable
-rwxrwxr-x 1 root root      2223 Jul  3 21:22 docker-php-ext-install
-rwxrwxr-x 3 root root       587 Jul  3 21:22 docker-php-source
-rwxr-xr-x 1 root staff 15841242 Jun 15 21:34 nanobox
-rwxr-xr-x 2 root staff      817 Jul  3 21:27 pear
-rwxr-xr-x 2 root staff      838 Jul  3 21:27 peardev
-rwxr-xr-x 2 root staff      754 Jul  3 21:27 pecl
lrwxrwxrwx 1 root staff        9 Jul  3 21:27 phar -> phar.phar
-rwxr-xr-x 2 root staff    14829 Jul  3 21:27 phar.phar
-rwxr-xr-x 2 root staff 11828056 Jul  3 21:27 php
-rwxr-xr-x 2 root staff     2620 Jul  3 21:27 php-config
-rwxr-xr-x 2 root staff 11959520 Jul  3 21:27 phpdbg
-rwxr-xr-x 2 root staff     4547 Jul  3 21:27 phpize
# nanobox

CONFIGURE NANOBOX
---------------------------------------------------------------
Please answer the following questions so we can customize your
nanobox configuration. Feel free to update your config at any
time by running: 'nanobox configure'

(Learn more at : https://docs.nanobox.io/local-config/configure-nanobox/)

How would you like to run nanobox?
  a) Inside a lightweight VM
  b) Via Docker Native

  Note : Mac users, we strongly recommend choosing (a) until Docker Native
         resolves an issue causing slow speeds : http://bit.ly/2jYFfWQ

Answer: ^C

Since this still does not work on Shippable, I'll ask over there.

[mbrodala]

See https://github.com/Shippable/support/issues/3707

[mbrodala]

That didn't work out. In the end I managed to get the Nanobox CLI to exist on the build host but that didn't make a difference.

[mbrodala]

BTW I also just tried CircleCI but again, server startup fails:

sudo nanobox server


   0.48/10.73MB [*                                         4.48%]   1.46/10.73MB [*****                               2.41/10.73MB [********        2.89/10.73MB [**********                                 3.37/10.73   3.85/10.73MB [**************            4.33/10.73MB [****************                          40.36%]
   4.81/10.73MB [******   5.28/10.73MB [*******************                 5.7   6.24/10.73MB [****************   6.72/10.73MB [*************************                 6   7.20/10.73MB    7.69/10.73MB [**************************   8.63/10.73MB [*********   9.11/10.73MB [*********************************      9.59/1   10.07/10.73MB [*      10.73/10.73MB [****************************************  100.00%]
2017/07/11 07:56:57 <nil>

[pablorsk]

I have the same problem, on Bitbucket Pipelines:

+ bash -c "$(curl -fsSL https://s3.amazonaws.com/tools.nanobox.io/bootstrap/ci.sh)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  9.7M  100  9.7M    0     0  53.3M      0 --:--:-- --:--:-- --:--:-- 53.6M
Successfully set 'ci-mode'
Successfully set 'provider'
Nanobox is ready to go!

+ nanobox -v
Nanobox Version 2.3.0-180117T2039 (4b3d14f)

+ nanobox config set ci-mode true
Successfully set 'ci-mode'

+ nanobox remote add apicultor

✓ You've successfully logged in

✓ Codebase linked to apicultor


+ nanobox deploy




Error   : nanobox-server service start was successful but the service is not running
Context : 


Skipping cache upload for failed step
Searching for test report files in directories named [test-results, failsafe-reports, test-reports, surefire-reports] down to a depth of 4

Also I try with

nanobox server --verbose &
sleep 30
nanobox deploy
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Failed to connect to bus: No such file or directory
Error   : nanobox-server service start was successful but the service is not running
Context : 

:cry:

[mbrodala]

@pablorsk Thanks for checking, that was on my TODO list but now I am sure that it won't ever work either.

[seanculver]

@mbrodala I've successfully ran CI with circleci using my nanobox thanks to using this Gist from @brandoncc - https://gist.github.com/brandoncc/e680d39490c434b059486fa08fc089c6

Here is my elixir version of the same thing and even has an auto deploy workflow built in. https://gist.github.com/seanculver/91bd62f31e2feec589e0f91644caf0df

I haven't figured out caching of the box yet but I'll update my gist with the caching stuff, it added a few minutes to my build but I'll see if I can't get it back to where it was.

Hope this helps!

What Brandon said was it might be the machine: true flag that kicks it out of a docker container and allows us to run our own docker container via nanobox, this is just a hunch though.

[mbrodala]

@seanculver Thanks a lot for the hints. I can confirm that this setup is working fine. I'll look forward to even faster builds once caching is working. :-)