Easier LibreCores CI Slave Setup

[imphil]

Overall Goal

For LibreCores CI, we want to give users the ability to easily hook their machines into the CI system to run CI on their own project.

Why?

• Users have the required tools setup (that we cannot provide for legal, license and money reasons)
• FOSSi does not need to pay for all CI instances

Workflow: Set up a new slave node (rough first idea)

prerequisite: User "activates CI" for his project on librecores: essentially a web page on LibreCores (CI) where CI is enabled for a given project.

1. User downloads a installable package from the LibreCores CI page
2. User installs this package on his/her machine. The installation results in a daemon being installed.
3. As part of the installation process, the user registers this node to build "CI for project XYZ on LibreCores"
4. Jenkins master registers required build triggers with Git or librecores (details to be figured out)

Now whenever a build trigger happens, the node builds & tests the code, and uploads the results (most likely no build artifacts for license reasons) to the Jenkins master.

Constraints

• Slaves should not be required to be "always on". Users may turn off their slave nodes, run them only once a day, etc.
• Many organizations have complex firewall or proxy setups. The only communication that reliably works in such cases are HTTPS requests, initiated from the slave itself. For us, this means:
  • all communication between slave and master runs over a encrypted HTTP (HTTPS) API
  • all communication is slave-initiated: polling for new build/test jobs, status reporting, uploading of build results.

Related work:

• GitLab CI Runner: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner

[oleg-nenashev]

Which packages do we need? RPM/deb only?

[imphil]

I guess for a first version, just a script would be sufficient. While I in general don't like the curl https://example.com/install.sh | sh way of doing things, its a sufficiently simple starting point

[Houkime]

I think the "prerequisite" of having a project on Librecores is a bit too restrictive. For dev tests and such it probably would be better to do it in a way:

1. User fetches a specialized mostly-preconfigured "LCCI-slave" Docker container. (Yep, no real installer needed)
2. Special Jenkins plugin provides GUI for slave, in which user types "ci.librecores.org" or some specific IP as a desired master and can also paste a link to his project page if he wants to restrict usage or deploy only specific tools. It also should be possible to restrict max resources usage. Restriction is optional, a man can just donate librecores a temporary slave with comfortable for him configs if he wants to.
3. Then the plugin regularly polls Jenkins master for jobs over https and does them.

It is also very convenient for development as you can have LCCI master and LCCI slave on the same machine in separate Docker containers and test really any possible interaction scenario with ease.

[Houkime]

It also might prove beneficial to use Tor instead of https for master-slave connection:

Pros:

1. No need for certificate authorities or self-signing stuff. "It just works"
2. Increased security... like... I don't even know what can be safer nowadays.
3. Tor is specially designed to punch through NATs, local censorship and whatever goes in the way.
4. User can hide his IP so LCCI master can't communicate without him polling even if it wants to.

Cons:

1. Slower speed and increased time awaiting for tunnel. However it shouldn't be a real problem for LCCI tasks where files to send are small, and on-node calculation time is long.

[wallento]

@imphil Actually that is what our command line tool is supposed to do. It currently requires the user to provide a yaml file that is supposed to be downloaded from LCCI. The command line tool itself is a python tool that then does the docker stuff in background. It is not a daemon but a wrapper around docker. The yaml file contains the credentials, plus it must be changed to point to the supposed tools installation path. I still need to write a README for it.

[wallento]

Regarding communication it uses plain https to talk to LCCI, correct @oleg-nenashev (method 4)?

[wallento]

@Houkime This is kind of a build in feature of jenkins. What it would need is the ability to change the remote URL which is currently not possible, but can be easily added.