YUM Repo

[ErisDS]

This is a place for the discussion of what needs to be done to create a YUM repo. Any / all interested parties please get stuck in as I know very little about how all this works.

cc @wolfeidau @mattwillsher

[mattwillsher]

The below is for the case where we build a ghost package and host it. The other option is to try and get ghost into EPEL, but given the likely amount of change in Ghost that may not be appropriate, going by their guidelines

For the DIY approach, taking building and hosting separately:

Building the package

Building the package needs a server running CentOS 6 or similar EL 6. From this RedHat 6 & family and Fedora packages can be built using mock.

I've reworked the package script significantly and it works but I'm not that happy with the results - it's pretty unreadable and lacks focus. But as I say, it works.

At the moment I have ansible code that can build a packaging server. This could be created at cloud provider on the fly as required. Some way of securely handling the package signing key would be needed.

One idea I did have it to have a (ansible|puppet) module that checks the current version of ghost from the Location header of the latest zip download and if the RPM doesn't exist, built it. That would decouple the RPM build from the rest of the build pipeline.

Hosting

A YUM repository is a simple web site and can be served out of S3 or similar quite easily and it can be hosted anywhere, though metadata needs creating with the createrepo tool.

Some discussion is needed about whether there are two repositories and the process of moving from one to the other. One would be for testing the packages to makes sure it works, akin to testing on Debian or EPEL, and a stable repository for more tested packages. Generally this is a good idea incase the packaging process has bugs.

There are also management tasks - removing old packages, moving packages between repositories - that will need to be done

So, that's a summary. My questions are:

• Is there currently a build system this could hook into and if so what is it? How does ghost-$latest.zip get produced?
• Is using a cloud hosting provider for the builder a possibility? Building new packages should only take a relatively short amount of time (although rebuilds are helped by having cache copies of the environments) so there isn't a need to having them running all the time. 1GB of RAM should be plenty to run a couple or more builds concurrently, though they are reasonably CPU & disk intensive.

[jgillich]

Great work. :thumbsup: I don't know much about building build rpm's either, but I could test them on my Fedora desktop.

[wolfeidau]

I had a read over the review of yum packaging, some great information there.

I have a couple of questions, as RPMs are not all born equal.

1. Who made this request, or has there just been some interest shown in the IRC / mailing discussion?
2. What is the main OS people are running?

My guess is Redhat/Centos/AmazonLinux, but will be interested in what people say.

I try and build RPMs on the base enterprise releases as it tends to be compatible across a wider range of releases than fedora, which is more edgy.

And lastly based on the project i have been hacking on it would be nice to settle on a rough layout for ghost, in a single installation situation, so at the moment I have.

/etc/ghost <- config

/opt/ghost <- installation

/var/log/ghost <- logs

/var/lib/ghost <- sqllite db

Some discussion needs to occur as to how this fits with plugins/themes/addons going forward.

So far I have built debian/ubuntu packages, these have had a couple of issues due to something relating to asset assembly, this i need to resolve.

https://github.com/wolfeidau/ghost-packaging

In this case I am using a ruby utility to generate the debian package, this can also build RPMs.

Just my two cents, hopefully gives you some things to think about.

[mattwillsher]

Hi @wolfeidau My packaging efforts came out of desire to run ghost myself and as a skill honing exercise. If it's useful to others, great.

The package will target machines runnings EPEL 6 (RHEL, CentOS, Scientific Linux, Oracle Linux, Amazon Linux and so on) and Fedora. Thanks to the low number of dependencies, packages it quite straight forward for cross distribution use.

The spec file in the RPM folder has been hand made. Tools like fpm for packaging are file, but the results can be variable. Hand writing ensures that the package files ghost well.

Regarding file paths, I agree, though there is only single configuration file so I've used /etc/ghost.conf rather than /etc/ghost/.

My current efforts are to create a build system to build standard Ghost packages in an automated way, essentially tidying up this script . Resulting packages would need to be signed and put somewhere for people to download.

[ErisDS]

So far I have built debian/ubuntu packages, these have had a couple of issues due to something relating to asset assembly, this i need to resolve.

Is this asset assembly for Ghost?

In case you aren't aware of it, there is a grunt release task available which is what creates the release zip from the Ghost source code.

In terms of separating out the config and data from Ghost... I'm not entirely sure that this is a good idea. I understand that this is how things are done in Linux - but it's going to make it mighty tricky to find all the different parts unless you are a Linux pro. If separating it is a must, I'd appreciate it if there was some docs / help to go with the package that made this very very clear. There is an open PR in Ghost for making the location of the config.js file configurable.

[wolfeidau]

aha

Firstly I would say grunt release is what I am after, is there any reason this can't just stage the install?

Reason I ask is I will need to build a zip using this then extract it again and reset permissions..

On how the platform is installed I really it depends on what your aim is with ghost, in a typical linux system one would prefer a small core, which is updated and managed by the packaging system, and a working data set which contains customisations and themes or the like.

Configuration is relocated to /etc just to reduce the proliferation of configuration files, as a dev you want it with the app, as an admin I would like it with the other configuration stuff.

So why have this core under the control of the packaging system?

• packages are signed and contain checksummed files
• package files can be owned by root and essentially immutable to the application which enables them perform privileged tasks, such as authenticity / signature checks
• package installation can be repaired or restored to original state

Again just food for thought, really up to you as to how you plan to manage security :)

[ErisDS]

Sorry, what do you mean by stage the install? This is the command I use to generate a zip for a release - there are a couple of similar ones.

We did have a jenkins server which did this on a weekly and nightly basis, I was talking with Matt about using the jenkins server to run the release plus other commands for creating packages.

[wolfeidau]

So typically you have a staging directory for the built files.

It seems you do have such a thing, if I run grunt release the contents of this archive are available in .build/release.

This should resolve the issue I had creating the package.

Cheers!

[mattwillsher]

The PR mentioned has all the work needed to build repositories for RHEL/CentOS/etc 6.x, Fedora 19+ (or any version with node 0.10.0+) for any of the support architectures though only ix86 and x86_64 have been tested.

[wolfeidau]

So is the aim for packaging to feed from the zip archives produce by the release process?

This probably makes more sense than what I am doing now, which is building out of the source tree.

Need to find some code to pop off the latest release from github and build then package that.

[mattwillsher]

@wolfeidau yes, it goes from the zip file - doing it from the source tree would couple the zip & rpm processes a little too tightly for my liking. I'm deliberately avoiding using the main ghost git repo and using the zip so I get a sanitised, pretty much ready to go set of code.

For the actual repo building, Jenkins has a git plugin that can checkout TryGhost/Ghost-Config then call the script. Once the zip build is in Jenkins the rpm build can be triggered by that job.

[mattwillsher]

I've setup a test yum repo at yum.monki.org.uk/ghost . The repo file can be copied to /etc/yum.repos.d/ and the RPM-GPG-KEY-GhostBlogTestingMonki to /etc/pki/rpm-gpg . Config is in /etc/ghost.conf. EPEL is required on CentOS/RHEL/EL 6

yum install ghost should then run. service ghost start will start ghost. The config supplied is the default, so Ghost will listen on localhost port 2368.

I've tried the package out on CentOS 6.4 x86_64

Note that this is a test - the signing key will expire in 3 months time. It may not be possible to upgrade directly from the RPMs in this repository to newer versions, it may eat your data, or blow up your house.

There are still a number of things do so:

• [ ] Automate setup of build environment
• [ ] Automatically xopy resulting yum tree to to server
• [ ] Repoint testing and stable symlinks on server
• [ ] Setup secure handling of GPG signing key
• [ ] Create ghost-release RPMs
• [ ] Test transition between versions

[jgillich]

Works fine for me on Fedora 19 x64. :thumbsup:

Did you test it on RHEL already? Otherwise I could do that.

[mattwillsher]

Thanks @jgillich . I've testing on CentOS, but RHEL wouldn't harm. You'll need to install EPEL for the nodejs package.

[ErisDS]

I really like the idea of setting this up to run from Jenkins - firing off a Jenkins build to do a release makes a lot of sense - the package will always be built in the same environment & we can hook it up our end to send the zip to Ghost.org etc.

We can package it off to as many places as possible this way :) Gimme a shout when you need Jenkins access.

[jgillich]

As you would expect, the package works perfectly fine on RHEL 6.5.

Since I've played with Jenkins and Ghost yesterday too, is there an interest in running Jenkins as an addition to Travis? It would allow us to run tests on more platforms like Windows or FreeBSD (where there are not running properly right now, thus I see a need for it).

[ErisDS]

@jgillich In a word YES! I'd love to have Jenkins setup to do all kinds of tests/builds on a nightly basis, to build docs from the code, and other things that I probably haven't thought of yet :)

Is this something you'd like to help setup? .. If so we should probably move this to a separate issue to discuss what to setup how :)

[jgillich]

@ErisDS done ^

[mattwillsher]

I too am keen on getting this integrated into Jenkins. I want to get the packages a good airing while that's pending though. My tentative aim for the packages is to have them 'official' and in their production state by 0.4.0, as long as that's a few weeks or more away.

[mattwillsher]

A few notes on the build process, that may affect how the Jenkins job(s) are structured:

• Two source RPMs are built - one for epel-6, the other fedora-19. They can be built concurrently, e.g.:

./build_ghost_rpm -r epel-6-x86_64 -s & 
./build_ghost_rpm -r fedora-19-x86_64 -s &

• Once the source RPMs are build, the binary RPMs can be built concurrently.

./build_ghost_rpm -r epel-6-x86_64 &
./build_ghost_rpm -r epel-6-i386 &
./build_ghost_rpm -r fedora-19-x86_64 &
./build_ghost_rpm -r fedora-19-i386 &

• RPM signing happens at the time the package is built. It could be batched at the end via a simple find -exec run, as noted in the README.md
• Adding a new build target requires a new root config in /etc/mock. I guess when fedora 20 ships, mock will be updated to have a fedora-20 config, likewise when EPEL 7 gets released.