nsq
nsq copied to clipboard
nsqio
dist: an official ubuntu repository
It'd be great to have an official repository (standalone or PPA) for NSQ. It'd make it a lot easier to install NSQ.
@mreiferson Probably the easiest way would be to have an official PPA (Personal Package Archive) for now. I have never published before but I started looking into this last week, but time was a limiting factor. Here's what I gathered so far:
Guides on how to get setup and publish a PPA: http://packaging.ubuntu.com/html/getting-set-up.html http://packaging.ubuntu.com/html/packaging-new-software.html https://help.launchpad.net/Packaging/PPA/BuildingASourcePackage
A good (and similar) example of a PPA (redis). It has config files, init scripts, logrotate, ...: https://launchpad.net/~chris-lea/+archive/ubuntu/redis-server
I started "reverse engineering" it by downloading the package from: http://ppa.launchpad.net/chris-lea/redis-server/ubuntu/pool/main/r/redis/
And extracting the deb packages using the method described here: http://www.g-loaded.eu/2008/01/28/how-to-extract-rpm-or-deb-packages/
I also was able to found a package written in Go that was published to PPA, just to figure out how to publish a Go package: https://gitorious.org/lastradio/lastradio/source/5c2199c9e263b72055b55963711753e827c0b1f0:
I was thinking the package layout should be something along the lines of: /etc/conf/nsq/nsqd.conf (config file for nsqd) /etc/conf/nsq/nsqlookupd.conf (config file for nsqlookupd) /etc/conf/nsq/nsqadmin.conf (config file for nsqadmin)
/etc/init.d/nsqd (init script for nsqd) /etc/init.d/nsqlookupd (init script for nsqlookupd) /etc/init.d/nsqadmin (init script for nsqadmin)
/var/lib/nsq (to store the data files) /var/log/nsq (to store the log files by capturing the sysout/syserr and using logrotate to rotate them. redis-server already has the config for this too)
Cool, do you want to take a crack at adding the necessary scripts to contrib for building the PPA?
@mreiferson this is only a part of the story. I got this far too, but where I got stuck (mainly due to not having enough time and experience with Go and PPAs) was building and publishing to Launchpad. I'm starting to think that it'd be better if you guys would be able to host and publish the packages yourselves to a non PPA repo (similar to Elasticsearch[1]or DataStax Cassandra[2]). Alternatively we can ask to see if @chrislea would be willing to add nsq to his PPA.
[1] http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-repositories.html [2] http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installDeb_t.html
Some time ago I managed to build debian packages, including most of the files @kutchar mentioned (logs, config, init/systemd etc). This was I believe with go 1.3 and by just doing a go get, but this isn't working anymore with 1.4.2/1.5 due to the handling of internal modules, don't know the exact details.
So I thought I'd give it another try by using the makefile, but just can't get it to compile, it can't resolve the internal modules:
go build -o build/nsqd ./apps/nsqd
apps/nsqd/nsqd.go:18:2: cannot find package "github.com/nsqio/nsq/internal/app" in any of:
/usr/lib/go/src/github.com/nsqio/nsq/internal/app (from $GOROOT)
/tmp/nsq/src/github.com/nsqio/nsq/internal/app (from $GOPATH)
/tmp/nsq/.godeps/src/github.com/nsqio/nsq/internal/app
If anyone would be willing to provide some hints on how to get the environment sorted then I'd like to give it another go!
Unfortunately that doesn't seem to work either. This does make that it can find the import but again an internal package error:
go build -o /tmp/nsq/build/apps/nsqd ./apps/nsqd
package ./apps/nsqd
imports github.com/nsqio/nsq/internal/app: use of internal package not allowed
package ./apps/nsqd
imports github.com/nsqio/nsq/internal/version: use of internal package not allowed
With go 1.4 I was able to put the nsq repo wherever I wanted, and make a symlink to it in the right place in GOPATH. With go 1.5 they closed that loophole and a symlink was not sufficient, I had to actually put the repo in the GOPATH. (also, I was only building recent master)
@misja can you echo $GOPATH and pwd and confirm you have the latest code on master?
Sure, GOPATH is /tmp/nsq/.godeps:/tmp/nsq with cwd /tmp/nsq. Using go 1.5, env is:
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/tmp/nsq/.godeps:/tmp/nsq"
GORACE=""
GOROOT="/usr/lib/go"
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT=""
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"
Am definitely on master, commit c02e25ffefe222a1aa448c81e1c337f67ad01aa2
if GOPATH=/tmp/nsq then your NSQ source checkout must be in /tmp/nsq/src/github.com/nsqio/nsq
Oh but it is there - I have two checkouts, one in /tmp/nsq and then another in /tmp/nsq/src/github.com/nsqio/nsq. It can find the latter just fine, that solved the 'cannot find package' error, now it's stuck at 'use of internal package not allowed'.
I think that's the problem though, you must build from the one in the proper GOPATH subdirectory.
You're exactly right, I just did! So given a release I'll need to copy itself into src/ and then build from there - is this really the Go way? Looks so very ... weird :)
Ah and also just discovered that one can make $GOPATH/src/github.com/nsqio/nsq a relative symlink to /tmp/nsq and compile from there. Thanks @mreiferson for the help!
go upstream would prefer that you check it out into $GOPATH/src/github.com/nsqio/nsq the first time, it's odd but doesn't require any copies
OK, a ppa is up at https://launchpad.net/~misja/+archive/ubuntu/nsq. I've split the source in five separate packages nsq-common, nsq-server, nsq-lookup, nsq-admin and nsq-tools which should help manage minimal installs. The build is currently only for 14.04 (trusty), need to find some time now to enable it for other series. Both upstart and systemd support is included, latter not well tested, give it a try.
Yes. And no. If the build machine has a debian toolchain installed then probably you will be able to build deb packages via the script, no problem. But would I use those packages? Probably not, here's why (and it's opiniated) -
In order to have it run everywhere regardless of distribution and series right now massive statically linked binaries are being distributed. When creating debian packages for the same purpose, then equally large binaries will need to be included. One of my goals was to reduce the size and make use of version control inherent to distributions, making it possible to link against shared libraries. E.g. my nsqd binary targeted at Ubuntu 14.04 is 7.4 Mb whereas the static binary is 11 Mb, quite a significant reduction.
Another reason for me to do packaging is to actually manage them via a trusted repository for my projects, just like other core system and app packages I need, again making use of version control characteristics of apt (or rpm for that matter). An added plus of this approach is that you can digitally sign the packages, increasing trust.
So personally I would not be much a fan of an all purpose deb (much how elasticsearch does it like @kutchar suggested), but would rather look at e.g. how mongodb manages this with debian, ubuntu and rpm repositories and fully signed releases. Given the maturity of nsq as a project I somehow think this would be more appropriate?
@misja thanks for detailing the tradeoffs. It's unlikely that we're going to host our own artifact repositories anytime soon, it just feels like a lot of effort for little gain.
It would be nice if we helped users who wanted to install NSQ like this by providing some scripts/config/etc. to accomplish this in our contrib directory in this repo.
Would you be interested in contributing those so that other users could follow the same path?
If you are ever ready to set up official repositories, just let me know. You'll find the packaging config in my debian branch https://github.com/misja/nsq/tree/debian (debian/) which currently maps to the v0.3.6 tag, maybe this will be the easiest way for you to add, just by keeping a separate branch which follows tags?
