Package distribution deb

[realbigsean]

Issue Addressed

#2082

Proposed Changes

I used this Polkadot pull request as a reference for most of this (not the rpm portion): https://github.com/paritytech/polkadot/pull/1676/files

• the tools to build a .deb package
• systemd service files for the beacon and validator
• used a combination of config found in that polkadot PR, as well as in lighthouse-ansible
• a postinst script that creates separate users for the bn and vc, configures the datadir
• managing cli flags in /etc/default files

Additional Info

TODOs:

• Distributing the .deb. I think we could self host it (parity does this), or use something like launchpad.net (geth does this)
• figure out how to help users transition to using this for updates
• figure how to make using the account manager minimally painful
• adding default cli flags in the /etc/default files, maybe adding common flags in comments

Would love some feedback! I'm learning most of this as I go, so anyone with expertise/experience in this area, please chime in.

[torfbolt]

Nice work!

• For distributing I would vote for launchpad, this has the advantage that it's very easy for a user to setup as an additional package source and automatically receive future updates. (basically just apt-add-repository ...)
• Can we activate this build process for both AMD64 and ARM64?
• Why do the users have home directories configured? As far as I can see they are not used as data directories.

[realbigsean]

@torfbolt thanks a lot for the feedback!

For distributing I would vote for launchpad, this has the advantage that it's very easy for a user to setup as an additional package source and automatically receive future updates. (basically just apt-add-repository ...)

I am also thinking launchpad is the better option. It would be a little less maintenance on Sigma Prime's end, I think it will make it easier for us to build for more architectures, and it handles PGP key verification, which is nice for users. Unfortunately I've learned uploading to launchpad will take a bit of reworking of this PR. Rather than cargo-deb which generates a .deb, we'll have to use something like debcargo in order to generate a .dsc file. Unfortunately debcargo either requires all of lighthouse's dependencies to be uploaded to crates.io or we have to be able to vendor lighthouse's dependencies. The latter would be easier to accomplish but is blocked on this PR: #1712. I'm going to start looking into that PR and going to mark this one as blocked for now.

Can we activate this build process for both AMD64 and ARM64?

I made an attempt to do this using cargo deb, but it's not quite working. I'm going to hold of on trying again until we make progress towards being able to upload to launchpad. But it's definitely a priority.

Why do the users have home directories configured? As far as I can see they are not used as data directories.

Thanks for pointing this out! This was a remnant of what I was looking at in Polkadot's CI, they seem to use the home directory to store data. But I've updated it now.

[jmcph4]

@realbigsean FWIW, your fix in 7a7197798 went in after the most recent CI job fell over, so I think this should be passing CI checks now?

Very keen to see .deb packaging for this project!

[realbigsean]

Hey @jmcph4,

I've left this as blocked because although we can now successfully vendor lighthouse dependencies using cargo vendor, this command doesn't work in our CI (https://github.com/sigp/lighthouse/pull/2076#issuecomment-784477635). But I saw in discord that you were interested in helping us publish all of our crates to crates.io that would be very helpful -- it's likely an easier route to unblocking this issue at this point.

[bachp]

@realbigsean Once https://github.com/sigp/lighthouse/pull/2076 is merged cargo vendorwill be tested in CI.

The PR also contains a workaround that makes cargo vendorwork on Github Actions by setting CARGO_HOME=$(readlink -f $HOME).

[realbigsean]

I do plan on picking this back up! but will need to think about how we want package distribution to work post-merge

[paulhauner]

will need to think about how we want package distribution to work post-merge

I think we'd still distribute as a stand-alone lighthouse package. Users would need to install lighthouse plus an execution client to get a "full node". Do you have thoughts otherwise?

[realbigsean]

I think we'd still distribute as a stand-alone lighthouse package. Users would need to install lighthouse plus an execution client to get a "full node". Do you have thoughts otherwise?

I'm not sure if it's possible but it'd be nice to do something like check if a compatible version of an execution client is installed. Generally though, yeah I think we should probably just keep the installation simple and leave it up to the user to make sure the execution client is properly configured