falco
falco copied to clipboard
falcosecurity
build: syncing the repo metadata even when an RPM or DEB package is deleted afterword
Motivation
When releasing a new version, a new git tag triggers the CI to build and publish packages. During the RPM and DEB publishing process, the new package is uploaded then repo metadata is updated. The following scripts implement the process:
- https://github.com/falcosecurity/falco/blob/master/scripts/publish-rpm
- https://github.com/falcosecurity/falco/blob/master/scripts/publish-deb
Basically, the scripts are intended to work only when a new package is available.
However, it might happen that a problem is noticed after the packages are being published, so overwrite or clean up packages can be necessary. Overwriting a package works out of the box (scripts use this strategy by default). On the other hand, there's no way just to remove a published package and update repo metadata consequently.
Feature
Scripts should permit us to sync the repo metadata even when an RPM or DEB package is deleted. This ability is likely needed when - in an emergency - a package or a whole release needs to be unpublished.
Furthermore, the infrastructure should provide for the possibility of running this task on demand.
Alternatives
Currently, the only way to sync the RPM/DEB repository metadata is by manually running a modified version of those scripts in a local environment. Nevertheless, I don't think it's a reliable solution.
Additional context
Those situations actually happened when we had to stop the 0.28.2 building process since we found a problem in packages. Later, we also decided to cancel the 0.28.2 in favor of 0.29.0, so packages needed to be deleted eventually. After manually deleting those packages from S3, I had to sync the RPM/DEB repositories manually. Below I copied the modified scripts that I used.
publish-deb
#!/usr/bin/env bash
set -e
usage() {
echo "usage: $0 -f <package.deb> -r <deb|deb-dev>"
exit 1
}
check_program() {
if ! command -v $1 &> /dev/null
then
echo "$1 is required and could not be found"
exit
fi
}
# Update the local DEB repository
#
# $1: path of the repository
# $2: suite (eg. "stable")
update_repo() {
# fixme(leogr): we cannot use apt-ftparchive --arch packages ...
# since our .deb files ends with "_x86_64" instead of "amd64".
# See https://manpages.debian.org/jessie/apt-utils/apt-ftparchive.1.en.html
#
# As a workaround, we temporarily stick here with "amd64"
# (the only supported arch at the moment)
local arch=amd64
local component=main
local debs_dir=$2
local release_dir=dists/$2
local packages_dir=${release_dir}/${component}/binary-${arch}
pushd $1 > /dev/null
# packages metadata
apt-ftparchive packages ${debs_dir} > ${packages_dir}/Packages
gzip -c ${packages_dir}/Packages > ${packages_dir}/Packages.gz
bzip2 -z -c ${packages_dir}/Packages > ${packages_dir}/Packages.bz2
# release metadata
apt-ftparchive release \
-o APT::FTPArchive::Release::Origin=Falco \
-o APT::FTPArchive::Release::Label=Falco \
-o APT::FTPArchive::Release::Suite=$2 \
-o APT::FTPArchive::Release::Codename=$2 \
-o APT::FTPArchive::Release::Components=${component} \
-o APT::FTPArchive::Release::Architectures=${arch} \
${release_dir} > ${release_dir}/Release
# release signature
gpg --detach-sign --armor ${release_dir}/Release
rm -f ${release_dir}/Release.gpg
mv ${release_dir}/Release.asc ${release_dir}/Release.gpg
popd > /dev/null
}
# parse options
while getopts ":f::r:" opt; do
case "${opt}" in
f )
file=${OPTARG}
;;
r )
repo="${OPTARG}"
[[ "${repo}" == "deb" || "${repo}" == "deb-dev" ]] || usage
;;
: )
echo "invalid option: ${OPTARG} requires an argument" 1>&2
exit 1
;;
\?)
echo "invalid option: ${OPTARG}" 1>&2
exit 1
;;
esac
done
shift $((OPTIND-1))
# check options
if [ -z "${repo}" ]; then
usage
fi
# check prerequisites
check_program apt-ftparchive
check_program gzip
check_program bzip2
check_program gpg
check_program aws
# settings
debSuite=stable
s3_bucket_repo="s3://falco-distribution/packages/${repo}"
cloudfront_path="/packages/${repo}"
tmp_repo_path=/tmp/falco-$repo
# prepare repository local copy
echo "Fetching ${s3_bucket_repo}..."
mkdir -p ${tmp_repo_path}
aws s3 cp ${s3_bucket_repo} ${tmp_repo_path} --recursive
update_repo ${tmp_repo_path} ${debSuite}
# publish
aws s3 sync ${tmp_repo_path}/dists ${s3_bucket_repo}/dists --delete --acl public-read
aws cloudfront create-invalidation --distribution-id ${AWS_CLOUDFRONT_DIST_ID} --paths ${cloudfront_path}/dists/*
publish-rpm
#!/usr/bin/env bash
set -e
usage() {
echo "usage: $0 -f <package.rpm> -r <rpm|rpm-dev>"
exit 1
}
check_program() {
if ! command -v $1 &> /dev/null
then
echo "$1 is required and could not be found"
exit
fi
}
# Update the local RPM repository
#
# $1: path of the repository.
update_repo() {
pushd $1 > /dev/null
createrepo --update --no-database .
rm -f repodata/repomd.xml.asc
gpg --detach-sign --armor repodata/repomd.xml
popd > /dev/null
}
# parse options
while getopts ":f::r:" opt; do
case "${opt}" in
f )
file=${OPTARG}
;;
r )
repo="${OPTARG}"
[[ "${repo}" == "rpm" || "${repo}" == "rpm-dev" ]] || usage
;;
: )
echo "invalid option: ${OPTARG} requires an argument" 1>&2
exit 1
;;
\?)
echo "invalid option: ${OPTARG}" 1>&2
exit 1
;;
esac
done
shift $((OPTIND-1))
if [ -z "${repo}" ]; then
usage
fi
# check prerequisites
check_program createrepo
check_program gpg
check_program aws
# settings
s3_bucket_repo="s3://falco-distribution/packages/${repo}"
cloudfront_path="/packages/${repo}"
tmp_repo_path=/tmp/falco-$repo
# prepare repository local copy
echo "Fetching ${s3_bucket_repo}..."
mkdir -p ${tmp_repo_path}
aws s3 cp ${s3_bucket_repo} ${tmp_repo_path} --recursive
update_repo ${tmp_repo_path}
# publish
aws s3 sync ${tmp_repo_path}/repodata ${s3_bucket_repo}/repodata --delete --acl public-read
aws cloudfront create-invalidation --distribution-id ${AWS_CLOUDFRONT_DIST_ID} --paths ${cloudfront_path}/repodata/*
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle rotten
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle rotten
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
@leogr: This request has been marked as needing help from a contributor.
Please ensure the request meets the requirements listed here.
If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.
In response to this:
/remove-lifecycle stale /help
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Unfortunately, this is still a thing.
/remove-lifecycle stale
/assign /assign @LucaGuerra
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
/remove-lifecycle stale
@LucaGuerra IIRC, this problem is partially mitigated now. But, it's still an issue in the rare case we need to remove a package (to unpublish it) manually. One possible solution would be adding a manually triggerable GH action when we need to sync.