Tidy up building and installing integration package

[axw]

Motivation/summary

Tidy up the Makefile targets for building the APM integration package:

• Separate building integration packages from linting
• Introduce separate build directory for release/snapshot integration packages, so they don't clobber each other.
• Use the git commit timestamp in the snapshot package build tag, for reproducibility.
• Because building the snapshot/preview integration package no longer clobbers the release package, I have updated the CI job to only sign/publish preview package zips.

We now install the integration package in system tests by uploading directly to Kibana, rather than injecting into package-registry, which is undergoing changes. This approach also ensures the system tests have an up-to-date package, rather than requiring developers to first run make build-package and restart the package-registry service.

To keep a simple method of installing the local APM integration package in Kibana outside of the system tests, one can use systemtest/cmd/runapm with the new -init flag. This will just initialise Fleet, install the integration package, and exit.

Checklist

~- [ ] Update CHANGELOG.asciidoc~ ~- [ ] Update package changelog.yml (only if changes to apmpackage have been made)~ ~- [ ] Documentation has been updated~

How to test these changes

N/A

Related issues

None

[mergify[bot]]

This pull request does not have a backport label. Could you fix it @axw? 🙏 To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-7.x is the label to automatically backport to the 7.x branch.
• backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[mergify[bot]]

This pull request does not have a backport label. Could you fix it @axw? 🙏 To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-7.x is the label to automatically backport to the 7.x branch.
• backport-7./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[apmmachine]

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-08-12T00:39:59.744+0000

• Duration: 24 min 12 sec

Test stats :test_tube:

Test	Results
Failed	0
Passed	137
Skipped	1
Total	138

:robot: GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate and publish the docker images.

• /test windows : Build & tests on Windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[apmmachine]

:books: Go benchmark report

Diff with the main branch

name                                                            old time/op    new time/op    delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/decoder goos:linux goarch:amd64
CompressedRequestReader/deflate_content_encoding-12               12.7µs ±70%    17.0µs ± 1%  +33.47%  (p=0.032 n=5+5)
CompressedRequestReader/deflate_sniff-12                          14.8µs ±60%    17.4µs ± 1%  +17.95%  (p=0.016 n=5+5)
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
ModelIndexer/NoCompression-12                                     2.29µs ±75%    1.30µs ±27%  -43.23%  (p=0.016 n=5+5)
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
ReadEvents/json_codec/199_events-12                               1.34ms ± 0%    1.36ms ± 3%   +1.45%  (p=0.032 n=4+5)
ReadEvents/nop_codec/1000_events-12                                884µs ± 7%     992µs ± 7%  +12.28%  (p=0.008 n=5+5)

name                                                            old alloc/op   new alloc/op   delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/decoder goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64
WriteTransaction/json_codec_big_tx-12                             3.52kB ± 0%    3.52kB ± 0%   +0.03%  (p=0.016 n=5+4)
ReadEvents/json_codec/10_events-12                                75.5kB ± 0%    75.5kB ± 0%   +0.04%  (p=0.032 n=5+5)

name                                                            old allocs/op  new allocs/op  delta
pkg:github.com/elastic/apm-server/internal/agentcfg goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/decoder goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/model/modelindexer goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/processor/stream goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/internal/publish goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/spanmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/aggregation/txmetrics goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling goos:linux goarch:amd64
pkg:github.com/elastic/apm-server/x-pack/apm-server/sampling/eventstorage goos:linux goarch:amd64

report generated with https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

[axw]

/package

[axw]

/package

[axw]

/package

[axw]

/package

[axw]

/package