[WIP] cf_exporter: bump exporter

[psycofdj]

This PR:

• bumps cf_exporter to latest release. See bosh-prometheus/cf_exporter#57.
• migrates to bpm
• read CF API url from bosh link and fallback to cf_exporter.cf.api_url property

Property changes

• cf_exporter.cf.api_v3_enabled: removed. Latest cf_exporter only use v3 API endpoints
• cf_exporter.events.query: removed. When events are enabled, latest cf_exporter always query events of the last 10 minutes with 5k elements per page
• cf_exporter.log_format: replaced by spec cf_exporter.log_stream and cf_exporter.log_json
• cf_exporter.log_level: defaults to error instead of info

Operator changes

• ops-file manifests/operators/enable-cf-api-v3.yml has been removed

Dependencies

This PR depends on:

• review and merge of bosh-prometheus/cf_exporter#57
• cutting release v1.0.0 in cf_exporter
• replace blob cf_exporter/cf_exporter-0.10.1.linux-amd64.tar.gz by cf_exporter/cf_exporter-1.0.0.linux-amd64.tar.gz

curl -fsSL https://github.com/bosh-prometheus/cf_exporter/releases/download/v1.0.0/cf_exporter-1.0.0.linux-amd64.tar.gz --output cf_exporter-1.0.0.linux-amd64.tar.gz
bosh add-blob cf_exporter-1.0.0.linux-amd64.tar.gz cf_exporter/cf_exporter-1.0.0.linux-amd64.tar.gz
bosh remove-blob cf_exporter/cf_exporter-0.10.1.linux-amd64.tar.gz

[benjaminguttmann-avtq]

I think removing api v2 compatibility removes compatibility for existing PCF installations. So I am wondering if it makes sense to use a similar approach like for prometheus2 and provide cf_exporter and cf_exporter2 so people can still use the old one?

[psycofdj]

I'm not very familiar with PCF, what makes you think that it will break compatibility ?

The capi-release started implementing v3 endpoints since 2017. I would be surprised that PCF has no v3 support at all. Any idea about which version of cf-deployment (or capi-release) is used in PCF so I can check that all endpoints used by cf_exporter are available in PCF ?

If we found that there is, indeed, an incompatibility, I would suggest to take this opportunity to create a new independent prometheus-cf-release as suggested in #431 rather than create a cf_exporter2 job and package. Of course I volunteer to create all related PRs and pumbling.

[benjaminguttmann-avtq]

I am still not exactly sure how such a prometheus-cf-release should look like :D not sure if is possible to get some kind of 'example' to get a better impression of it

[psycofdj]

I'm suggesting the same move that cloud foundry made when transitioning from cf-release to cf-deployment.

At start, we have a bosh-release with many many jobs and packages which brings many problems, among which the one your pointed out: as a user, I can't get the latest version of prometheus-boshrelease if I don't want to update to latest version of its package cf-exporter (unless the jobs has been kinda forked in a cf-exporter2)

This idea is to:

1. separate current jobs and packages to smaller bosh-releases

• not 1 package per release
• but may be 3 or 4 when is make sense and when they are tightly coupled

2. create a new bosh-prometheus/prometheus-deployment repository, containing all the yaml plumbing:

• the main manifest
• the versions of all smaller bosh releases
• the ops-files This repository is the main entry point for users as it also carry the release and changelog

In such a world, yaml files in prometheus-deployment could reference a given version of its child-release prometheus-cf-release but, as a user, I'll always be able to force a downgrade on this particular component with a simple obs-file like the follwing:

- type: replace
  path: /releases/name=prometheus-cf-release/version
  value: "the-version-i-want"

That's the big picture of what thing should look like in the end.

The transition between the current situation and this target can be very progressive. At start, the prometheus-deployment repository can only hold the manifest.yml and ops-files with all jobs referencing the current bosh release prometheus. Every time we extract a given set of jobs from the legacy release in a new bosh-release, prometheus-deployment only have to reference the new bosh-release name in its manifest.

To make things explicit (simple example of what the split could look like in the end):

• bosh-prometheus/prometheus-deployment
  • ./prometheus-manifest.yml
  • ./operations/monitor-cf.yml
  • ./operations/monitor-credhub.yml
  • ./operations/<all-other-ops-files....>
  • holds github release, changelog and reference versions
• bosh-prometheus/prometheus-core-release
  • ./{jobs,packages}/prometheus
  • ./{jobs,packages}/alertmanager
  • ./{jobs,packages}/grafana
  • ./{jobs,packages}/nginx
  • ./{jobs,packages}/probe-exporter
  • ./jobs/prometheus-alerts
  • ./jobs/prometheus-dashboards
  • ./jobs/grafana-dashboards
• bosh-prometheus/prometheus-bosh-release
  • ./{jobs,packages}/bosh-exporter
  • ./{jobs,packages}/bosh-tsdb-exporter
  • ./jobs/bosh-alerts
  • ./jobs/bosh-dashboards
• bosh-prometheus/prometheus-credhub-release
  • ./{jobs,packages}/credhub-exporter
  • ./jobs/credhub-alerts
  • ./jobs/credhub-dashboards
• bosh-prometheus/prometheus-cf-release
  • ./{jobs,packages}/cf-exporter
  • ./{jobs,packages}/firehose-exporter
  • ./jobs/cf-dashboards
  • ./jobs/cf-alerts
• and so on

[psycofdj]

closed in favor to #457