Introduce client-side micrometer, start with exposing watcher revision metrics

[jrhee17]

Related to #516

Motivation We often add logs to check if each watcher has called its notifiers successfully.

Using meters should improve this since:

1. We can more easily visualize the current revision state for each watcher (logs only provide whether the update occurred during a certain time frame)
2. Integration with our stack (prometheus/grafana) provides a better interface to check if any servers/watchers have failed to update.

Rather than migrating our 50+ watchers, I would like to propose centraldogma provides this functionality out of the box.

Modifications

• Allow *CentralDogmaBuilder to specify MeterRegistry
• Add MeterRegistry to constructors for *CentralDogma
• Add a Gauge for watcher revisions

Known Limitations

• Watchers with duplicate tags won't report metrics (metrics for a watcher with the same project/repository/path will be ignored)
  • I suppose later if needed we can introduce an identifier for each Watcher if someone needs this functionality. Hard to think of a scenario where multiple watchers of this nature will be needed though.
• *CentralDogma.meterRegistry and ClientFactory.meterRegistry might be different, which might cause confusion.
  • I guess later if needed *CentralDogma can use ClientFactory.meterRegistry by default if not specified. I think this can also be supported without any breaking changes in the future though.

[minwoox]

This looks good!

[codecov[bot]]

Codecov Report

Merging #542 (b92985d) into master (3799e35) will increase coverage by 0.05%. The diff coverage is 79.48%.

@@             Coverage Diff              @@
##             master     #542      +/-   ##
============================================
+ Coverage     70.22%   70.28%   +0.05%     
- Complexity     3354     3361       +7     
============================================
  Files           336      336              
  Lines         13359    13388      +29     
  Branches       1443     1447       +4     
============================================
+ Hits           9382     9410      +28     
- Misses         3084     3085       +1     
  Partials        893      893              

Impacted Files	Coverage Δ
...com/linecorp/centraldogma/client/CentralDogma.java	57.14% <0.00%> (-2.12%)	:arrow_down:
...ient/armeria/legacy/LegacyCentralDogmaBuilder.java	82.14% <50.00%> (-3.05%)	:arrow_down:
...gma/client/armeria/ArmeriaCentralDogmaBuilder.java	76.47% <50.00%> (-4.78%)	:arrow_down:
...ntraldogma/client/AbstractCentralDogmaBuilder.java	73.33% <50.00%> (-0.81%)	:arrow_down:
...corp/centraldogma/client/AbstractCentralDogma.java	68.57% <60.00%> (-2.40%)	:arrow_down:
.../centraldogma/internal/client/AbstractWatcher.java	80.98% <95.45%> (+1.67%)	:arrow_up:
...ogma/client/armeria/legacy/LegacyCentralDogma.java	82.74% <100.00%> (ø)
...ntraldogma/client/armeria/ArmeriaCentralDogma.java	72.35% <100.00%> (ø)
...nal/client/ReplicationLagTolerantCentralDogma.java	87.55% <100.00%> (ø)
...erver/internal/storage/PurgeSchedulingService.java	70.11% <0.00%> (-6.90%)	:arrow_down:
... and 9 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3799e35...b92985d. Read the comment docs.

[minwoox]

I guess later if needed *CentralDogma can use ClientFactory.meterRegistry by default if not specified.

How about implementing it in this PR?

*CentralDogma.meterRegistry and ClientFactory.meterRegistry might be different,

Shouldn't we warn in this case?

[jrhee17]

*CentralDogma.meterRegistry and ClientFactory.meterRegistry might be different, Shouldn't we warn in this case?

I tried this out, but this means for the user either:

1. Provide the same meterRegistry should be provided for both ClientFactory, CentralDogma
2. Or set meterRegistry for ClientFactory only relying on the fallback logic

I'm thinking whether simply removing the option to set meterRegistry in CentralDogma and relying on ClientFactory only may provide a more intuitive API

Let me know what you think

---

Update 2021/6/4

I realized AbstractCentralDogma is public, and we can't avoid users setting meterRegistry directory on *CentralDogma without introducing a breaking change.

I guess this boils down to

1. Do we introduce a slight breaking change at the cost of disallowing users from setting their own meterRegistry to *CentralDogma and rely on ClientFactory.meterRegistry
2. Do we just let users set meterRegistry which may be different from ClientFactory.meterRegistry, and just let users know via warning logs if the two are different

[jrhee17]

rebased once - gentle ping, as I still think this PR has value

[trustin]

I'm thinking whether simply removing the option to set meterRegistry in CentralDogma and relying on ClientFactory only may provide a more intuitive API

Let me know what you think

+1 for always using ClientFactory.meterRegistry(). How about adding disableMetrics() instead for those who want to opt out of metrics? (although not very likely)

[jrhee17]

Ready for the next round of reviews 🙏

[trustin]

What's the status of this PR since then? 😅