exportarr
exportarr copied to clipboard
onedr0p
[bazarr] Add bounded concurrency to episodes endpoint
Description of the change
This PR adds pagination to collection from the /api/episodes endpoint in the bazarr collector. This pagination uses bounded concurrency, sending no more than --series-batch-concurrency (default 10) batches to the bazarr server at a time, and asking for no more than --series-batch-size (default 50) seriesIds at a time.
fixes #237
A test image is available at ghcr.io/rtrox/exportarr:episodes-concurrency.
Benefits
Allows retrieval of series metrics from bazarr instances with a large number of serieses.
Possible drawbacks
More goroutines?
Applicable issues
- fixes #237
Additional information
@rtrox ill run this later tonight to get some benchmark numbers for comparision
@phyzical - that'd be great, I don't actually have a bazarr instance to test against >.< I'm reasonably confident from the unittests, but mind comparing the metric values too and just double checking that there aren't any logic bugs in the counting?
@phyzical - one more note -- in lidarr & sonarr we have a similar pattern, where a few specific metrics require us to send an API call per object. In both places, we hide these metrics behind the EnableAdditionalMetrics config value. I haven't made that change here yet, as it looks like I'd be gating the majority of the interesting metrics, but I'd like to think through what metrics are actually vital, and see if we can find ways to get them from the higher order APIs (or make some PRs to bazarr to do so).
i did have that if sitting there ready for heavy lifting, but as you've guessed it was kinda of all or nothing.
The other reason i didn't bother was that i thought i had a fairly large sample size (3k+ series) and i was able to get it all within 30 seconds, which was more than enough for my usecase (checking a dashboard once in a blue moon)
i personally think we should maybe be suggesting that these exporters are configured for longer intervals i.e once every 5-10 minutes anyway, that way if it is an expensive scrape it doesn't matter as much. the only outlier i can think of is alerts tied to disk space/health check, but even then id say finding out at 5-10 minutes for hobby stuff is still decent.
But ill give it a test still because if its does speed things up why not!
Yea, that makes sense. The only thing to keep in mind is that grafana's prometheus integration's max_source_resolution is somewhat low (I think it's a minute, but don't quote me on that). So if we decided to give that advice, we'll need to document the process of adding max_source_resolution as a query parameter to the grafana prometheus integration, given most of our users are hobbyists.
That said, breaking this up was probably necessary anyway, in the linked issue the url alone was near 100K, which means the memory needed to deserialize the response was definitely going to be a bit crazy >.<.
yeah i saw the size of the log in the 20 mb range 😆
TBH ive been grafanaing for about 4 months so good to know the grafana limitation.
But i think another part of the issue here is also that we assume each request should perform a new scrape, most of the exporters ive used tend to work with a scrape interval internally and cache the result for the prometheus instance to just get served some cold hard data.
Not saying that should be done in this pr. But something that should be considered if it hasnt already.
(i assume itll be a "oh its on the list" and there just wasnt enough justification)
Believe it or not, caching and scrape intervals are actually anti-patterns in Prometheus exporters 🙃 From the exporter instrumentation docs:
Metrics should only be pulled from the application when Prometheus scrapes them, exporters should not perform scrapes based on their own timers. That is, all scrapes should be synchronous.
Accordingly, you should not set timestamps on the metrics you expose, let Prometheus take care of that. If you think you need timestamps, then you probably need the Pushgateway instead.
If a metric is particularly expensive to retrieve, i.e. takes more than a minute, it is acceptable to cache it. This should be noted in the HELP string.
The default scrape timeout for Prometheus is 10 seconds. If your exporter can be expected to exceed this, you should explicitly call this out in your user documentation.
We have a couple places that we do cache - in prowlarr, for example, as prowlarr exposes stats in an odd way which requires us to calculate and cache deltas to create accurate counts. Otherwise, we generally avoid caching and stick with synchronous calls.
yeah that makes sense. in most cases just adjusting interval and scrape times would suffice (has for me).
And its probably like anything prometheus keep the rules simple and locked down and it should just work out the box.
The caching is it a simple toggleable feature or is it fairly deep? it does feel like an edgecase at this point i would personally consider my library the large end but this other users has double what i do so 😆 (no saying it shouldnt be a feature but should depend on how complicated it becomes to support optional caching)
i did actually try asyncing the calls originally too but found it made things slower and i think i'm seeing the same during testing?
old:
# HELP bazarr_episode_subtitles_downloaded_total Total number of downloaded episode subtitles
# TYPE bazarr_episode_subtitles_downloaded_total gauge
bazarr_episode_subtitles_downloaded_total{url="http://bazarr:6767"} 64927
# HELP bazarr_episode_subtitles_filesize_total Total filesize of all episode subtitles
# TYPE bazarr_episode_subtitles_filesize_total gauge
bazarr_episode_subtitles_filesize_total{url="http://bazarr:6767"} 3.256260812e+09
# HELP bazarr_episode_subtitles_history_total Total number of history episode subtitles
# TYPE bazarr_episode_subtitles_history_total gauge
bazarr_episode_subtitles_history_total{url="http://bazarr:6767"} 17952
# HELP bazarr_episode_subtitles_missing_total Total number of missing episode subtitles
# TYPE bazarr_episode_subtitles_missing_total gauge
bazarr_episode_subtitles_missing_total{url="http://bazarr:6767"} 18619
# HELP bazarr_episode_subtitles_monitored_total Total number of monitored episode subtitles
# TYPE bazarr_episode_subtitles_monitored_total gauge
bazarr_episode_subtitles_monitored_total{url="http://bazarr:6767"} 26877
# HELP bazarr_episode_subtitles_unmonitored_total Total number of unmonitored episode subtitles
# TYPE bazarr_episode_subtitles_unmonitored_total gauge
bazarr_episode_subtitles_unmonitored_total{url="http://bazarr:6767"} 61021
# HELP bazarr_episode_subtitles_wanted_total Total number of wanted episode subtitles
# TYPE bazarr_episode_subtitles_wanted_total gauge
bazarr_episode_subtitles_wanted_total{url="http://bazarr:6767"} 18618
# HELP bazarr_movie_subtitles_downloaded_total Total number of downloaded movie subtitles
# TYPE bazarr_movie_subtitles_downloaded_total gauge
bazarr_movie_subtitles_downloaded_total{url="http://bazarr:6767"} 1973
# HELP bazarr_movie_subtitles_filesize_total Total filesize of all movie subtitles
# TYPE bazarr_movie_subtitles_filesize_total gauge
bazarr_movie_subtitles_filesize_total{url="http://bazarr:6767"} 2.42571861e+08
# HELP bazarr_movie_subtitles_history_total Total number of history movie subtitles
# TYPE bazarr_movie_subtitles_history_total gauge
bazarr_movie_subtitles_history_total{url="http://bazarr:6767"} 61
# HELP bazarr_movie_subtitles_missing_total Total number of missing movie subtitles
# TYPE bazarr_movie_subtitles_missing_total gauge
bazarr_movie_subtitles_missing_total{url="http://bazarr:6767"} 166
# HELP bazarr_movie_subtitles_monitored_total Total number of monitored movie subtitles
# TYPE bazarr_movie_subtitles_monitored_total gauge
bazarr_movie_subtitles_monitored_total{url="http://bazarr:6767"} 2126
# HELP bazarr_movie_subtitles_unmonitored_total Total number of unmonitored movie subtitles
# TYPE bazarr_movie_subtitles_unmonitored_total gauge
bazarr_movie_subtitles_unmonitored_total{url="http://bazarr:6767"} 5
# HELP bazarr_movie_subtitles_wanted_total Total number of wanted movie subtitles
# TYPE bazarr_movie_subtitles_wanted_total gauge
bazarr_movie_subtitles_wanted_total{url="http://bazarr:6767"} 166
# HELP bazarr_scrape_duration_seconds Duration of the last scrape of metrics from Exportarr.
# TYPE bazarr_scrape_duration_seconds gauge
bazarr_scrape_duration_seconds{url="http://bazarr:6767"} 28.936000262
# HELP bazarr_scrape_requests_total Total number of HTTP requests made.
# TYPE bazarr_scrape_requests_total counter
bazarr_scrape_requests_total{code="200",url="http://bazarr:6767"} 10
# HELP bazarr_subtitles_downloaded_total Total number of downloaded subtitles
# TYPE bazarr_subtitles_downloaded_total gauge
bazarr_subtitles_downloaded_total{url="http://bazarr:6767"} 66900
# HELP bazarr_subtitles_filesize_total Total filesize of all subtitles
# TYPE bazarr_subtitles_filesize_total gauge
bazarr_subtitles_filesize_total{url="http://bazarr:6767"} 3.498832673e+09
# HELP bazarr_subtitles_history_total Total number of history subtitles
# TYPE bazarr_subtitles_history_total gauge
bazarr_subtitles_history_total{url="http://bazarr:6767"} 18013
# HELP bazarr_subtitles_language_total Total number of downloaded subtitles by language
# TYPE bazarr_subtitles_language_total gauge
bazarr_subtitles_language_total{language="Arabic",url="http://bazarr:6767"} 51
bazarr_subtitles_language_total{language="Danish",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Dutch",url="http://bazarr:6767"} 4
bazarr_subtitles_language_total{language="English",url="http://bazarr:6767"} 66628
bazarr_subtitles_language_total{language="Finnish",url="http://bazarr:6767"} 42
bazarr_subtitles_language_total{language="Hebrew",url="http://bazarr:6767"} 3
bazarr_subtitles_language_total{language="Italian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Norwegian",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Polish",url="http://bazarr:6767"} 21
bazarr_subtitles_language_total{language="Portuguese",url="http://bazarr:6767"} 24
bazarr_subtitles_language_total{language="Romanian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Swedish",url="http://bazarr:6767"} 43
# HELP bazarr_subtitles_missing_total Total number of missing subtitles
# TYPE bazarr_subtitles_missing_total gauge
bazarr_subtitles_missing_total{url="http://bazarr:6767"} 18785
# HELP bazarr_subtitles_monitored_total Total number of monitored subtitles
# TYPE bazarr_subtitles_monitored_total gauge
bazarr_subtitles_monitored_total{url="http://bazarr:6767"} 29003
# HELP bazarr_subtitles_provider_total Total number of downloaded subtitles by provider
# TYPE bazarr_subtitles_provider_total gauge
bazarr_subtitles_provider_total{provider="betaseries",url="http://bazarr:6767"} 1486
bazarr_subtitles_provider_total{provider="bsplayer",url="http://bazarr:6767"} 52
bazarr_subtitles_provider_total{provider="manual",url="http://bazarr:6767"} 24
bazarr_subtitles_provider_total{provider="opensubtitles",url="http://bazarr:6767"} 6169
bazarr_subtitles_provider_total{provider="opensubtitlescom",url="http://bazarr:6767"} 763
bazarr_subtitles_provider_total{provider="podnapisi",url="http://bazarr:6767"} 3069
bazarr_subtitles_provider_total{provider="subscene",url="http://bazarr:6767"} 2
bazarr_subtitles_provider_total{provider="supersubtitles",url="http://bazarr:6767"} 1773
bazarr_subtitles_provider_total{provider="tusubtitulo",url="http://bazarr:6767"} 11
bazarr_subtitles_provider_total{provider="tvsubtitles",url="http://bazarr:6767"} 27
bazarr_subtitles_provider_total{provider="yifysubtitles",url="http://bazarr:6767"} 18
# HELP bazarr_subtitles_score_total Total number of downloaded subtitles by score
# TYPE bazarr_subtitles_score_total gauge
bazarr_subtitles_score_total{score="100.0%",url="http://bazarr:6767"} 1298
bazarr_subtitles_score_total{score="66.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="66.94%",url="http://bazarr:6767"} 570
bazarr_subtitles_score_total{score="67.5%",url="http://bazarr:6767"} 58
bazarr_subtitles_score_total{score="67.78%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="68.06%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.89%",url="http://bazarr:6767"} 29
bazarr_subtitles_score_total{score="69.44%",url="http://bazarr:6767"} 8
bazarr_subtitles_score_total{score="69.72%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.28%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="71.11%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="75.0%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="75.83%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="77.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="78.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="79.17%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="80.0%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="80.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="81.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="82.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="83.33%",url="http://bazarr:6767"} 6
bazarr_subtitles_score_total{score="84.17%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="85.0%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="85.83%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="87.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="91.67%",url="http://bazarr:6767"} 115
bazarr_subtitles_score_total{score="91.94%",url="http://bazarr:6767"} 4723
bazarr_subtitles_score_total{score="92.22%",url="http://bazarr:6767"} 17
bazarr_subtitles_score_total{score="92.5%",url="http://bazarr:6767"} 991
bazarr_subtitles_score_total{score="92.78%",url="http://bazarr:6767"} 177
bazarr_subtitles_score_total{score="93.06%",url="http://bazarr:6767"} 84
bazarr_subtitles_score_total{score="93.33%",url="http://bazarr:6767"} 217
bazarr_subtitles_score_total{score="93.61%",url="http://bazarr:6767"} 27
bazarr_subtitles_score_total{score="93.89%",url="http://bazarr:6767"} 1382
bazarr_subtitles_score_total{score="94.17%",url="http://bazarr:6767"} 48
bazarr_subtitles_score_total{score="94.44%",url="http://bazarr:6767"} 1289
bazarr_subtitles_score_total{score="94.72%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="95.0%",url="http://bazarr:6767"} 401
bazarr_subtitles_score_total{score="95.28%",url="http://bazarr:6767"} 132
bazarr_subtitles_score_total{score="95.83%",url="http://bazarr:6767"} 185
bazarr_subtitles_score_total{score="96.11%",url="http://bazarr:6767"} 24
bazarr_subtitles_score_total{score="96.67%",url="http://bazarr:6767"} 20
bazarr_subtitles_score_total{score="97.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="98.06%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="98.61%",url="http://bazarr:6767"} 889
bazarr_subtitles_score_total{score="98.89%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="99.17%",url="http://bazarr:6767"} 360
bazarr_subtitles_score_total{score="99.44%",url="http://bazarr:6767"} 47
bazarr_subtitles_score_total{score="99.72%",url="http://bazarr:6767"} 57
# HELP bazarr_subtitles_unmonitored_total Total number of unmonitored subtitles
# TYPE bazarr_subtitles_unmonitored_total gauge
bazarr_subtitles_unmonitored_total{url="http://bazarr:6767"} 61026
# HELP bazarr_subtitles_wanted_total Total number of wanted subtitles
# TYPE bazarr_subtitles_wanted_total gauge
bazarr_subtitles_wanted_total{url="http://bazarr:6767"} 18784
# HELP bazarr_system_health_issues Total number of health issues by object and issue
# TYPE bazarr_system_health_issues gauge
bazarr_system_health_issues{issue="",object="",url="http://bazarr:6767"} 0
# HELP bazarr_system_status System Status
# TYPE bazarr_system_status gauge
bazarr_system_status{url="http://bazarr:6767"} 1
# HELP exportarr_app_info A metric with a constant '1' value labeled by app name, version, build time, and revision.
# TYPE exportarr_app_info gauge
exportarr_app_info{app_name="exportarr",build_time="2023-10-18T12:55:57.008Z",revision="294191ecae08120e407f1a8b2c5dda12ffb7b007",version="v1.6.0"} 1
new:
# HELP bazarr_episode_subtitles_downloaded_total Total number of downloaded episode subtitles
# TYPE bazarr_episode_subtitles_downloaded_total gauge
bazarr_episode_subtitles_downloaded_total{url="http://bazarr:6767"} 64927
# HELP bazarr_episode_subtitles_filesize_total Total filesize of all episode subtitles
# TYPE bazarr_episode_subtitles_filesize_total gauge
bazarr_episode_subtitles_filesize_total{url="http://bazarr:6767"} 3.256260812e+09
# HELP bazarr_episode_subtitles_history_total Total number of history episode subtitles
# TYPE bazarr_episode_subtitles_history_total gauge
bazarr_episode_subtitles_history_total{url="http://bazarr:6767"} 17952
# HELP bazarr_episode_subtitles_missing_total Total number of missing episode subtitles
# TYPE bazarr_episode_subtitles_missing_total gauge
bazarr_episode_subtitles_missing_total{url="http://bazarr:6767"} 18619
# HELP bazarr_episode_subtitles_monitored_total Total number of monitored episode subtitles
# TYPE bazarr_episode_subtitles_monitored_total gauge
bazarr_episode_subtitles_monitored_total{url="http://bazarr:6767"} 26877
# HELP bazarr_episode_subtitles_unmonitored_total Total number of unmonitored episode subtitles
# TYPE bazarr_episode_subtitles_unmonitored_total gauge
bazarr_episode_subtitles_unmonitored_total{url="http://bazarr:6767"} 61021
# HELP bazarr_episode_subtitles_wanted_total Total number of wanted episode subtitles
# TYPE bazarr_episode_subtitles_wanted_total gauge
bazarr_episode_subtitles_wanted_total{url="http://bazarr:6767"} 18618
# HELP bazarr_movie_subtitles_downloaded_total Total number of downloaded movie subtitles
# TYPE bazarr_movie_subtitles_downloaded_total gauge
bazarr_movie_subtitles_downloaded_total{url="http://bazarr:6767"} 1973
# HELP bazarr_movie_subtitles_filesize_total Total filesize of all movie subtitles
# TYPE bazarr_movie_subtitles_filesize_total gauge
bazarr_movie_subtitles_filesize_total{url="http://bazarr:6767"} 2.42571861e+08
# HELP bazarr_movie_subtitles_history_total Total number of history movie subtitles
# TYPE bazarr_movie_subtitles_history_total gauge
bazarr_movie_subtitles_history_total{url="http://bazarr:6767"} 61
# HELP bazarr_movie_subtitles_missing_total Total number of missing movie subtitles
# TYPE bazarr_movie_subtitles_missing_total gauge
bazarr_movie_subtitles_missing_total{url="http://bazarr:6767"} 166
# HELP bazarr_movie_subtitles_monitored_total Total number of monitored movie subtitles
# TYPE bazarr_movie_subtitles_monitored_total gauge
bazarr_movie_subtitles_monitored_total{url="http://bazarr:6767"} 2126
# HELP bazarr_movie_subtitles_unmonitored_total Total number of unmonitored movie subtitles
# TYPE bazarr_movie_subtitles_unmonitored_total gauge
bazarr_movie_subtitles_unmonitored_total{url="http://bazarr:6767"} 5
# HELP bazarr_movie_subtitles_wanted_total Total number of wanted movie subtitles
# TYPE bazarr_movie_subtitles_wanted_total gauge
bazarr_movie_subtitles_wanted_total{url="http://bazarr:6767"} 166
# HELP bazarr_scrape_duration_seconds Duration of the last scrape of metrics from Exportarr.
# TYPE bazarr_scrape_duration_seconds gauge
bazarr_scrape_duration_seconds{url="http://bazarr:6767"} 25.886018311
# HELP bazarr_scrape_requests_total Total number of HTTP requests made.
# TYPE bazarr_scrape_requests_total counter
bazarr_scrape_requests_total{code="200",url="http://bazarr:6767"} 6
# HELP bazarr_subtitles_downloaded_total Total number of downloaded subtitles
# TYPE bazarr_subtitles_downloaded_total gauge
bazarr_subtitles_downloaded_total{url="http://bazarr:6767"} 66900
# HELP bazarr_subtitles_filesize_total Total filesize of all subtitles
# TYPE bazarr_subtitles_filesize_total gauge
bazarr_subtitles_filesize_total{url="http://bazarr:6767"} 3.498832673e+09
# HELP bazarr_subtitles_history_total Total number of history subtitles
# TYPE bazarr_subtitles_history_total gauge
bazarr_subtitles_history_total{url="http://bazarr:6767"} 18013
# HELP bazarr_subtitles_language_total Total number of downloaded subtitles by language
# TYPE bazarr_subtitles_language_total gauge
bazarr_subtitles_language_total{language="Arabic",url="http://bazarr:6767"} 51
bazarr_subtitles_language_total{language="Danish",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Dutch",url="http://bazarr:6767"} 4
bazarr_subtitles_language_total{language="English",url="http://bazarr:6767"} 66628
bazarr_subtitles_language_total{language="Finnish",url="http://bazarr:6767"} 42
bazarr_subtitles_language_total{language="Hebrew",url="http://bazarr:6767"} 3
bazarr_subtitles_language_total{language="Italian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Norwegian",url="http://bazarr:6767"} 41
bazarr_subtitles_language_total{language="Polish",url="http://bazarr:6767"} 21
bazarr_subtitles_language_total{language="Portuguese",url="http://bazarr:6767"} 24
bazarr_subtitles_language_total{language="Romanian",url="http://bazarr:6767"} 1
bazarr_subtitles_language_total{language="Swedish",url="http://bazarr:6767"} 43
# HELP bazarr_subtitles_missing_total Total number of missing subtitles
# TYPE bazarr_subtitles_missing_total gauge
bazarr_subtitles_missing_total{url="http://bazarr:6767"} 18785
# HELP bazarr_subtitles_monitored_total Total number of monitored subtitles
# TYPE bazarr_subtitles_monitored_total gauge
bazarr_subtitles_monitored_total{url="http://bazarr:6767"} 29003
# HELP bazarr_subtitles_provider_total Total number of downloaded subtitles by provider
# TYPE bazarr_subtitles_provider_total gauge
bazarr_subtitles_provider_total{provider="betaseries",url="http://bazarr:6767"} 1486
bazarr_subtitles_provider_total{provider="bsplayer",url="http://bazarr:6767"} 52
bazarr_subtitles_provider_total{provider="manual",url="http://bazarr:6767"} 24
bazarr_subtitles_provider_total{provider="opensubtitles",url="http://bazarr:6767"} 6169
bazarr_subtitles_provider_total{provider="opensubtitlescom",url="http://bazarr:6767"} 763
bazarr_subtitles_provider_total{provider="podnapisi",url="http://bazarr:6767"} 3069
bazarr_subtitles_provider_total{provider="subscene",url="http://bazarr:6767"} 2
bazarr_subtitles_provider_total{provider="supersubtitles",url="http://bazarr:6767"} 1773
bazarr_subtitles_provider_total{provider="tusubtitulo",url="http://bazarr:6767"} 11
bazarr_subtitles_provider_total{provider="tvsubtitles",url="http://bazarr:6767"} 27
bazarr_subtitles_provider_total{provider="yifysubtitles",url="http://bazarr:6767"} 18
# HELP bazarr_subtitles_score_total Total number of downloaded subtitles by score
# TYPE bazarr_subtitles_score_total gauge
bazarr_subtitles_score_total{score="100.0%",url="http://bazarr:6767"} 1298
bazarr_subtitles_score_total{score="66.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="66.94%",url="http://bazarr:6767"} 570
bazarr_subtitles_score_total{score="67.5%",url="http://bazarr:6767"} 58
bazarr_subtitles_score_total{score="67.78%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="68.06%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="68.89%",url="http://bazarr:6767"} 29
bazarr_subtitles_score_total{score="69.44%",url="http://bazarr:6767"} 8
bazarr_subtitles_score_total{score="69.72%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.28%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="70.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="71.11%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="75.0%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="75.83%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="77.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="78.33%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="79.17%",url="http://bazarr:6767"} 5
bazarr_subtitles_score_total{score="80.0%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="80.83%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="81.67%",url="http://bazarr:6767"} 9
bazarr_subtitles_score_total{score="82.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="83.33%",url="http://bazarr:6767"} 6
bazarr_subtitles_score_total{score="84.17%",url="http://bazarr:6767"} 1
bazarr_subtitles_score_total{score="85.0%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="85.83%",url="http://bazarr:6767"} 7
bazarr_subtitles_score_total{score="87.5%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="91.67%",url="http://bazarr:6767"} 115
bazarr_subtitles_score_total{score="91.94%",url="http://bazarr:6767"} 4723
bazarr_subtitles_score_total{score="92.22%",url="http://bazarr:6767"} 17
bazarr_subtitles_score_total{score="92.5%",url="http://bazarr:6767"} 991
bazarr_subtitles_score_total{score="92.78%",url="http://bazarr:6767"} 177
bazarr_subtitles_score_total{score="93.06%",url="http://bazarr:6767"} 84
bazarr_subtitles_score_total{score="93.33%",url="http://bazarr:6767"} 217
bazarr_subtitles_score_total{score="93.61%",url="http://bazarr:6767"} 27
bazarr_subtitles_score_total{score="93.89%",url="http://bazarr:6767"} 1382
bazarr_subtitles_score_total{score="94.17%",url="http://bazarr:6767"} 48
bazarr_subtitles_score_total{score="94.44%",url="http://bazarr:6767"} 1289
bazarr_subtitles_score_total{score="94.72%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="95.0%",url="http://bazarr:6767"} 401
bazarr_subtitles_score_total{score="95.28%",url="http://bazarr:6767"} 132
bazarr_subtitles_score_total{score="95.83%",url="http://bazarr:6767"} 185
bazarr_subtitles_score_total{score="96.11%",url="http://bazarr:6767"} 24
bazarr_subtitles_score_total{score="96.67%",url="http://bazarr:6767"} 20
bazarr_subtitles_score_total{score="97.5%",url="http://bazarr:6767"} 2
bazarr_subtitles_score_total{score="98.06%",url="http://bazarr:6767"} 77
bazarr_subtitles_score_total{score="98.61%",url="http://bazarr:6767"} 889
bazarr_subtitles_score_total{score="98.89%",url="http://bazarr:6767"} 4
bazarr_subtitles_score_total{score="99.17%",url="http://bazarr:6767"} 360
bazarr_subtitles_score_total{score="99.44%",url="http://bazarr:6767"} 47
bazarr_subtitles_score_total{score="99.72%",url="http://bazarr:6767"} 57
# HELP bazarr_subtitles_unmonitored_total Total number of unmonitored subtitles
# TYPE bazarr_subtitles_unmonitored_total gauge
bazarr_subtitles_unmonitored_total{url="http://bazarr:6767"} 61026
# HELP bazarr_subtitles_wanted_total Total number of wanted subtitles
# TYPE bazarr_subtitles_wanted_total gauge
bazarr_subtitles_wanted_total{url="http://bazarr:6767"} 18784
# HELP bazarr_system_health_issues Total number of health issues by object and issue
# TYPE bazarr_system_health_issues gauge
bazarr_system_health_issues{issue="",object="",url="http://bazarr:6767"} 0
# HELP bazarr_system_status System Status
# TYPE bazarr_system_status gauge
bazarr_system_status{url="http://bazarr:6767"} 1
# HELP exportarr_app_info A metric with a constant '1' value labeled by app name, version, build time, and revision.
# TYPE exportarr_app_info gauge
exportarr_app_info{app_name="exportarr",build_time="",revision="",version="development"} 1
so yeah its actually slower, which is something i noticed when i tried batching it originally, granted i didn't try hard to workout why
i did my best to be fair too. 1 req at a time randomly switching between who was next and no prometheus running causing additional random load.
Logic wise its the same though
The caching is it a simple toggleable feature or is it fairly deep? it does feel like an edgecase at this point i would personally consider my library the large end but this other users has double what i do so 😆 (no saying it shouldnt be a feature but should depend on how complicated it becomes to support optional caching)
It's tied into the prowlarr logic, but it's also not terribly complicated. You can see the indexerStatCache here. That said, I only did caching there due to the shape of the data returned by prowlarr (their API accepts a time range, and gives aggregated stats by day. So we have to cache last seen values to calculate differentials).
so yeah its actually slower, which is something i noticed when i tried batching it originally, granted i didn't try hard to workout why
It's probably just the sheer number of calls. I set the default concurrency limit to 10 inflight requests. You can test bumping that up to 20 or 30 by setting --series-batch-concurrency. You can also bump up the batch size with --series-batch-size (default 50). If bumping either of those up makes a difference I'll just change the defaults. I should also add a collector stat for the number of calls being sent as well.
If it's still not faster, I think I'll still merge it, it looks like it's about a ~10% regression, which is likely worth it to reduce the size of the API requests. But if we can't actually get faster, I'll likely put the iterative call behind EnableAdditionalMetrics so folks can decide whether they want to see less metrics, or have a higher poll rate.
(@onedr0p feel free to veto me on that)
But if we can't actually get faster, I'll likely put the iterative call behind EnableAdditionalMetrics so folks can decide whether they want to see less metrics, or have a higher poll rate.
Sounds good to me :)
@rtrox ill give a few more tests around --series-batch-concurrency tonight after work 👍
But i think leveraging "EnableAdditionalMetrics" for sync v async i think is fair, though may be a bit misleading
But i think if it does actually allow users to use the implementation i think its a good idea.
@phyzical
But i think leveraging "EnableAdditionalMetrics" for sync v async i think is fair, though may be a bit misleading
That's not quite what I mean -- either way I'll want to switch to the threaded requests here, I'm not super comfortable leaving the episode history calls in place as is, the URLs are too large for me to be comfortable with on the wire, and the memory required for the responses is concerning. So I'm actually ok with a wall clock regression to move to the threaded model where request size is bounded.
What I mean is -- if we can't find defaults that let us reduce the wall clock time, I plan to put that call and the metrics it generates behind EnableAdditionalMetrics, I've been looking through the API code in bazarr, it looks like we can get a pretty reasonable number of metrics directly from the /series endpoint (episode files downloaded, episode files missing, series monitored.
So let's see if we can find some defaults that boost wall clock time, otherwise I'll start some refactoring to pull more metrics from the series api, and put the episode stats calls behind a flag (I'll likely do the refactoring either way, as any metric we can generate from the pre-aggregated series data is going to be much more performant. There are some other changes, like #239 that I'll want to make anyway).
Ah yep, sorry i get you now. i think your safe to just chuck both the history blocks behind the flag, i believe i didn't just cause i was focusing on timings and not ram usage.
I don't think you can get the data found in history in the main api call (which is why i started using it) but i could be wrong.
ah sorry, life then work got in the way 😆
batch concurrency = 10 is what was being used above
~40mb
~24s
batch concurrency = 20
~42mb
~24-25s
batch concurrency = 30
~45-50mb
~24-25s
batch concurrency = 40
~50mb
~33s
batch concurrency = 50
~50mb
~33-34
its odd that things get slower the more concurrency i wonder if its more a limitation bazarr side? as in how well the python code can deal with concurrency vs a giant payload.
ill do a few more tests playing with the batch size now for some more numbers
batch-size 50 is the first one above
batch-size 100
~50 mb
~28s
batch-size 150
~50mb
~27s
batch-size 200
~28s
~50mb
batch-size 300
~21s
~70-100mb
batch-size 500
~21s
~100mb
So almost all the attempts floated around the 50mb range until i increased the batch size which makes sense.
But i think the "recommendation" should be increase batch to reduce ram usage at the cost of longer requests, increase batch size for faster responses at the cost of more ram usage.
personally i think ill just configure it to, have a batch size of 300-500 and use the default concurrency.
@rtrox feel free to list some combos of vars you want the metrics of if it helps at all, and i will do some more tests
Thanks @phyzical, great stuff! Yea, I'd suspect the same. I'd guess either Bazarr, SQLite, or query performance as the issue, but this is probably good enough, 20-30 seconds at your library size seems reasonably solid.
So to land this, I think two updates:
- I'll update the defaults to 300 batch size.
- I'll put the history calls behind
EnableAdditionalMetrics - [stretch] I may add some
serieslevel metrics, to ensure we can get somemissing/monitoreddata for sonarr even withoutEnableAdditionalMetrics.
I should be be able to get to that over the long weekend so we can land this early next week.
Sorry, took a bit to get back to this PR, life got hectic >.< I think this should be ready for final review+merge.
ah sorry i forgot then the holidays happened, i will give this a look this week
was able to do a proper benchmark comparison diff wise for information it is exactly the same with the EnableAdditionalMetrics flag
👍