helm optimisations

[pjbgf]

Helm Optimisations

1. Optimise index loading using JSON

Use of JSON serialisation to improve subsequent loads from an index. For big indexes (e.g. bitnami) the subsequent loads were 10x faster, and consumed less than 1/3 of memory footprint.

Performance Stats

	Unmarshal from YAML 50 times (CURRENT)	Unmarshal YAML once then JSON 49 times (OptimizedIndex)	Delta
Total allocs	26144961872	3020506792	-23124455080 (good)
Total mallocs	371797081	19583171	-352213910 (good)
Total elapsed time	49.568572182s	5.344214857s	-44.224357325s (great!)

The test above was based on loading 50 times the bitnami chart repository index: https://charts.bitnami.com/bitnami/index.yaml.

2. Use of If-Modified-Since to avoid downloading chart index

:warning: approach may need to be changed as this depends on https://github.com/helm/helm/pull/10916 which may or may not be accepted upstream. :warning:

We currently skip HelmRepository reconciliation when target index hasn't changed. However, we still download the index. This uses If-Modified-Since to also skip the download of the chart repository index if it hasn't changed. Depending on the interval set for the repository, and the chart index file size, this could save a substantial amount of bandwidth. Meaning that users would be able to decrease their Helm Repository reconciliation intervals without having the network usage impact that would generally come with it.

[souleb]

Nice!

This made me think, can't we just store a json serialised index from the beginning? Do we specifically need to keep the yaml format?

[hiddeco]

This made me think, can't we just store a json serialised index from the beginning? Do we specifically need to keep the yaml format?

The idea of the source-controller is that the Source Artifacts can be consumed by others (like Weave GitOps, VMWare Tanzu, etc.). Given this, it is best to keep the original format.