HorizontalScaling of StorageNodes

[maheshevizio]

Hi, I have setup a vicotriaLogs cluster and have 2 storage Nodes. Iam using AWS EBS st1 volumes. For the horizontalPodAutoscaling, I see that there is a metrics: [] line which is supposed to decide when the vlstorage should scale and for what metrics?

https://github.com/VictoriaMetrics/helm-charts/blob/8b7034b5f96315b2f5843a79d143a6c658df5141/charts/victoria-logs-cluster/values.yaml#L744

Questions:

1. Iam looking to scale the storagePods when the storage nears like 75% or CPU/memory maxes to a particular setting. Could you share some examples how that can be specified.

2. Also technically the overall storage would increase as an each pod is added right. Ex: 2 replicas with each 100GiB are present and if it scales by 1 and become 3 total, The storage will increase to 750GiB is my understanding. Could you please confirm ?

3. The retentionDiskSpaceUsage: Should it be set for the total space ie.. 700GiB or it will be 250GiB mapped to each PVC (Iam confused here as to whether the max of each Pvc of each storage node or the entire vlstorage Statefulset size which would be 250GiB * 3 will be set for retentionDiskSpaceusage

[tiny-pangolin]

Adding a new storage will increase the total storage capacity by the size of the PVC, so if you start with 2 storage nodes with 100GiB of storage and add a 3rd the total capacity of the cluster will increase to 300GiB. When the new node is added it will start out empty, and new logs will be evenly distributed among all 3 nodes, so the cluster will have uneven storage utilization for 1 retention period. If the node is removed from the cluster, then the data that node was responsible for will no longer be available to the cluster, unless it was redistributed to the remaining nodes before decommissioning the node. Finally, when a new storage node is added it will require the list of nodes used by vlinsert and vlselect to be updated which currently cannot be done dynamically.

Based on this information, I would avoid using an HPA with vlstorage especially based on CPU and memory utilization since an expensive query could trigger adding of a new storage node that would not be much help since it would not have any of the logs that the query is asking for.

The retentionDiskSpaceUsage: Should it be set for the total space ie.. 700GiB or it will be 250GiB mapped to each PVC (Iam confused here as to whether the max of each Pvc of each storage node or the entire vlstorage Statefulset size which would be 250GiB * 3 will be set for retentionDiskSpaceusage

Each storage node is independent and unaware of the other nodes in the cluster, so retentionDiskSpaceUsage should be based on the amount of storage available to that storage node.

[maheshevizio]

Thanks much

If I need to scale by storage metric, lets say if the storage of PVC maxes out can I use HPA for that ? What is the metric I need to refer to ?

[tiny-pangolin]

I think alerting on disk usage and scaling manually is the best approach since in order for autoscaling to work properly it would require a custom aggregate metric to scale, the HPA would have to have scaling down disabled, vlinsert, and vlselect would need to be restarted in order for an HPA with vlstorage to be effective so vlinsert and vlselect could be made aware of the new nodes every time the new storage nodes are added.

The metric that measures disk utilization is vl_free_disk_space_bytes, but you would need to measure the metric based on the available space in the entire cluster rather than just 1 node since the existing nodes will continue to ingest data. If you only did vl_free_disk_space_bytes from a single node instead of something like sum(vl_free_disk_space_bytes) then new instances would continue to be created over until the existing storage nodes started aging out data.