self-hosted icon indicating copy to clipboard operation
self-hosted copied to clipboard

Upgrade Clickhouse to 23.8

Open hubertdeng123 opened this issue 9 months ago • 3 comments

Prod is being upgraded, so we should also do this for self-hosted users.

Looks like 21.8 -> 22.8 -> 23.3 -> 23.8

Although, perhaps it is possible to go from 22.8 -> 23.8 because self-hosted upgrades will always incur downtime.

https://docs.altinity.com/releasenotes/altinity-stable-release-notes/23.8/#major-changes-that-require-attention

We should also look to automate this process or make it easier for devs to verify clickhouse upgrades for self-hosted.

hubertdeng123 avatar Apr 26 '24 19:04 hubertdeng123

By coincidence I upgraded to 23.8.8.21.altinitystable

apt install clickhouse-server=22.8.20.12.altinitystable clickhouse-common-static=22.8.20.12.altinitystable
apt install clickhouse-server=23.8.8.21.altinitystable clickhouse-common-static=23.8.8.21.altinitystable

But https://github.com/getsentry/self-hosted/blob/f84bb3d8e8e5f185dfb952b220a071efcbc8f070/clickhouse/config.xml#L3-L6

Will break Clickhouse. You need to use (default value for example)

<max_server_memory_usage_to_ram_ratio>0.9</max_server_memory_usage_to_ram_ratio>

williamdes avatar Apr 29 '24 09:04 williamdes

@williamdes I was able to get past the error by removing the newlines 🤔

hubertdeng123 avatar May 01 '24 23:05 hubertdeng123

@williamdes I was able to get past the error by removing the newlines 🤔

Okay, well I did not want to fight more as it was a nightmare finding this block. Great that the env can be kept as is

williamdes avatar May 02 '24 19:05 williamdes

We run a self-hosted setup in which we cannot rely on the provided scripts for migrations, so we need to rewrite them for our configuration, and while migrating to 24.5.0 I was just confronted with this issue, it definitely didn't cross my mind that breaking the xml into multiple lines for readibility would kill clickhouse 😅

Perhaps a short comment on https://github.com/getsentry/self-hosted/blob/6032d980254bafc8858acd53ede53669747f9d00/clickhouse/config.xml#L3 referencing the upstream issue will help others in the future.

dlouzan avatar May 23 '24 09:05 dlouzan

@dlouzan I've added this in https://github.com/getsentry/self-hosted/pull/3085. Closing this issue as this has been completed

hubertdeng123 avatar May 28 '24 19:05 hubertdeng123