posthog
posthog copied to clipboard
PostHog
Clean install missing clickhouse migration `posthog.infi_clickhouse_orm_migrations_distributed`
Bug description
Hi, i am trying (playing with posthog) to clean install of posthog with clickhouse. Several installs were just fine, but now, out of nowhere, next clean install will throw an clickhouse migration error.
On clean docker container with clean data (postgre, ... everything)
Postgre migrations will apply just fine, but on python manage.py migrate_clickhouse it will throw
Traceback (most recent call last):
File "manage.py", line 21, in <module>
main()
File "manage.py", line 17, in main
2022-07-14 11:18:27.362 UTC [1] LOG C-0x55f355f9f160: posthog/[email protected]:48648 closing because: client close request (age=0s)
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/home/posthog/code/ee/management/commands/migrate_clickhouse.py", line 42, in handle
self.migrate(CLICKHOUSE_HTTP_URL, options)
File "/home/posthog/code/ee/management/commands/migrate_clickhouse.py", line 55, in migrate
migrations = list(self.get_migrations(database, options["upto"]))
File "/home/posthog/code/ee/management/commands/migrate_clickhouse.py", line 85, in get_migrations
applied_migrations = self.get_applied_migrations(database)
File "/home/posthog/code/ee/management/commands/migrate_clickhouse.py", line 95, in get_applied_migrations
return database._get_applied_migrations(MIGRATIONS_PACKAGE_NAME, replicated=CLICKHOUSE_REPLICATION)
File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 379, in _get_applied_migrations
return set(obj.module_name for obj in self.select(query))
File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 379, in <genexpr>
return set(obj.module_name for obj in self.select(query))
File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 276, in select
r = self._send(query, settings, True)
File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 389, in _send
raise ServerError(r.text)
infi.clickhouse_orm.database.ServerError: Code: 60. DB::Exception: Table posthog.infi_clickhouse_orm_migrations_distributed doesn't exist. (UNKNOWN_TABLE) (version 22.3.7.28 (official build))
I do now know, where this migration is from, it looks like its dynamically loaded from somewhere.
How to reproduce
- Clean install
- Run migrations
Environment
- [ ] PostHog Cloud
- [x] self-hosted PostHog (ClickHouse-based), version/commit:
1.37.1 - [ ] self-hosted PostHog (Postgres-based, legacy), version/commit: please provide
Oops, it looks like error not only for me :-) https://github.com/PostHog/posthog/issues/10789
Okey. Now after 2 hours it, again, out of nowhere started to work without any changes. These migrations are fetched from somewhere or?
Okey. Now after 2 hours it, again, out of nowhere started to work without any changes. These migrations are fetched from somewhere or?
same, posthog_worker_1 is stuck on restarting with same error
Stack trace from clickhouse error log
0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb371dfa in /usr/bin/clickhouse
1. DB::Exception::Exception<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) @ 0xb3de224 in /usr/bin/clickhouse
2. void std::__1::__optional_storage_base<DB::Exception, false>::__construct<int const&, char const (&) [23], std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(int const&, char const (&) [23], std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) @ 0x1553004f in /usr/bin/clickhouse
3. DB::DatabaseCatalog::getTableImpl(DB::StorageID const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Exception>*) const @ 0x1551e161 in /usr/bin/clickhouse
4. DB::DatabaseCatalog::getTable(DB::StorageID const&, std::__1::shared_ptr<DB::Context const>) const @ 0x1552638b in /usr/bin/clickhouse
5. DB::JoinedTables::getLeftTableStorage() @ 0x15b11b7c in /usr/bin/clickhouse
6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::unordered_map<DB::PreparedSetKey, std::__1::shared_ptr<DB::Set>, DB::PreparedSetKey::Hash, std::__1::equal_to<DB::PreparedSetKey>, std::__1::allocator<std::__1::pair<DB::PreparedSetKey const, std::__1::shared_ptr<DB::Set> > > >) @ 0x15a8a366 in /usr/bin/clickhouse
7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15a895d4 in /usr/bin/clickhouse
8. ? @ 0x15f41600 in /usr/bin/clickhouse
9. DB::ClusterProxy::SelectStreamFactory::createForShard(DB::Cluster::ShardInfo const&, std::__1::shared_ptr<DB::IAST> const&, DB::StorageID const&, std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<std::__1::unique_ptr<DB::QueryPlan, std::__1::default_delete<DB::QueryPlan> >, std::__1::allocator<std::__1::unique_ptr<DB::QueryPlan, std::__1::default_delete<DB::QueryPlan> > > >&, std::__1::vector<DB::ClusterProxy::IStreamFactory::Shard, std::__1::allocator<DB::ClusterProxy::IStreamFactory::Shard> >&, unsigned int) @ 0x15f3fd60 in /usr/bin/clickhouse
10. DB::ClusterProxy::executeQuery(DB::QueryPlan&, DB::Block const&, DB::QueryProcessingStage::Enum, DB::StorageID const&, std::__1::shared_ptr<DB::IAST> const&, DB::ClusterProxy::IStreamFactory&, Poco::Logger*, std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::ExpressionActions> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Cluster> const&) @ 0x15ef7e7d in /usr/bin/clickhouse
11. DB::StorageDistributed::read(DB::QueryPlan&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::shared_ptr<DB::StorageSnapshot const> const&, DB::SelectQueryInfo&, std::__1::shared_ptr<DB::Context const>, DB::QueryProcessingStage::Enum, unsigned long, unsigned int) @ 0x15f29301 in /usr/bin/clickhouse
12. DB::InterpreterSelectQuery::executeFetchColumns(DB::QueryProcessingStage::Enum, DB::QueryPlan&) @ 0x15a9d41c in /usr/bin/clickhouse
13. DB::InterpreterSelectQuery::executeImpl(DB::QueryPlan&, std::__1::optional<DB::Pipe>) @ 0x15a94aaf in /usr/bin/clickhouse
14. DB::InterpreterSelectQuery::buildQueryPlan(DB::QueryPlan&) @ 0x15a94430 in /usr/bin/clickhouse
15. DB::InterpreterSelectWithUnionQuery::buildQueryPlan(DB::QueryPlan&) @ 0x15ae7503 in /usr/bin/clickhouse
16. DB::InterpreterSelectWithUnionQuery::execute() @ 0x15ae8728 in /usr/bin/clickhouse
17. ? @ 0x15d2b18f in /usr/bin/clickhouse
18. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::__1::shared_ptr<DB::Context>, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>, std::__1::optional<DB::FormatSettings> const&) @ 0x15d2e3ea in /usr/bin/clickhouse
19. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::__1::optional<DB::CurrentThread::QueryScope>&) @ 0x16640747 in /usr/bin/clickhouse
20. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) @ 0x16644ef2 in /usr/bin/clickhouse
21. DB::HTTPServerConnection::run() @ 0x168cc43b in /usr/bin/clickhouse
22. Poco::Net::TCPServerConnection::start() @ 0x19b83c4f in /usr/bin/clickhouse
23. Poco::Net::TCPServerDispatcher::run() @ 0x19b860a1 in /usr/bin/clickhouse
24. Poco::PooledThread::run() @ 0x19d43069 in /usr/bin/clickhouse
25. Poco::ThreadImpl::runnableEntry(void*) @ 0x19d403c0 in /usr/bin/clickhouse
26. ? @ 0x7f837f42d609 in ?
27. __clone @ 0x7f837f352133 in ?
(version 22.3.7.28 (official build))
2022.07.14 17:03:34.354393 [ 255 ] {6b9c38f0-3135-4d14-81af-115a54c4825a} <Error> executeQuery: Code: 60. DB::Exception: Table posthog.infi_clickhouse_orm_migrations doesn't exist. (UNKNOWN_TABLE) (version 22.3.7.28 (official build)) (from 172.18.0.1:37568) (in query: SELECT DISTINCT module_name FROM `posthog`.`infi_clickhouse_orm_migrations_distributed` WHERE package_name = 'ee.clickhouse.migrations' FORMAT TabSeparatedWithNamesAndTypes)
Another weird error on clean install
Traceback (most recent call last):
File "manage.py", line 21, in <module>
main()
File "manage.py", line 17, in main
execute_from_command_line(sys.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
self.execute(*args, **cmd_options)
File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/home/posthog/code/ee/management/commands/migrate_clickhouse.py", line 42, in handle
self.migrate(CLICKHOUSE_HTTP_URL, options)
File "/home/posthog/code/ee/management/commands/migrate_clickhouse.py", line 80, in migrate
database.migrate(MIGRATIONS_PACKAGE_NAME, options["upto"], replicated=CLICKHOUSE_REPLICATION)
File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 346, in migrate
applied_migrations = self._get_applied_migrations_and_create_tables(migrations_package_name, replicated=replicated)
File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 366, in _get_applied_migrations_and_create_tables
self.create_table(MigrationHistoryReplicated)
File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 152, in create_table
self._send(model_class.create_table_sql(self))
File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 389, in _send
raise ServerError(r.text)
infi.clickhouse_orm.database.ServerError: Code: 253. DB::Exception: There was an error on [sklik-dev-clickhouse:9000]: Code: 253. DB::Exception: Replica /clickhouse/prod/tables/noshard/posthog/infi_clickhouse_orm_migrations/replicas/ch1-01 already exists. (REPLICA_IS_ALREADY_EXIST) (version 22.3.7.28 (official build)). (REPLICA_IS_ALREADY_EXIST) (version 22.3.7.28 (official build))
Looks like same problem occurs here https://github.com/PostHog/posthog/issues/10694
Thanks for the report! I've reproduced it on our side as well. Looking into it currently
Is there any workaround to get posthog running again? Currently worker and plugin server are in a crashloop :/
What worked for me was adding data volumes for Postgres and Clickhouse to docker-compose.hobby.yml:
db:
volumes:
- postgres-data:/var/lib/postgresql/data
clickhouse:
volumes:
...
- clickhouse-data:/var/lib/clickhouse
volumes:
...
postgres-data:
clickhouse-data:
Edit: these need to be added to the docker compose template so they are available when Posthog first launches
I'm having the same issue, adding a volume did not resolve the issue. Hopefully we can see a fix soon as I really wanna play around with this! 😄
+1 on this issue. I am running the legacy 1.30 Postgres instance, was hoping to get a Clickhouse instance up and running.
When running the hobby script, everything comes online perfectly, but the first time I do a docker-compose down / up or restart the host, I run into this issue.
I have also tried adding the volumes to docker-compose as described above, then deleting all volumes and rebooting the stack from scratch, but I'm still having this issue.
Full trace below:
worker_1 | Traceback (most recent call last):
worker_1 | File "manage.py", line 21, in <module>
worker_1 | main()
worker_1 | File "manage.py", line 17, in main
worker_1 | execute_from_command_line(sys.argv)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
worker_1 | utility.execute()
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
worker_1 | self.fetch_command(subcommand).run_from_argv(self.argv)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
worker_1 | self.execute(*args, **cmd_options)
worker_1 | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
worker_1 | output = self.handle(*args, **options)
worker_1 | File "/code/posthog/management/commands/migrate_clickhouse.py", line 42, in handle
worker_1 | self.migrate(CLICKHOUSE_HTTP_URL, options)
worker_1 | File "/code/posthog/management/commands/migrate_clickhouse.py", line 55, in migrate
worker_1 | migrations = list(self.get_migrations(database, options["upto"]))
worker_1 | File "/code/posthog/management/commands/migrate_clickhouse.py", line 85, in get_migrations
worker_1 | applied_migrations = self.get_applied_migrations(database)
worker_1 | File "/code/posthog/management/commands/migrate_clickhouse.py", line 95, in get_applied_migrations
worker_1 | return database._get_applied_migrations(MIGRATIONS_PACKAGE_NAME, replicated=CLICKHOUSE_REPLICATION)
worker_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 379, in _get_applied_migrations
worker_1 | return set(obj.module_name for obj in self.select(query))
worker_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 379, in <genexpr>
worker_1 | return set(obj.module_name for obj in self.select(query))
worker_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 276, in select
worker_1 | r = self._send(query, settings, True)
worker_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 389, in _send
worker_1 | raise ServerError(r.text)
worker_1 | infi.clickhouse_orm.database.ServerError: Code: 60. DB::Exception: Table posthog.infi_clickhouse_orm_migrations doesn't exist. (UNKNOWN_TABLE) (version 22.3.10.22 (official build))
worker_1 | (0)
root_worker_1 exited with code 1
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 362, in _get_applied_migrations_and_create_tables
web_1 | return self._get_applied_migrations(migrations_package_name, replicated)
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 379, in _get_applied_migrations
web_1 | return set(obj.module_name for obj in self.select(query))
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 379, in <genexpr>
web_1 | return set(obj.module_name for obj in self.select(query))
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 276, in select
web_1 | r = self._send(query, settings, True)
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 389, in _send
web_1 | raise ServerError(r.text)
web_1 | infi.clickhouse_orm.database.ServerError: Code: 60. DB::Exception: Table posthog.infi_clickhouse_orm_migrations doesn't exist. (UNKNOWN_TABLE) (version 22.3.10.22 (official build))
web_1 | (0)
web_1 |
web_1 | During handling of the above exception, another exception occurred:
web_1 |
web_1 | Traceback (most recent call last):
web_1 | File "manage.py", line 21, in <module>
web_1 | main()
web_1 | File "manage.py", line 17, in main
web_1 | execute_from_command_line(sys.argv)
web_1 | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
web_1 | utility.execute()
web_1 | File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
web_1 | self.fetch_command(subcommand).run_from_argv(self.argv)
web_1 | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
web_1 | self.execute(*args, **cmd_options)
web_1 | File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
web_1 | output = self.handle(*args, **options)
web_1 | File "/code/posthog/management/commands/migrate_clickhouse.py", line 42, in handle
web_1 | self.migrate(CLICKHOUSE_HTTP_URL, options)
web_1 | File "/code/posthog/management/commands/migrate_clickhouse.py", line 80, in migrate
web_1 | database.migrate(MIGRATIONS_PACKAGE_NAME, options["upto"], replicated=CLICKHOUSE_REPLICATION)
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 346, in migrate
web_1 | applied_migrations = self._get_applied_migrations_and_create_tables(migrations_package_name, replicated=replicated)
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 371, in _get_applied_migrations_and_create_tables
web_1 | return self._get_applied_migrations_and_create_tables(migrations_package_name, replicated, allow_missing_tables=False)
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 359, in _get_applied_migrations_and_create_tables
web_1 | return self._get_applied_migrations(migrations_package_name, replicated)
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 379, in _get_applied_migrations
web_1 | return set(obj.module_name for obj in self.select(query))
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 379, in <genexpr>
web_1 | return set(obj.module_name for obj in self.select(query))
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 276, in select
web_1 | r = self._send(query, settings, True)
web_1 | File "/usr/local/lib/python3.8/site-packages/infi/clickhouse_orm/database.py", line 389, in _send
web_1 | raise ServerError(r.text)
web_1 | infi.clickhouse_orm.database.ServerError: Code: 60. DB::Exception: Table posthog.infi_clickhouse_orm_migrations doesn't exist. (UNKNOWN_TABLE) (version 22.3.10.22 (official build))
web_1 | (0)
I've asked @simfish85 to be added to the list of customers to be notified when PH cloud launches in EU, but until then, I have to keep data here in the EU...
Thanks in advance!
Created pull request to address this ^^ https://github.com/PostHog/posthog/pull/11256