testcontainers-go icon indicating copy to clipboard operation
testcontainers-go copied to clipboard
verified publisher icon testcontainers

[Bug]: random clickhouse test failures

Open stevenh opened this issue 1 year ago • 2 comments

Testcontainers version

0.34.0

Using the latest Testcontainers version?

Yes

Host OS

N/A

Host arch

N/A

Go version

N/A

Docker version

N/A

Docker info

N/A

What happened?

Tests are randomly failing in CI with clickhouse_test.go:290: [handshake] unexpected packet [72] from server see: https://github.com/testcontainers/testcontainers-go/actions/runs/11478021546/job/31941936972?pr=2846

Relevant log output

=== FAIL: . TestClickHouseWithZookeeper (unknown) 2024/10/23 11:11:19 🐳 Creating container for image zookeeper:3.7 2024/10/23 11:11:19 ✅ Container created: 3205da1f43da 2024/10/23 11:11:19 🐳 Starting container: 3205da1f43da 2024/10/23 11:11:19 ✅ Container started: 3205da1f43da 2024/10/23 11:11:19 ⏳ Waiting for container id 3205da1f43da image: zookeeper:3.7. Waiting for: &{Port:2181/tcp timeout: PollInterval:100ms skipInternalCheck:false} 2024/10/23 11:11:19 🔔 Container is ready: 3205da1f43da 2024/10/23 11:11:19 🐳 Creating container for image clickhouse/clickhouse-server:23.3.8.21-alpine 2024/10/23 11:11:20 ✅ Container created: 2916bb606516 2024/10/23 11:11:20 🐳 Starting container: 2916bb606516 2024/10/23 11:11:20 ✅ Container started: 2916bb606516 2024/10/23 11:11:20 ⏳ Waiting for container id 2916bb606516 image: clickhouse/clickhouse-server:23.3.8.21-alpine. Waiting for: &{timeout: deadline: Strategies:[0xc00052ba70]} 2024/10/23 11:11:23 🔔 Container is ready: 2916bb606516 clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server clickhouse_test.go:290: [handshake] unexpected packet [72] from server coverage: 85.0% of statements ERROR rerun aborted because previous run had a suspected panic and some test may not have run make: *** [../../commons-test.mk:42: test-unit] Error 3

Additional information

No response

stevenh avatar Oct 23 '24 11:10 stevenh

Logs in clickhouse:


  2024.10.28 15:34:42.570353 [ 66 ] {} <Information> Application: It looks like the process has no CAP_SYS_NICE capability, the setting 'os_thread_priority' will have no effect. It could happen due to incorrect ClickHouse package installation. You could resolve the problem manually with 'sudo setcap cap_sys_nice=+ep /usr/bin/clickhouse'. Note that it will not work on 'nosuid' mounted filesystems.
  2024.10.28 15:34:42.733650 [ 66 ] {} <Warning> Application: Listen [0.0.0.0]:8123 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:8123 (version 23.3.8.21 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
  2024.10.28 15:34:42.733819 [ 66 ] {} <Warning> Application: Listen [0.0.0.0]:9000 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9000 (version 23.3.8.21 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
  2024.10.28 15:34:42.733984 [ 66 ] {} <Warning> Application: Listen [0.0.0.0]:9004 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9004 (version 23.3.8.21 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
  2024.10.28 15:34:42.734141 [ 66 ] {} <Warning> Application: Listen [0.0.0.0]:9005 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9005 (version 23.3.8.21 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
  2024.10.28 15:34:42.734348 [ 66 ] {} <Warning> Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, Net Exception: Address already in use: 0.0.0.0:9009 (version 23.3.8.21 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>

was able to repro it in CI

laskoviymishka avatar Oct 28 '24 14:10 laskoviymishka

Another incident of this here

FAIL TestClickHouseWithInitScripts (5.36s)
  2024/11/05 19:04:55 🐳 Creating container for image clickhouse/clickhouse-server:23.3.8.21-alpine
  2024/11/05 19:04:55 ✅ Container created: 88b56ce7d3a5
  2024/11/05 19:04:55 🐳 Starting container: 88b56ce7d3a5
  2024/11/05 19:04:55 ✅ Container started: 88b56ce7d3a5
  2024/11/05 19:04:55 ⏳ Waiting for container id 88b56ce7d3a5 image: clickhouse/clickhouse-server:23.3.8.21-alpine. Waiting for: &{timeout:<nil> deadline:<nil> Strategies:[0xc0003aa7e0]}
  2024/11/05 19:04:58 🔔 Container is ready: 88b56ce7d3a5
      clickhouse_test.go:155: 
          	Error Trace:	/home/runner/work/testcontainers-go/testcontainers-go/modules/clickhouse/clickhouse_test.go:155
          	Error:      	Received unexpected error:
          	            	[handshake] unexpected packet [72] from server
          	Test:       	TestClickHouseWithInitScripts
  2024/11/05 19:04:58 🐳 Stopping container: 88b56ce7d3a5
  2024/11/05 19:05:00 ✅ Container stopped: 88b56ce7d3a5
  2024/11/05 19:05:00 🐳 Terminating container: 88b56ce7d3a5
  2024/11/05 19:05:00 🚫 Container terminated: 88b56ce7d3a5

stevenh avatar Nov 06 '24 09:11 stevenh