etcd won't run on OpenBSD: set keepalive period not supported

[tangruize]

What happened?

I tried to build and run etcd on OpenBSD, but it failed with the message: SetKeepAlivePeriod failed, set tcp 127.0.0.1:2379->127.0.0.1:25154: protocol not available" I found out that OpenBSD only allows global sysctl settings to set the keepalive intervals.

Here is a link to a community discussion about this issue: https://community.mongoose-os.com/t/openbsd-does-not-support-tcp-keepalive/1903.

What did you expect to happen?

Instead of aborting running in etcd, it would be better to log a warning message informing the user to set the global keepalive intervals manually if setting the keepalive intervals fails on OpenBSD.

I was able to get etcd to work by commenting out kac.SetKeepAlivePeriod().

diff --git a/client/pkg/transport/keepalive_listener.go b/client/pkg/transport/keepalive_listener.go
index 07b2dc136..988bb7ed1 100644
--- a/client/pkg/transport/keepalive_listener.go
+++ b/client/pkg/transport/keepalive_listener.go
@@ -65,9 +65,9 @@ func (kln *keepaliveListener) Accept() (net.Conn, error) {
 	if err := kac.SetKeepAlive(true); err != nil {
 		return nil, fmt.Errorf("SetKeepAlive failed, %w", err)
 	}
-	if err := kac.SetKeepAlivePeriod(30 * time.Second); err != nil {
-		return nil, fmt.Errorf("SetKeepAlivePeriod failed, %w", err)
-	}
+//	if err := kac.SetKeepAlivePeriod(30 * time.Second); err != nil {
+//		return nil, fmt.Errorf("SetKeepAlivePeriod failed, %w", err)
+//	}
 	return kac, nil
 }

How can we reproduce it (as minimally and precisely as possible)?

• Clone the project on OpenBSD
• make build
• bin/etcd

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.6.0-alpha.0
Git SHA: 0deaec0e4
Go Version: go1.20.1
Go OS/Arch: openbsd/amd64

$ etcdctl version
etcdctl version: 3.6.0-alpha.0
API version: 3.6

Etcd configuration (command line flags or environment variables)

no special config

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

No response

Relevant log output

{"level":"warn","ts":"2023-05-02T12:15:17.786466+0800","caller":"embed/config.go:705","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2023-05-02T12:15:17.787157+0800","caller":"embed/config.go:806","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"info","ts":"2023-05-02T12:15:17.787188+0800","caller":"etcdmain/etcd.go:64","msg":"Running: ","args":["bin/etcd"]}
{"level":"info","ts":"2023-05-02T12:15:17.787221+0800","caller":"etcdmain/etcd.go:91","msg":"failed to detect default host","error":"default host not supported on openbsd_amd64"}
{"level":"warn","ts":"2023-05-02T12:15:17.787254+0800","caller":"etcdmain/etcd.go:96","msg":"'data-dir' was empty; using default","data-dir":"default.etcd"}
{"level":"info","ts":"2023-05-02T12:15:17.787515+0800","caller":"etcdmain/etcd.go:107","msg":"server has already been initialized","data-dir":"default.etcd","dir-type":"member"}
{"level":"warn","ts":"2023-05-02T12:15:17.787559+0800","caller":"embed/config.go:705","msg":"Running http and grpc server on single port. This is not recommended for production."}
{"level":"warn","ts":"2023-05-02T12:15:17.787576+0800","caller":"embed/config.go:806","msg":"it isn't recommended to use default name, please set a value for --name. Note that etcd might run into issue when multiple members have the same default name","name":"default"}
{"level":"info","ts":"2023-05-02T12:15:17.7876+0800","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}
{"level":"info","ts":"2023-05-02T12:15:17.788402+0800","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["http://localhost:2379"]}
{"level":"info","ts":"2023-05-02T12:15:17.788727+0800","caller":"embed/etcd.go:312","msg":"starting an etcd server","etcd-version":"3.6.0-alpha.0","git-sha":"0deaec0e4","go-version":"go1.20.1","go-os":"openbsd","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"default","data-dir":"default.etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"default.etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","wait-cluster-ready-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","discovery-token":"","discovery-endpoints":"","discovery-dial-timeout":"2s","discovery-request-timeout":"5s","discovery-keepalive-time":"2s","discovery-keepalive-timeout":"6s","discovery-insecure-transport":true,"discovery-insecure-skip-tls-verify":false,"discovery-cert":"","discovery-key":"","discovery-cacert":"","discovery-user":"","downgrade-check-interval":"5s","max-learners":1}
{"level":"info","ts":"2023-05-02T12:15:17.798337+0800","caller":"storage/backend.go:81","msg":"opened backend db","path":"default.etcd/member/snap/db","took":"263.626µs"}
{"level":"info","ts":"2023-05-02T12:15:17.798627+0800","caller":"etcdserver/bootstrap.go:440","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2023-05-02T12:15:17.801288+0800","caller":"etcdserver/bootstrap.go:579","msg":"restarting local member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","commit-index":33}
{"level":"info","ts":"2023-05-02T12:15:17.802476+0800","caller":"api/capability.go:77","msg":"enabled capabilities for version","cluster-version":"3.6"}
{"level":"info","ts":"2023-05-02T12:15:17.802543+0800","caller":"membership/cluster.go:288","msg":"recovered/added member from store","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","recovered-remote-peer-id":"8e9e05c52164694d","recovered-remote-peer-urls":["http://localhost:2380"],"recovered-remote-peer-is-learner":false}
{"level":"info","ts":"2023-05-02T12:15:17.802587+0800","caller":"membership/cluster.go:298","msg":"set cluster version from store","cluster-version":"3.6"}
{"level":"info","ts":"2023-05-02T12:15:17.80273+0800","logger":"raft","caller":"[email protected]/raft.go:1697","msg":"8e9e05c52164694d switched to configuration voters=()"}
{"level":"info","ts":"2023-05-02T12:15:17.802985+0800","logger":"raft","caller":"[email protected]/raft.go:728","msg":"8e9e05c52164694d became follower at term 20"}
{"level":"info","ts":"2023-05-02T12:15:17.803015+0800","logger":"raft","caller":"[email protected]/raft.go:382","msg":"newRaft 8e9e05c52164694d [peers: [], term: 20, commit: 33, applied: 0, lastindex: 40, lastterm: 20]"}
{"level":"warn","ts":"2023-05-02T12:15:17.803901+0800","caller":"auth/store.go:1123","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-05-02T12:15:17.80472+0800","caller":"mvcc/kvstore.go:385","msg":"kvstore restored","current-rev":1}
{"level":"info","ts":"2023-05-02T12:15:17.806288+0800","caller":"storage/quota.go:93","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-05-02T12:15:17.806355+0800","caller":"etcdserver/server.go:574","msg":"starting etcd server","local-member-id":"8e9e05c52164694d","local-server-version":"3.6.0-alpha.0","cluster-id":"cdf818194e3a8c32","cluster-version":"3.6"}
{"level":"info","ts":"2023-05-02T12:15:17.806568+0800","caller":"embed/etcd.go:281","msg":"now serving peer/client/metrics","local-member-id":"8e9e05c52164694d","initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["http://localhost:2379"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-05-02T12:15:17.806628+0800","caller":"etcdserver/server.go:468","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"8e9e05c52164694d","forward-ticks":9,"forward-duration":"900ms","election-ticks":10,"election-timeout":"1s"}
{"level":"info","ts":"2023-05-02T12:15:17.807083+0800","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"default.etcd/member/snap","suffix":"snap.db","max":5,"interval":"30s"}
{"level":"info","ts":"2023-05-02T12:15:17.807252+0800","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"default.etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}
{"level":"info","ts":"2023-05-02T12:15:17.807361+0800","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"default.etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
{"level":"info","ts":"2023-05-02T12:15:17.807564+0800","caller":"embed/etcd.go:606","msg":"serving peer traffic","address":"127.0.0.1:2380"}
{"level":"info","ts":"2023-05-02T12:15:17.807717+0800","caller":"embed/etcd.go:578","msg":"cmux::serve","address":"127.0.0.1:2380"}
{"level":"warn","ts":"2023-05-02T12:15:17.807086+0800","caller":"etcdserver/metrics.go:199","msg":"failed to get file descriptor usage","error":"cannot get FDUsage on openbsd"}
{"level":"info","ts":"2023-05-02T12:15:17.807778+0800","logger":"raft","caller":"[email protected]/raft.go:1697","msg":"8e9e05c52164694d switched to configuration voters=(10276657743932975437)"}
{"level":"info","ts":"2023-05-02T12:15:17.808285+0800","caller":"membership/cluster.go:404","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"],"added-peer-is-learner":false}
{"level":"info","ts":"2023-05-02T12:15:17.808637+0800","caller":"membership/cluster.go:539","msg":"updated cluster version","cluster-id":"cdf818194e3a8c32","local-member-id":"8e9e05c52164694d","from":"3.6","to":"3.6"}
{"level":"info","ts":"2023-05-02T12:15:18.022088+0800","logger":"raft","caller":"[email protected]/raft.go:819","msg":"8e9e05c52164694d is starting a new election at term 20"}
{"level":"info","ts":"2023-05-02T12:15:18.022212+0800","logger":"raft","caller":"[email protected]/raft.go:757","msg":"8e9e05c52164694d became pre-candidate at term 20"}
{"level":"info","ts":"2023-05-02T12:15:18.022254+0800","logger":"raft","caller":"[email protected]/raft.go:879","msg":"8e9e05c52164694d received MsgPreVoteResp from 8e9e05c52164694d at term 20"}
{"level":"info","ts":"2023-05-02T12:15:18.022274+0800","logger":"raft","caller":"[email protected]/raft.go:741","msg":"8e9e05c52164694d became candidate at term 21"}
{"level":"info","ts":"2023-05-02T12:15:18.022305+0800","logger":"raft","caller":"[email protected]/raft.go:879","msg":"8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 21"}
{"level":"info","ts":"2023-05-02T12:15:18.022317+0800","logger":"raft","caller":"[email protected]/raft.go:797","msg":"8e9e05c52164694d became leader at term 21"}
{"level":"info","ts":"2023-05-02T12:15:18.022329+0800","logger":"raft","caller":"[email protected]/node.go:330","msg":"raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 21"}
{"level":"info","ts":"2023-05-02T12:15:18.023869+0800","caller":"etcdserver/server.go:1738","msg":"published local member to cluster through raft","local-member-id":"8e9e05c52164694d","local-member-attributes":"{Name:default ClientURLs:[http://localhost:2379]}","cluster-id":"cdf818194e3a8c32","publish-timeout":"7s"}
{"level":"info","ts":"2023-05-02T12:15:18.023907+0800","caller":"embed/serve.go:112","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-05-02T12:15:18.025078+0800","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
{"level":"info","ts":"2023-05-02T12:15:18.025172+0800","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
{"level":"info","ts":"2023-05-02T12:15:18.026444+0800","caller":"embed/serve.go:196","msg":"serving client traffic insecurely; this is strongly discouraged!","traffic":"grpc+http","address":"127.0.0.1:2379"}
{"level":"warn","ts":"2023-05-02T12:15:18.026504+0800","caller":"embed/serve.go:169","msg":"stopping insecure grpc server due to error","error":"SetKeepAlivePeriod failed, set tcp 127.0.0.1:2379->127.0.0.1:13254: protocol not available"}
{"level":"error","ts":"2023-05-02T12:15:18.026567+0800","caller":"embed/etcd.go:861","msg":"setting up serving from embedded etcd failed.","error":"mux: server closed","stacktrace":"go.etcd.io/etcd/server/v3/embed.(*Etcd).errHandler\n\tgo.etcd.io/etcd/server/v3/embed/etcd.go:861\ngo.etcd.io/etcd/server/v3/embed.(*serveCtx).serve.func3\n\tgo.etcd.io/etcd/server/v3/embed/serve.go:184"}
{"level":"fatal","ts":"2023-05-02T12:15:18.026607+0800","caller":"etcdmain/etcd.go:197","msg":"listener failed","error":"mux: server closed","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:197\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:250"}
{"level":"warn","ts":"2023-05-02T12:15:18.026633+0800","caller":"embed/serve.go:171","msg":"stopped insecure grpc server due to error","error":"SetKeepAlivePeriod failed, set tcp 127.0.0.1:2379->127.0.0.1:13254: protocol not available"}

[jmhbnz]

Thanks for the detailed issue @tangruize. OpenBSD is not currently a supported platform for etcd, refer: https://etcd.io/docs/v3.5/op-guide/supported-platform

With that in mind I'm going to remove the bug label and replace with feature as adding support would probably be treated as a new feature.

We welcome new contributions so if you wanted to open a pull request and propose a change to enable rudimentary support for OpenBSD please go ahead!

[ahrtr]

OpenBSD has no user-settable per-socket TCP keepalive options. tcpsockopt_openbsd.go#L13-L15

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[ahrtr]

Followup https://github.com/etcd-io/etcd/pull/15812#issuecomment-1929504990

cc @Elbehery

[Elbehery]

raised https://github.com/etcd-io/etcd/pull/17386 & https://github.com/etcd-io/etcd/pull/17387 to backport also raised https://github.com/etcd-io/etcd/pull/17388 for changelog

[jmhbnz]

Discussed during sig-etcd triage meeting. Closing as the work has been done and backported to 3.5.

We will not backport to 3.4 as it would be a breaking change.