sidecar spawns a lot of defunct xtrabackup when running backup with wrong aws credentials

[f84anton]

Host can even run out of PIDs because backup job restarts failed POD. Had to delete POD or docker rm -f <sidecar container id> to get rid of zombie processes.

sidecar container logs:

xtrabackup: recognized server arguments: --innodb_file_per_table=1 --innodb_flush_log_at_trx_commit=2 --innodb_flush_method=O_DIRECT --innodb_log_files_in_group=2 --log_bin=/var/lib/mysql/mysql-bin --open_files_limit=65535 --innodb_buffer_pool_size=2669M --innodb_log_file_size=128M --innodb_io_capacity=2000 --innodb_read_io_threads=64 --innodb_write_io_threads=64 --open_files_limit=65400 --server-id=101 
xtrabackup: recognized client arguments: --backup=1 --slave-info=1 --stream=xbstream --tables-exclude=sys_operator.status --host=127.0.0.1 --user=sys_replication --password=* --target-dir=/tmp/xtrabackup_backupfiles/ 
210723 07:04:27  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;host=127.0.0.1' as 'sys_replication'  (using password: YES).
210723 07:04:27  version_check Connected to MySQL server
210723 07:04:27  version_check Executing a version check against the server...
210723 07:04:27  version_check Done.
210723 07:04:27 Connecting to MySQL server host: 127.0.0.1, user: sys_replication, password: set, port: not set, socket: not set
Using server version 5.7.26-29-log
xtrabackup version 2.4.20 based on MySQL server 5.7.26 Linux (x86_64) (revision id: c8b4056)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql/
xtrabackup: open files limit requested 65400, set to 1048576
xtrabackup: using the following InnoDB configuration:
xtrabackup:   innodb_data_home_dir = .
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 2
xtrabackup:   innodb_log_file_size = 134217728
xtrabackup: using O_DIRECT
InnoDB: Number of pools: 1
2021-07-23T07:04:27.884080651Z	ERROR	sidecar	failed to copy buffer	{"error": "write tcp 10.233.107.235:8080->10.233.105.138:56304: write: connection reset by peer"}
github.com/presslabs/mysql-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error
	/go/src/github.com/presslabs/mysql-operator/vendor/github.com/go-logr/zapr/zapr.go:128
github.com/presslabs/mysql-operator/pkg/sidecar.(*server).backupHandler
	/go/src/github.com/presslabs/mysql-operator/pkg/sidecar/server.go:116
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2007
github.com/presslabs/mysql-operator/pkg/sidecar.maxClients.func1
	/go/src/github.com/presslabs/mysql-operator/pkg/sidecar/server.go:146
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2007
net/http.(*ServeMux).ServeHTTP
	/usr/local/go/src/net/http/server.go:2387
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2802
net/http.(*conn).serve
	/usr/local/go/src/net/http/server.go:1890
2021/07/23 07:04:27 http: superfluous response.WriteHeader call from github.com/presslabs/mysql-operator/pkg/sidecar.(*server).backupHandler (server.go:117)
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
xtrabackup: Error: write to logfile failed
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xtrabackup: Error: xtrabackup_copy_logfile() failed.

process list:

root     60818  0.0  0.0 110128  6236 ?        Sl   jul22   0:08  \_ containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux
/moby/2a108e487bf703bccb5aa2c488931f475446b89df467a2a8c8c5b69a75ea6655 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-
root /var/run/docker/runtime-runc
kube     60865  0.0  0.0  18004  2996 ?        Ss   jul22   0:00  |   \_ /bin/bash /usr/local/bin/sidecar-entrypoint.sh config-and-serve
kube     60936  0.3  0.0 120420 14188 ?        Sl   jul22   2:51  |       \_ mysql-operator-sidecar --debug run
kube     30062  0.0  0.0      0     0 ?        Z    00:00   0:00  |           \_ [xtrabackup] <defunct>
kube     33850  0.0  0.0      0     0 ?        Z    00:00   0:00  |           \_ [xtrabackup] <defunct>
kube     38118  0.0  0.0      0     0 ?        Z    00:00   0:00  |           \_ [xtrabackup] <defunct>

[cndoit18]

/good-first-issue 👍