Insufficient information in logs to debug issues

[nirs]

Current logs lack basic details to be useful for debugging issues:

• timestamp - allows synchronizing with other logs
• connect/disconnect: log more info about the connection (mac address?, ip address?)
• errors: include more context for the failed options for unexpected errors
  • mac address
  • ip address
  • called function and arguments

Detailed error logging may be too slow, need a way to disable it

The following example logs are from 50 runs, each starting 3 lima k8s clusters, provisioning a complex setup, running some tests, and deleting the clusters. 49 of the runs were successful, 1 failed with:

Error building the cluster.Info for the default configuration: error retrieving Submariner: failed to get API group resources: unable to retrieve the complete list of server APIs: submariner.io/v1alpha1: Get "https://192.168.105.11:6443/apis/submariner.io/v1alpha1": dial tcp 192.168.105.11:6443: i/o timeout

Because the logs lack basic details, there is no way to find the related errors.

socket_vmnet was run by launchd, adding KeepAlive to keep it running after errorrs: https://github.com/nirs/socket_vmnet/commit/d92f77207ab935570cb70c5eae86f1d8cba33fdf

I cleared the logs before starting the launchd daemon so the logs should be only from this run. We can see that the daemon was terminated by signal 13 (EACCESS) once, not enough info in the logs to tell why this happened.

Example stdout

Initializing vmnet.framework (mode 1001)
* vmnet_subnet_mask: 255.255.255.0
* vmnet_mtu: 1500
* vmnet_end_address: 192.168.105.254
* vmnet_start_address: 192.168.105.1
* vmnet_interface_id: 71381640-9A80-4CD1-A1EE-AE746AD8E2BB
* vmnet_max_packet_size: 1514
* vmnet_nat66_prefix: fd9b:5a14:ba57:e3d3::
* vmnet_mac_address: 26:73:89:56:71:b6
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)

Received signal 15
Closing a connection (fd 6)
Closing a connection (fd 5)
Initializing vmnet.framework (mode 1001)
* vmnet_subnet_mask: 255.255.255.0
* vmnet_mtu: 1500
* vmnet_end_address: 192.168.105.254
* vmnet_start_address: 192.168.105.1
* vmnet_interface_id: B9124E69-6DD3-415F-A303-F89236C08AA7
* vmnet_max_packet_size: 1514
* vmnet_nat66_prefix: fd9b:5a14:ba57:e3d3::
* vmnet_mac_address: 62:d9:ba:7a:e3:43
Accepted a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 7)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Closing a connection (fd 7)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)

Received signal 13
Closing a connection (fd 7)
Closing a connection (fd 6)
Initializing vmnet.framework (mode 1001)
* vmnet_subnet_mask: 255.255.255.0
* vmnet_mtu: 1500
* vmnet_end_address: 192.168.105.254
* vmnet_start_address: 192.168.105.1
* vmnet_interface_id: 36D40E7D-1C36-4B0F-9C63-12D7EDB81215
* vmnet_max_packet_size: 1514
* vmnet_nat66_prefix: fd9b:5a14:ba57:e3d3::
* vmnet_mac_address: 66:c1:18:9f:80:62
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 7)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 7)
Closing a connection (fd 6)
Closing a connection (fd 5)
Accepted a connection (fd 5)
Accepted a connection (fd 6)
Accepted a connection (fd 7)
Closing a connection (fd 5)
Closing a connection (fd 6)

Received signal 13
Closing a connection (fd 7)

Example stderr

on_accept(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_write: Undefined error: 0
on_accept(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_write: Undefined error: 0
writev: No buffer space available
writev: Broken pipe
writev: Broken pipe
writev: Bad file descriptor
on_accept(): vmnet_return_t VMNET_INVALID_ARGUMENT
on_accept(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_write: Undefined error: 0
vmnet_write: Broken pipe
_on_vmnet_packets_available(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_read: Bad file descriptor
_on_vmnet_packets_available(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_read: Bad file descriptor
writev: Broken pipe