incusd randomly consuming 100% of a CPU until restarted

Open C0rn3j opened this issue 7 months ago • 8 comments

Is there an existing issue for this?

[x] There is no existing issue for this bug

Is this happening on an up to date version of Incus?

[x] This is happening on a supported version of Incus

Incus system details

[0] # incus info
config:
  core.https_address: '[::]:8443'
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- network_sriov
- console
- restrict_dev_incus
- migration_pre_copy
- infiniband
- dev_incus_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- dev_incus_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- backup_compression
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- images_all_projects
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- instances_nic_host_name
- image_copy_profile
- container_syscall_intercept_sysinfo
- clustering_evacuation_mode
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- network_load_balancer
- vsock_api
- instance_ready_state
- network_bgp_holdtime
- storage_volumes_all_projects
- metrics_memory_oom_total
- storage_buckets
- storage_buckets_create_credentials
- metrics_cpu_effective_total
- projects_networks_restricted_access
- storage_buckets_local
- loki
- acme
- internal_metrics
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- storage_volumes_created_at
- cpu_hotplug
- projects_networks_zones
- network_txqueuelen
- cluster_member_state
- instances_placement_scriptlet
- storage_pool_source_wipe
- zfs_block_mode
- instance_generation_id
- disk_io_cache
- amd_sev
- storage_pool_loop_resize
- migration_vm_live
- ovn_nic_nesting
- oidc
- network_ovn_l3only
- ovn_nic_acceleration_vdpa
- cluster_healing
- instances_state_total
- auth_user
- security_csm
- instances_rebuild
- numa_cpu_placement
- custom_volume_iso
- network_allocations
- zfs_delegate
- storage_api_remote_volume_snapshot_copy
- operations_get_query_all_projects
- metadata_configuration
- syslog_socket
- event_lifecycle_name_and_project
- instances_nic_limits_priority
- disk_initial_volume_configuration
- operation_wait
- image_restriction_privileged
- cluster_internal_custom_volume_copy
- disk_io_bus
- storage_cephfs_create_missing
- instance_move_config
- ovn_ssl_config
- certificate_description
- disk_io_bus_virtio_blk
- loki_config_instance
- instance_create_start
- clustering_evacuation_stop_options
- boot_host_shutdown_action
- agent_config_drive
- network_state_ovn_lr
- image_template_permissions
- storage_bucket_backup
- storage_lvm_cluster
- shared_custom_block_volumes
- auth_tls_jwt
- oidc_claim
- device_usb_serial
- numa_cpu_balanced
- image_restriction_nesting
- network_integrations
- instance_memory_swap_bytes
- network_bridge_external_create
- network_zones_all_projects
- storage_zfs_vdev
- container_migration_stateful
- profiles_all_projects
- instances_scriptlet_get_instances
- instances_scriptlet_get_cluster_members
- instances_scriptlet_get_project
- network_acl_stateless
- instance_state_started_at
- networks_all_projects
- network_acls_all_projects
- storage_buckets_all_projects
- resources_load
- instance_access
- project_access
- projects_force_delete
- resources_cpu_flags
- disk_io_bus_cache_filesystem
- instance_oci
- clustering_groups_config
- instances_lxcfs_per_instance
- clustering_groups_vm_cpu_definition
- disk_volume_subpath
- projects_limits_disk_pool
- network_ovn_isolated
- qemu_raw_qmp
- network_load_balancer_health_check
- oidc_scopes
- network_integrations_peer_name
- qemu_scriptlet
- instance_auto_restart
- storage_lvm_metadatasize
- ovn_nic_promiscuous
- ovn_nic_ip_address_none
- instances_state_os_info
- network_load_balancer_state
- instance_nic_macvlan_mode
- storage_lvm_cluster_create
- network_ovn_external_interfaces
- instances_scriptlet_get_instances_count
- cluster_rebalance
- custom_volume_refresh_exclude_older_snapshots
- storage_initial_owner
- storage_live_migration
- instance_console_screenshot
- image_import_alias
- authorization_scriptlet
- console_force
- network_ovn_state_addresses
- network_bridge_acl_devices
- instance_debug_memory
- init_preseed_storage_volumes
- init_preseed_profile_project
- instance_nic_routed_host_address
- instance_smbios11
- api_filtering_extended
- acme_dns01
- security_iommu
- network_ipv4_dhcp_routes
- network_state_ovn_ls
- network_dns_nameservers
- acme_http01_port
- network_ovn_ipv4_dhcp_expiry
- instance_state_cpu_time
- network_io_bus
- disk_io_bus_usb
- storage_driver_linstor
- instance_oci_entrypoint
- network_address_set
- server_logging
- network_forward_snat
- memory_hotplug
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
auth_user_name: root
auth_user_method: unix
environment:
  addresses:
  - 192.168.50.201:8443
  - 10.164.69.1:8443
  - '[fd42:1ee:a97:164e::1]:8443'
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIICAzCCAYqgAwIBAgIRAPLhg/Io/urmLGfp5mkAi2MwCgYIKoZIzj0EAwMwNDEc
    MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEUMBIGA1UEAwwLcm9vdEBQYXRy
    aWswHhcNMjIxMjA2MjA0NjU0WhcNMzIxMjAzMjA0NjU0WjA0MRwwGgYDVQQKExNs
    aW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QFBhdHJpazB2MBAGByqG
    SM49AgEGBSuBBAAiA2IABPmdLjlyvdAq/zAgvi+APEb1pGUzufXKJoU0HWDKwF0T
    KVsdqan6mwfa0YM0tVasTP9JYl6rGLiqFyzr6ewvx6I6QG1QIR+L+FP7F29PBgTt
    SHaqPAYLlk/pcIQnDF1kfaNgMF4wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoG
    CCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0RBCIwIIIGUGF0cmlrhwR/AAAB
    hxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2cAMGQCMBqI/ulJ2KsDeazB
    z7QLREJgofSsqrvT/KKIBsO+/OcqrfyvlBLBKsD15C33730F2AIwCDG7i3YGhl3Z
    1aziKbDxt5ey2SAwtfaxXquYnVCSdu6aCJIggkbNhfdXlAZfmvqq
    -----END CERTIFICATE-----
  certificate_fingerprint: 715a5a733bfde6c870e0c589c2dad743d59df626b6f4d5337ddc7bd2172eb9ff
  driver: lxc
  driver_version: 6.0.4
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    uevent_injection: "true"
    unpriv_binfmt: "true"
    unpriv_fscaps: "true"
  kernel_version: 6.14.6-arch1-1
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Arch Linux
  os_version: ""
  project: default
  server: incus
  server_clustered: false
  server_event_mode: full-mesh
  server_name: Patrik
  server_pid: 469
  server_version: "6.12"
  storage: dir
  storage_version: "1"
  storage_supported_drivers:
  - name: dir
    version: "1"
    remote: false
  - name: lvm
    version: 2.03.32(2) (2025-05-05) / 1.02.206 (2025-05-05) / 4.49.0
    remote: false
  - name: lvmcluster
    version: 2.03.32(2) (2025-05-05) / 1.02.206 (2025-05-05) / 4.49.0
    remote: true

Instance details

No response

Instance log

No response

Current behavior

incusd randomly consumes 100% of CPU, I can reboot the server or the incus service to make it stop, but it comes back in a couple hours.

There is a single small container running on this instance.

My other two+ Arch machines do not seem to suffer from this.

This seems to be happening since a week or two ago, until then Incus was running fine, there was no update of core incus in that time either.

This is the full incusd.log from a server reboot, there seems to be nothing interesting:

time="2025-05-21T00:30:17+02:00" level=warning msg="Ignoring signal, shutdown already in progress" signal=terminated
time="2025-05-21T00:30:18+02:00" level=warning msg="Failed shutting down instance, forcefully stopping" err="Failed shutting down instance, status is \"Running\": context deadline exceeded" instance=guard-guild project=default
time="2025-05-21T00:30:38+02:00" level=warning msg="AppArmor support has been disabled because of lack of kernel support"
time="2025-05-21T00:30:38+02:00" level=warning msg=" - AppArmor support has been disabled, Disabled because of lack of kernel support"
time="2025-05-21T00:30:38+02:00" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine

Expected behavior

No response

Steps to reproduce

Start incusd
Wait a few hours
100% CPU usage with nothing in the logs about it

Side note, is it not possible to wrap the incus template in a <details><summary>...</summary> LOG HERE </details> tags so it does not consume the bug report visually by default?

May 25 '25 19:05 C0rn3j

Can you try running:

incus config set core.debug_address :8444

That should then let you access pprof profiles such as the list of all goroutines:

curl http://127.0.0.1:8444/debug/pprof/goroutine?debug=2

Or get a CPU measurement:

curl "http://127.0.0.1:8444/debug/pprof/profile" -o cpu.pprof
go tool pprof cpu.pprof
  top

May 25 '25 20:05 stgraber

Will do the next time it happens, which will probably be within 24h.

May 25 '25 20:05 C0rn3j

I noticed one of my other servers is actually experiencing this right now, albeit that seems to be an older incusd instance as the uptime there is 188 days, so it may not be the same issue or something that was fixed already - routine.zip

The up to date rebooted server hasn't ran into it again yet.

May 27 '25 06:05 C0rn3j

And it finally hit the OG server too.

Log

[0] # curl http://127.0.0.1:8444/debug/pprof/goroutine\?debug\=2

goroutine 111028 [running]:
runtime/pprof.writeGoroutineStacks({0x64b18186e460, 0xc000dd00e0})
        /usr/lib/go/src/runtime/pprof/pprof.go:764 +0x6a
runtime/pprof.writeGoroutine({0x64b18186e460?, 0xc000dd00e0?}, 0x16?)
        /usr/lib/go/src/runtime/pprof/pprof.go:753 +0x25
runtime/pprof.(*Profile).WriteTo(0x64b1826a9050?, {0x64b18186e460?, 0xc000dd00e0?}, 0xc?)
        /usr/lib/go/src/runtime/pprof/pprof.go:377 +0x14b
net/http/pprof.handler.ServeHTTP({0xc00047e011, 0x9}, {0x64b18187d468, 0xc000dd00e0}, 0xc00067c140)
        /usr/lib/go/src/net/http/pprof/pprof.go:272 +0x52f
net/http/pprof.Index({0x64b18187d468, 0xc000dd00e0}, 0xc00067c140?)
        /usr/lib/go/src/net/http/pprof/pprof.go:389 +0xde
net/http.HandlerFunc.ServeHTTP(0x64b1827b73a0?, {0x64b18187d468?, 0xc000dd00e0?}, 0x64b17fabed96?)
        /usr/lib/go/src/net/http/server.go:2294 +0x29
net/http.(*ServeMux).ServeHTTP(0x64b17f823c19?, {0x64b18187d468, 0xc000dd00e0}, 0xc00067c140)
        /usr/lib/go/src/net/http/server.go:2822 +0x1c4
net/http.serverHandler.ServeHTTP({0xc0006c81b0?}, {0x64b18187d468?, 0xc000dd00e0?}, 0x1?)
        /usr/lib/go/src/net/http/server.go:3301 +0x8e
net/http.(*conn).serve(0xc000456120, {0x64b181881590, 0xc0006da030})
        /usr/lib/go/src/net/http/server.go:2102 +0x625
created by net/http.(*Server).Serve in goroutine 1943
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 1 [select, 5943 minutes]:
main.(*cmdDaemon).Run(0xc00031eee8, 0xc0004b9a50?, {0xc000530da0, 0x0, 0x0?})
        /build/incus/src/incus-6.12/cmd/incusd/main_daemon.go:80 +0x5b2
github.com/spf13/cobra.(*Command).execute(0xc00058a008, {0xc00003e520, 0x2, 0x2})
        /build/incus/src/incus-6.12/vendor/github.com/spf13/cobra/command.go:1015 +0xa94
github.com/spf13/cobra.(*Command).ExecuteC(0xc00058a008)
        /build/incus/src/incus-6.12/vendor/github.com/spf13/cobra/command.go:1148 +0x40c
github.com/spf13/cobra.(*Command).Execute(...)
        /build/incus/src/incus-6.12/vendor/github.com/spf13/cobra/command.go:1071
main.main()
        /build/incus/src/incus-6.12/cmd/incusd/main.go:201 +0x17ba

goroutine 19 [syscall, 5943 minutes]:
os/signal.signal_recv()
        /usr/lib/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
        /usr/lib/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/lib/go/src/os/signal/signal.go:152 +0x1f

goroutine 20 [select, 5943 minutes]:
database/sql.(*DB).connectionOpener(0xc0005c0270, {0x64b1818815c8, 0xc0003213b0})
        /usr/lib/go/src/database/sql/sql.go:1261 +0x87
created by database/sql.OpenDB in goroutine 1
        /usr/lib/go/src/database/sql/sql.go:841 +0x130

goroutine 351 [select, 2 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00031ee28, {0x64b1818815c8, 0xc00075aaa0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x5)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 354 [select, 185 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00031eeb8, {0x64b1818815c8, 0xc00075aaa0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x8)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 350 [select, 4 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00031edf8, {0x64b1818815c8, 0xc00075aaa0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x4)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 51 [select, 5943 minutes]:
database/sql.(*DB).connectionOpener(0xc000121e10, {0x64b1818815c8, 0xc0003cb130})
        /usr/lib/go/src/database/sql/sql.go:1261 +0x87
created by database/sql.OpenDB in goroutine 1
        /usr/lib/go/src/database/sql/sql.go:841 +0x130

goroutine 50 [IO wait, 66 minutes]:
internal/poll.runtime_pollWait(0x7a2cc683b4c0, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000545100?, 0x0?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000545100)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc000545100)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000324780)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc000324780)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
github.com/lxc/incus/v6/internal/server/endpoints/listeners.(*FancyTLSListener).Accept(0xc0003caff0)
        /build/incus/src/incus-6.12/internal/server/endpoints/listeners/fancytls.go:37 +0x4a
net/http.(*Server).Serve(0xc0003fe500, {0x64b18187d888, 0xc0003caff0})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.12/internal/server/endpoints/endpoints.go:432 +0x1f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0003cb040, 0xc0003216d0?)
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xdb

goroutine 49 [IO wait, 5943 minutes]:
internal/poll.runtime_pollWait(0x7a2cc683b5d8, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000544f80?, 0x7a2cc69a85c0?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000544f80)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc000544f80)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x64b1815acd01?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc0004b5f20)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
net/http.(*Server).Serve(0xc0003fe600, {0x64b18187cf88, 0xc0004b5f20})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.12/internal/server/endpoints/endpoints.go:432 +0x1f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0003cb040, 0xc000540f80?)
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xdb

goroutine 48 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7a2cc683b6f0, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000544e80?, 0xc0006aa000?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000544e80)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc000544e80)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x10?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc0004b5e60)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
github.com/lxc/incus/v6/internal/server/endpoints/listeners.(*StarttlsListener).Accept(0xc0004b5ec0)
        /build/incus/src/incus-6.12/internal/server/endpoints/listeners/starttls.go:36 +0x4d
net/http.(*Server).Serve(0xc0003fe500, {0x64b18187d8e8, 0xc0004b5ec0})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.12/internal/server/endpoints/endpoints.go:432 +0x1f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0003cb040, 0x64b1826991f0?)
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xdb

goroutine 346 [select, 185 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00031ed38, {0x64b1818815c8, 0xc00075aaa0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x0)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 349 [select, 185 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00031edc8, {0x64b1818815c8, 0xc00075aaa0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x3)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 348 [select, 4 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00031ed98, {0x64b1818815c8, 0xc00075aaa0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x2)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 109031 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc00061fb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d1900?, {0x64b180f7b2b8?, 0xc000782180?}, {0xc00061ff88?, 0x64b18172a720?, 0xc0007820f0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f99e0, {0x64b181881590, 0xc000782150})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 285 [chan receive, 5943 minutes]:
github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load.func1()
        /build/incus/src/incus-6.12/internal/server/fsmonitor/drivers/driver_fanotify.go:66 +0x2c
created by github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load in goroutine 1
        /build/incus/src/incus-6.12/internal/server/fsmonitor/drivers/driver_fanotify.go:65 +0x28c

goroutine 109688 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000855b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0006af340?, {0x64b180f7b2b8?, 0xc0006c9650?}, {0xc000855f88?, 0x64b18172a720?, 0xc0006c95c0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000106d80, {0x64b181881590, 0xc0006c9620})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 256 [select, 5943 minutes]:
main.deviceEventListener(0xc0007877a0)
        /build/incus/src/incus-6.12/cmd/incusd/devices.go:623 +0x1dc
created by main.(*Daemon).init in goroutine 1
        /build/incus/src/incus-6.12/cmd/incusd/daemon.go:1583 +0x4165

goroutine 297 [IO wait, 5943 minutes]:
internal/poll.runtime_pollWait(0x7a2cc683af48, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0006d4b80?, 0x7a2cc69a8108?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0006d4b80)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc0006d4b80)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x64b17f7bd8f7?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc0007a1230)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
github.com/lxc/incus/v6/internal/server/seccomp.NewSeccompServer.func1()
        /build/incus/src/incus-6.12/internal/server/seccomp/seccomp.go:1083 +0x59
created by github.com/lxc/incus/v6/internal/server/seccomp.NewSeccompServer in goroutine 1
        /build/incus/src/incus-6.12/internal/server/seccomp/seccomp.go:1081 +0x1ed

goroutine 286 [syscall, 5943 minutes]:
syscall.Syscall(0x0, 0x1b, 0xc000326800, 0x100)
        /usr/lib/go/src/syscall/syscall_linux.go:73 +0x25
golang.org/x/sys/unix.read(0xc00006dcd8?, {0xc000326800?, 0x7a2cc69a85c0?, 0xc000071008?})
        /build/incus/src/incus-6.12/vendor/golang.org/x/sys/unix/zsyscall_linux.go:1394 +0x45
golang.org/x/sys/unix.Read(...)
        /build/incus/src/incus-6.12/vendor/golang.org/x/sys/unix/syscall_unix.go:167
github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).getEvents(0xc000719180, {0x64b1818815c8, 0xc0003211d0}, 0x1c)
        /build/incus/src/incus-6.12/internal/server/fsmonitor/drivers/driver_fanotify.go:86 +0x78
created by github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load in goroutine 1
        /build/incus/src/incus-6.12/internal/server/fsmonitor/drivers/driver_fanotify.go:71 +0x2fe

goroutine 257 [syscall, 5873 minutes]:
syscall.Syscall(0x0, 0x1d, 0xc0007a7000, 0x1000)
        /usr/lib/go/src/syscall/syscall_linux.go:73 +0x25
golang.org/x/sys/unix.read(0xc0005ba700?, {0xc0007a7000?, 0x64b180f0d0a3?, 0x2?})
        /build/incus/src/incus-6.12/vendor/golang.org/x/sys/unix/zsyscall_linux.go:1394 +0x45
golang.org/x/sys/unix.Read(...)
        /build/incus/src/incus-6.12/vendor/golang.org/x/sys/unix/syscall_unix.go:167
main.deviceNetlinkListener.func1(0xc0007947e0, 0xc0006cb3b0, 0xc0006cb420)
        /build/incus/src/incus-6.12/cmd/incusd/devices.go:69 +0x8d
created by main.deviceNetlinkListener in goroutine 256
        /build/incus/src/incus-6.12/cmd/incusd/devices.go:66 +0x190

goroutine 347 [select, 185 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00031ed68, {0x64b1818815c8, 0xc00075aaa0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x1)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 109713 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000771b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc00011fdc0?, {0x64b180f7b2b8?, 0xc0006dbcb0?}, {0xc000771f88?, 0x64b18172a720?, 0xc0006dbb30?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f9b90, {0x64b181881590, 0xc0006dbc50})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109027 [runnable]:
bytes.indexFunc({0xc0006484d0, 0x70, 0x70}, 0x64b181863478, 0x0)
        /usr/lib/go/src/bytes/bytes.go:913 +0x111
bytes.TrimLeftFunc(...)
        /usr/lib/go/src/bytes/bytes.go:852
github.com/lxc/incus/v6/internal/server/endpoints.networkServerErrorLogWriter.stripLog({{0x0?, 0x70?, 0xc000764680?}}, {0xc0006484d0?, 0xc000247080?, 0x64b17fa8cccc?})
        /build/incus/src/incus-6.12/internal/server/endpoints/network_util.go:31 +0x93
github.com/lxc/incus/v6/internal/server/endpoints.networkServerErrorLogWriter.Write({{0x0?, 0x64b180fae597?, 0x37?}}, {0xc0006484d0?, 0x70, 0x2?})
        /build/incus/src/incus-6.12/internal/server/endpoints/network_util.go:19 +0x25
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000247270)
        /usr/lib/go/src/log/log.go:245 +0x45a
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x64b18186dc48?, {0x64b180fae597?, 0x64b18186ed40?}, {0xc0002476d8?, 0x64b17f9eae1b?, 0xc0002472b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc000414000, {0x64b18186dc48, 0xc000787f30}, {0x64b180fae597, 0x37}, {0xc0002476d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc000414000, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc000452000, {0x64b18188ee70, 0xc0002ba008}, 0xc000247af8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0003fe500, {0x64b18188ee70, 0xc0002ba008}, {0x64b18186e3c0, 0xc0005309e0}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x64b18172a480?, 0xc000247ea8?, {0x64b18186e3c0?, 0xc0005309e0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0000f98c0, {0x64b181881590, 0xc0006dbbf0})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 353 [select, 185 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00031ee88, {0x64b1818815c8, 0xc00075aaa0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x7)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 352 [select, 2 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00031ee58, {0x64b1818815c8, 0xc00075aaa0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x6)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 355 [select, 2 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00031ef78, {0x64b1818815c8, 0xc00075aaa0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x9)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 233 [syscall, 5943 minutes]:
syscall.Syscall6(0xf7, 0x3, 0x1e, 0xc000619600, 0x4, 0xc0006b2360, 0x0)
        /usr/lib/go/src/syscall/syscall_linux.go:95 +0x39
internal/syscall/unix.Waitid(0x20000007f83a580?, 0x7a2c7d222c68?, 0x7a2cc69a8108?, 0x10?, 0xc000548808?)
        /usr/lib/go/src/internal/syscall/unix/waitid_linux.go:18 +0x39
os.(*Process).pidfdWait.func1(...)
        /usr/lib/go/src/os/pidfd_linux.go:106
os.ignoringEINTR(...)
        /usr/lib/go/src/os/file_posix.go:251
os.(*Process).pidfdWait(0x64b182d07a68?)
        /usr/lib/go/src/os/pidfd_linux.go:105 +0x209
os.(*Process).wait(0xc000445320?)
        /usr/lib/go/src/os/exec_unix.go:27 +0x25
os.(*Process).Wait(...)
        /usr/lib/go/src/os/exec.go:358
os/exec.(*Cmd).Wait(0xc0006ad080)
        /usr/lib/go/src/os/exec/exec.go:922 +0x45
github.com/lxc/incus/v6/shared/subprocess.(*Process).start.func3()
        /build/incus/src/incus-6.12/shared/subprocess/proc.go:204 +0x59
created by github.com/lxc/incus/v6/shared/subprocess.(*Process).start in goroutine 1
        /build/incus/src/incus-6.12/shared/subprocess/proc.go:201 +0x6c6

goroutine 109073 [sync.Mutex.Lock, 102 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc0006e1b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000476380?, {0x64b180f7b2b8?, 0xc0006c93e0?}, {0xc0006e1f88?, 0x64b18172a720?, 0xc0006c92f0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000456fc0, {0x64b181881590, 0xc0006c93b0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 111029 [runnable]:
net/http.(*connReader).startBackgroundRead.gowrap2()
        /usr/lib/go/src/net/http/server.go:686
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1
created by net/http.(*connReader).startBackgroundRead in goroutine 111028
        /usr/lib/go/src/net/http/server.go:686 +0xb6

goroutine 109032 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000623b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d1a00?, {0x64b180f7b2b8?, 0xc000782240?}, {0xc000623f88?, 0x64b18172a720?, 0xc0007821b0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f9a70, {0x64b181881590, 0xc000782210})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109074 [sync.Mutex.Lock, 102 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000857b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000476440?, {0x64b180f7b2b8?, 0xc0006c9500?}, {0xc000857f88?, 0x64b18172a720?, 0xc0006c9410?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000457050, {0x64b181881590, 0xc0006c94d0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109693 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc00076bb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0006134a0?, {0x64b180f7b2b8?, 0xc0006c9ad0?}, {0xc00076bf88?, 0x64b18172a720?, 0xc0006c9a40?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000107050, {0x64b181881590, 0xc0006c9aa0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109689 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc00061db58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0006af480?, {0x64b180f7b2b8?, 0xc0006c9710?}, {0xc00061df88?, 0x64b18172a720?, 0xc0006c9680?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000106e10, {0x64b181881590, 0xc0006c96e0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109709 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc00076fb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000476b40?, {0x64b180f7b2b8?, 0xc0006db740?}, {0xc00076ff88?, 0x64b18172a720?, 0xc0006db6b0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f9710, {0x64b181881590, 0xc0006db710})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109714 [sync.Mutex.Lock, 66 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc0006e3b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d0140?, {0x64b180f7b2b8?, 0xc0006c80f0?}, {0xc0006e3f88?, 0x64b18172a720?, 0xc0006c8060?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000106000, {0x64b181881590, 0xc0006c80c0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109712 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc00088bb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x64b180f046f7?, {0x64b180f7b2b8?, 0xc0006dba40?}, {0xc00088bf88?, 0x64b18172a720?, 0xc0006db9b0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f9b00, {0x64b181881590, 0xc0006dba10})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109715 [sync.Mutex.Lock, 66 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc00076db58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d0200?, {0x64b180f7b2b8?, 0xc0006c81b0?}, {0xc00076df88?, 0x64b18172a720?, 0xc0006c8120?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000106090, {0x64b181881590, 0xc0006c8180})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109030 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000b15b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d1800?, {0x64b180f7b2b8?, 0xc0007820c0?}, {0xc000b15f88?, 0x64b18172a720?, 0xc000782030?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f9950, {0x64b181881590, 0xc000782090})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109716 [sync.Mutex.Lock, 66 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc00089fb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d0340?, {0x64b180f7b2b8?, 0xc0006c8270?}, {0xc00089ff88?, 0x64b18172a720?, 0xc0006c81e0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000106120, {0x64b181881590, 0xc0006c8240})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109691 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000853b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0006af680?, {0x64b180f7b2b8?, 0xc0006c9890?}, {0xc000853f88?, 0x64b18172a720?, 0xc0006c9800?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000106f30, {0x64b181881590, 0xc0006c9860})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109694 [sync.Mutex.Lock, 66 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000877b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0006ae080?, {0x64b180f7b2b8?, 0xc0006da120?}, {0xc000877f88?, 0x64b18172a720?, 0xc0006da060?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f83f0, {0x64b181881590, 0xc0006da0f0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109711 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc00087fb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x64b180fb14ca?, {0x64b180f7b2b8?, 0xc0006db980?}, {0xc00087ff88?, 0x64b18172a720?, 0xc0006db8f0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f9830, {0x64b181881590, 0xc0006db950})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109690 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000851b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0006af580?, {0x64b180f7b2b8?, 0xc0006c97d0?}, {0xc000851f88?, 0x64b18172a720?, 0xc0006c9740?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000106ea0, {0x64b181881590, 0xc0006c97a0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109708 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc0006dfb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000476a40?, {0x64b180f7b2b8?, 0xc0006db680?}, {0xc0006dff88?, 0x64b18172a720?, 0xc0006db5f0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f9680, {0x64b181881590, 0xc0006db650})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109022 [sync.Mutex.Lock, 102 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000051270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x64b18186dc48?, {0x64b180fae597?, 0x64b18186ed40?}, {0xc0000516d8?, 0x64b17f9eae1b?, 0xc0000512b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc000414780, {0x64b18186dc48, 0xc00069e4f0}, {0x64b180fae597, 0x37}, {0xc0000516d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc000414780, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc000452000, {0x64b18188ee70, 0xc0007d5508}, 0xc000051af8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0003fe500, {0x64b18188ee70, 0xc0007d5508}, {0x64b18186e3c0, 0xc000530340}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x64b18172a480?, 0xc000051ea8?, {0x64b18186e3c0?, 0xc000530340?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0000f8090, {0x64b181881590, 0xc0007820c0})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109025 [sync.Mutex.Lock, 102 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc0006ed270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x64b18186dc48?, {0x64b180fae597?, 0x64b18186ed40?}, {0xc0006ed6d8?, 0x64b17f9eae1b?, 0xc0006ed2b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc000414960, {0x64b18186dc48, 0xc00069e620}, {0x64b180fae597, 0x37}, {0xc0006ed6d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc000414960, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc000452000, {0x64b18188ee70, 0xc0007d5c08}, 0xc0006edaf8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0003fe500, {0x64b18188ee70, 0xc0007d5c08}, {0x64b18186e3c0, 0xc0005307a0}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x64b18172a480?, 0xc0006edea8?, {0x64b18186e3c0?, 0xc0005307a0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0000f8120, {0x64b181881590, 0xc000782540})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 1943 [IO wait]:
internal/poll.runtime_pollWait(0x7a2cc683b290, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000540b80?, 0x9007c59be?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000540b80)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc000540b80)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc00077d000)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc00077d000)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
net/http.(*Server).Serve(0xc0003feb00, {0x64b18187cf58, 0xc00077d000})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.12/internal/server/endpoints/endpoints.go:432 +0x1f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0003cb040, 0xc00079c370?)
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1929
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xdb

goroutine 109707 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc0006e5b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0004764c0?, {0x64b180f7b2b8?, 0xc0006db560?}, {0xc0006e5f88?, 0x64b18172a720?, 0xc0006db4d0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f95f0, {0x64b181881590, 0xc0006db530})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109021 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000621b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0004760c0?, {0x64b180f7b2b8?, 0xc0006c8ba0?}, {0xc000621f88?, 0x64b18172a720?, 0xc0006c8b10?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000456a20, {0x64b181881590, 0xc0006c8b70})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109044 [sync.Mutex.Lock, 102 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc0006e9270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x64b18186dc48?, {0x64b180fae597?, 0x64b18186ed40?}, {0xc0006e96d8?, 0x64b17f9eae1b?, 0xc0006e92b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc0004150e0, {0x64b18186dc48, 0xc00069e6a0}, {0x64b180fae597, 0x37}, {0xc0006e96d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc0004150e0, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc000452000, {0x64b18188ee70, 0xc000437888}, 0xc0006e9af8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0003fe500, {0x64b18188ee70, 0xc000437888}, {0x64b18186e3c0, 0xc000530bc0}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x64b18172a480?, 0xc0006e9ea8?, {0x64b18186e3c0?, 0xc000530bc0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0000f82d0, {0x64b181881590, 0xc000782cc0})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109038 [sync.Mutex.Lock, 102 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000b17b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0004761c0?, {0x64b180f7b2b8?, 0xc0006c8090?}, {0xc000b17f88?, 0x64b18172a720?, 0xc0006c8000?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000456090, {0x64b181881590, 0xc0006c8060})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109710 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000883b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x64b180f046f7?, {0x64b180f7b2b8?, 0xc0006db800?}, {0xc000883f88?, 0x64b18172a720?, 0xc0006db770?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000f97a0, {0x64b181881590, 0xc0006db7d0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109035 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000243270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x64b18186dc48?, {0x64b180fae597?, 0x64b18186ed40?}, {0xc0002436d8?, 0x64b17f9eae1b?, 0xc0002432b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc0004145a0, {0x64b18186dc48, 0xc00069e810}, {0x64b180fae597, 0x37}, {0xc0002436d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc0004145a0, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc000452000, {0x64b18188ee70, 0xc0007d5888}, 0xc000243af8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0003fe500, {0x64b18188ee70, 0xc0007d5888}, {0x64b18186e3c0, 0xc000531120}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x64b18172a480?, 0xc000243ea8?, {0x64b18186e3c0?, 0xc000531120?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0000f9c20, {0x64b181881590, 0xc000782810})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 109692 [sync.Mutex.Lock, 68 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006c9350)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006c9350, 0x0, 0x2, 0xc000037b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x64b180f046f7?, {0x64b180f7b2b8?, 0xc0006c9950?}, {0xc000037f88?, 0x64b18172a720?, 0xc0006c98c0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000106fc0, {0x64b181881590, 0xc0006c9920})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 50
        /usr/lib/go/src/net/http/server.go:3454 +0x485

And the secondary one, which is also up to date now:

Log secondary

[0] # curl http://127.0.0.1:8444/debug/pprof/goroutine\?debug\=2
goroutine 74052 [running]:
runtime/pprof.writeGoroutineStacks({0x562926a50460, 0xc0006360e0})
        /usr/lib/go/src/runtime/pprof/pprof.go:764 +0x6a
runtime/pprof.writeGoroutine({0x562926a50460?, 0xc0006360e0?}, 0x26?)
        /usr/lib/go/src/runtime/pprof/pprof.go:753 +0x25
runtime/pprof.(*Profile).WriteTo(0x56292788b050?, {0x562926a50460?, 0xc0006360e0?}, 0xc?)
        /usr/lib/go/src/runtime/pprof/pprof.go:377 +0x14b
net/http/pprof.handler.ServeHTTP({0xc000608011, 0x9}, {0x562926a5f468, 0xc0006360e0}, 0xc00077e000)
        /usr/lib/go/src/net/http/pprof/pprof.go:272 +0x52f
net/http/pprof.Index({0x562926a5f468, 0xc0006360e0}, 0xc00077e000?)
        /usr/lib/go/src/net/http/pprof/pprof.go:389 +0xde
net/http.HandlerFunc.ServeHTTP(0x5629279993a0?, {0x562926a5f468?, 0xc0006360e0?}, 0x562924ca0d96?)
        /usr/lib/go/src/net/http/server.go:2294 +0x29
net/http.(*ServeMux).ServeHTTP(0x562924a05c19?, {0x562926a5f468, 0xc0006360e0}, 0xc00077e000)
        /usr/lib/go/src/net/http/server.go:2822 +0x1c4
net/http.serverHandler.ServeHTTP({0xc0006140f0?}, {0x562926a5f468?, 0xc0006360e0?}, 0x1?)
        /usr/lib/go/src/net/http/server.go:3301 +0x8e
net/http.(*conn).serve(0xc0000d6ab0, {0x562926a63590, 0xc00064a120})
        /usr/lib/go/src/net/http/server.go:2102 +0x625
created by net/http.(*Server).Serve in goroutine 42
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 1 [select, 3857 minutes]:
main.(*cmdDaemon).Run(0xc000140840, 0xc0004bba50?, {0xc000119d80, 0x0, 0x0?})
        /build/incus/src/incus-6.12/cmd/incusd/main_daemon.go:80 +0x5b2
github.com/spf13/cobra.(*Command).execute(0xc0005d8008, {0xc000112160, 0x2, 0x2})
        /build/incus/src/incus-6.12/vendor/github.com/spf13/cobra/command.go:1015 +0xa94
github.com/spf13/cobra.(*Command).ExecuteC(0xc0005d8008)
        /build/incus/src/incus-6.12/vendor/github.com/spf13/cobra/command.go:1148 +0x40c
github.com/spf13/cobra.(*Command).Execute(...)
        /build/incus/src/incus-6.12/vendor/github.com/spf13/cobra/command.go:1071
main.main()
        /build/incus/src/incus-6.12/cmd/incusd/main.go:201 +0x17ba

goroutine 8 [syscall, 3857 minutes]:
os/signal.signal_recv()
        /usr/lib/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
        /usr/lib/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/lib/go/src/os/signal/signal.go:152 +0x1f

goroutine 43 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x7f3cf5ac1578, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0005f8400?, 0xc0007c7000?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0005f8400)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc0005f8400)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x10?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc000525c80)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
github.com/lxc/incus/v6/internal/server/endpoints/listeners.(*StarttlsListener).Accept(0xc000525ce0)
        /build/incus/src/incus-6.12/internal/server/endpoints/listeners/starttls.go:36 +0x4d
net/http.(*Server).Serve(0xc0004a0900, {0x562926a5f8e8, 0xc000525ce0})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.12/internal/server/endpoints/endpoints.go:432 +0x1f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000584870, 0xc00045c200?)
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xdb

goroutine 61 [select, 3857 minutes]:
github.com/mdlayher/socket.rwT[...].func2()
        /build/incus/src/incus-6.12/vendor/github.com/mdlayher/socket/conn.go:778 +0xb5
created by github.com/mdlayher/socket.rwT[...] in goroutine 45
        /build/incus/src/incus-6.12/vendor/github.com/mdlayher/socket/conn.go:775 +0x4f2

goroutine 42 [IO wait]:
internal/poll.runtime_pollWait(0x7f3cf5ac1348, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0005f8800?, 0x9009a79be?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0005f8800)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc0005f8800)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc0001210c0)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc0001210c0)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
net/http.(*Server).Serve(0xc0004a0f00, {0x562926a5ef58, 0xc0001210c0})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.12/internal/server/endpoints/endpoints.go:432 +0x1f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000584870, 0x56292787b1f0?)
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xdb

goroutine 30 [select, 3857 minutes]:
database/sql.(*DB).connectionOpener(0xc0006136c0, {0x562926a635c8, 0xc000205270})
        /usr/lib/go/src/database/sql/sql.go:1261 +0x87
created by database/sql.OpenDB in goroutine 1
        /usr/lib/go/src/database/sql/sql.go:841 +0x130

goroutine 45 [IO wait, 3857 minutes]:
internal/poll.runtime_pollWait(0x7f3cf5ac1230, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0005bb4a0?, 0x7f3d3cd835c0?, 0x1)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).RawRead(0xc0005bb4a0, 0xc00044eaf0)
        /usr/lib/go/src/internal/poll/fd_unix.go:717 +0x125
os.(*rawConn).Read(0xc000117048, 0x5629255ffc01?)
        /usr/lib/go/src/os/rawconn.go:31 +0x4a
github.com/mdlayher/socket.rwT[...](0xc000121040, {{0x562926a63478, 0x562927ee9a20}, 0x0, {0x5629260f04a0, 0x7}, 0xc000012f70})
        /build/incus/src/incus-6.12/vendor/github.com/mdlayher/socket/conn.go:795 +0x5e6
github.com/mdlayher/socket.readT[...](0xc000082cc0?, {0x562926a63478?, 0x562927ee9a20?}, {0x5629260f04a0?, 0x30?}, 0x5629261152d2?)
        /build/incus/src/incus-6.12/vendor/github.com/mdlayher/socket/conn.go:666 +0x6e
github.com/mdlayher/socket.(*Conn).Accept(0xc000121040, {0x562926a63478, 0x562927ee9a20}, 0x0)
        /build/incus/src/incus-6.12/vendor/github.com/mdlayher/socket/conn.go:421 +0x70
github.com/mdlayher/vsock.(*listener).Accept(0xc000532910)
        /build/incus/src/incus-6.12/vendor/github.com/mdlayher/vsock/listener_linux.go:33 +0x2f
github.com/mdlayher/vsock.(*Listener).Accept(0xc000117050)
        /build/incus/src/incus-6.12/vendor/github.com/mdlayher/vsock/vsock.go:133 +0x1b
github.com/lxc/incus/v6/internal/server/endpoints/listeners.(*FancyTLSListener).Accept(0xc000584730)
        /build/incus/src/incus-6.12/internal/server/endpoints/listeners/fancytls.go:37 +0x4a
net/http.(*Server).Serve(0xc0004a0d00, {0x562926a5f888, 0xc000584730})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.12/internal/server/endpoints/endpoints.go:432 +0x1f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000584870, 0xc00044e320?)
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xdb

goroutine 46 [IO wait, 16 minutes]:
internal/poll.runtime_pollWait(0x7f3cf5ac1460, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0005f8700?, 0xc000102808?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0005f8700)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc0005f8700)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000121080)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc000121080)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
github.com/lxc/incus/v6/internal/server/endpoints/listeners.(*FancyTLSListener).Accept(0xc0005847d0)
        /build/incus/src/incus-6.12/internal/server/endpoints/listeners/fancytls.go:37 +0x4a
net/http.(*Server).Serve(0xc0004a0900, {0x562926a5f888, 0xc0005847d0})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.12/internal/server/endpoints/endpoints.go:432 +0x1f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000584870, 0xc00045c100?)
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xdb

goroutine 44 [IO wait, 3857 minutes]:
internal/poll.runtime_pollWait(0x7f3cf5ac1690, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0005f8500?, 0x7f3d3cd835c0?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0005f8500)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc0005f8500)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x56292678ed01?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc000525d40)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
net/http.(*Server).Serve(0xc0004a0a00, {0x562926a5ef88, 0xc000525d40})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.12/internal/server/endpoints/endpoints.go:432 +0x1f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000584870, 0xc00045c080?)
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xdb

goroutine 47 [semacquire]:
internal/poll.runtime_Semacquire(0x562924a97c25?)
        /usr/lib/go/src/runtime/sema.go:76 +0x25
internal/poll.(*fdMutex).rwlock(0xc0005f8800, 0x36?)
        /usr/lib/go/src/internal/poll/fd_mutex.go:154 +0xc5
internal/poll.(*FD).readLock(...)
        /usr/lib/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Accept(0xc0005f8800)
        /usr/lib/go/src/internal/poll/fd_unix.go:602 +0x58
net.(*netFD).accept(0xc0005f8800)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc0001210c0)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc0001210c0)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
net/http.(*Server).Serve(0xc0004a0f00, {0x562926a5ef58, 0xc0001210c0})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.12/internal/server/endpoints/endpoints.go:432 +0x1f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000584870, 0x56292787b1f0?)
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x2b
created by gopkg.in/tomb%2ev2.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.12/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xdb

goroutine 48 [select, 3857 minutes]:
database/sql.(*DB).connectionOpener(0xc00051edd0, {0x562926a635c8, 0xc000584a00})
        /usr/lib/go/src/database/sql/sql.go:1261 +0x87
created by database/sql.OpenDB in goroutine 1
        /usr/lib/go/src/database/sql/sql.go:841 +0x130

goroutine 366 [select, 978 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00062fc50, {0x562926a635c8, 0xc000804eb0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x0)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 72187 [sync.Mutex.Lock, 100 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc000305b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000224680?, {0x56292615d2b8?, 0xc000618720?}, {0xc000305f88?, 0x56292690c720?, 0xc000618690?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc00066d320, {0x562926a63590, 0xc0006186f0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72075 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc0008f1b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc002feca40?, {0x56292615d2b8?, 0xc000215170?}, {0xc0008f1f88?, 0x56292690c720?, 0xc000214fc0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0001a4900, {0x562926a63590, 0xc000215140})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72196 [sync.Mutex.Lock, 100 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc000643b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc002aec400?, {0x56292615d2b8?, 0xc0004a8630?}, {0xc000643f88?, 0x56292690c720?, 0xc0004a8480?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000d62d0, {0x562926a63590, 0xc0004a8570})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72086 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc0003eb270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x562926a4fc48?, {0x562926190597?, 0x562926a50d40?}, {0xc0003eb6d8?, 0x562924bcce1b?, 0xc0003eb2b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc00064c780, {0x562926a4fc48, 0xc000659bb0}, {0x562926190597, 0x37}, {0xc0003eb6d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc00064c780, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc00060ebd0, {0x562926a70e70, 0xc000320008}, 0xc0003ebaf8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0004a0900, {0x562926a70e70, 0xc000320008}, {0x562926a503c0, 0xc00083e700}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x56292690c480?, 0xc0003ebea8?, {0x562926a503c0?, 0xc00083e700?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0000d70e0, {0x562926a63590, 0xc002de7080})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 261 [syscall, 3857 minutes]:
syscall.Syscall6(0xf7, 0x3, 0x24, 0xc000689e00, 0x4, 0xc000752510, 0x0)
        /usr/lib/go/src/syscall/syscall_linux.go:95 +0x39
internal/syscall/unix.Waitid(0x200000000000000?, 0x7f3ce7fa0088?, 0x7f3d3cd83108?, 0x10?, 0xc000103008?)
        /usr/lib/go/src/internal/syscall/unix/waitid_linux.go:18 +0x39
os.(*Process).pidfdWait.func1(...)
        /usr/lib/go/src/os/pidfd_linux.go:106
os.ignoringEINTR(...)
        /usr/lib/go/src/os/file_posix.go:251
os.(*Process).pidfdWait(0xc000689f88?)
        /usr/lib/go/src/os/pidfd_linux.go:105 +0x209
os.(*Process).wait(0x0?)
        /usr/lib/go/src/os/exec_unix.go:27 +0x25
os.(*Process).Wait(...)
        /usr/lib/go/src/os/exec.go:358
os/exec.(*Cmd).Wait(0xc000739080)
        /usr/lib/go/src/os/exec/exec.go:922 +0x45
github.com/lxc/incus/v6/shared/subprocess.(*Process).start.func3()
        /build/incus/src/incus-6.12/shared/subprocess/proc.go:204 +0x59
created by github.com/lxc/incus/v6/shared/subprocess.(*Process).start in goroutine 1
        /build/incus/src/incus-6.12/shared/subprocess/proc.go:201 +0x6c6

goroutine 367 [select, 978 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00062fc80, {0x562926a635c8, 0xc000804eb0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x1)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 369 [select, 977 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00062fce0, {0x562926a635c8, 0xc000804eb0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x3)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 342 [IO wait, 3857 minutes]:
internal/poll.runtime_pollWait(0x7f3cf5ac0cb8, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0005f9980?, 0x7f3d3cd83108?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0005f9980)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc0005f9980)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x56292499f8f7?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc000779c20)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
github.com/lxc/incus/v6/internal/server/seccomp.NewSeccompServer.func1()
        /build/incus/src/incus-6.12/internal/server/seccomp/seccomp.go:1083 +0x59
created by github.com/lxc/incus/v6/internal/server/seccomp.NewSeccompServer in goroutine 1
        /build/incus/src/incus-6.12/internal/server/seccomp/seccomp.go:1081 +0x1ed

goroutine 320 [select, 3857 minutes]:
main.deviceEventListener(0xc000810510)
        /build/incus/src/incus-6.12/cmd/incusd/devices.go:623 +0x1dc
created by main.(*Daemon).init in goroutine 1
        /build/incus/src/incus-6.12/cmd/incusd/daemon.go:1583 +0x4165

goroutine 270 [syscall, 3857 minutes]:
syscall.Syscall(0x0, 0x20, 0xc000881000, 0x1000)
        /usr/lib/go/src/syscall/syscall_linux.go:73 +0x25
golang.org/x/sys/unix.read(0x56292678c3e0?, {0xc000881000?, 0x5629260eb566?, 0x6?})
        /build/incus/src/incus-6.12/vendor/golang.org/x/sys/unix/zsyscall_linux.go:1394 +0x45
golang.org/x/sys/unix.Read(...)
        /build/incus/src/incus-6.12/vendor/golang.org/x/sys/unix/syscall_unix.go:167
main.deviceNetlinkListener.func1(0xc000766e70, 0xc000721490, 0xc000721500)
        /build/incus/src/incus-6.12/cmd/incusd/devices.go:69 +0x8d
created by main.deviceNetlinkListener in goroutine 320
        /build/incus/src/incus-6.12/cmd/incusd/devices.go:66 +0x190

goroutine 368 [select, 18 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00062fcb0, {0x562926a635c8, 0xc000804eb0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x2)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 321 [chan receive, 3857 minutes]:
github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load.func1()
        /build/incus/src/incus-6.12/internal/server/fsmonitor/drivers/driver_fanotify.go:66 +0x2c
created by github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load in goroutine 1
        /build/incus/src/incus-6.12/internal/server/fsmonitor/drivers/driver_fanotify.go:65 +0x28c

goroutine 322 [syscall, 3857 minutes]:
syscall.Syscall(0x0, 0x1f, 0xc00010f500, 0x100)
        /usr/lib/go/src/syscall/syscall_linux.go:73 +0x25
golang.org/x/sys/unix.read(0x0?, {0xc00010f500?, 0x0?, 0x0?})
        /build/incus/src/incus-6.12/vendor/golang.org/x/sys/unix/zsyscall_linux.go:1394 +0x45
golang.org/x/sys/unix.Read(...)
        /build/incus/src/incus-6.12/vendor/golang.org/x/sys/unix/syscall_unix.go:167
github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).getEvents(0xc0007a5b80, {0x562926a635c8, 0xc000204e60}, 0x21)
        /build/incus/src/incus-6.12/internal/server/fsmonitor/drivers/driver_fanotify.go:86 +0x78
created by github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load in goroutine 1
        /build/incus/src/incus-6.12/internal/server/fsmonitor/drivers/driver_fanotify.go:71 +0x2fe

goroutine 386 [select, 18 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00062fd10, {0x562926a635c8, 0xc000804eb0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x4)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 387 [select, 2 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00062fd40, {0x562926a635c8, 0xc000804eb0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x5)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 388 [select, 2 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00062fd70, {0x562926a635c8, 0xc000804eb0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x6)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 389 [select, 978 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00062fda0, {0x562926a635c8, 0xc000804eb0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x7)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 390 [select, 978 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00062fdd0, {0x562926a635c8, 0xc000804eb0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x8)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 391 [select, 2 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc00062fe00, {0x562926a635c8, 0xc000804eb0})
        /build/incus/src/incus-6.12/internal/server/task/task.go:68 +0x15f
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x9)
        /build/incus/src/incus-6.12/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.12/internal/server/task/group.go:60 +0x2c5

goroutine 72065 [runnable]:
bytes.indexFunc({0xc0002840e0, 0x71, 0xe0}, 0x562926a45478, 0x0)
        /usr/lib/go/src/bytes/bytes.go:913 +0x111
bytes.TrimLeftFunc(...)
        /usr/lib/go/src/bytes/bytes.go:852
github.com/lxc/incus/v6/internal/server/endpoints.networkServerErrorLogWriter.stripLog({{0x0?, 0x562924a0c53e?, 0xe0?}}, {0xc0002840e0?, 0xc00003b000?, 0x562924c6eccc?})
        /build/incus/src/incus-6.12/internal/server/endpoints/network_util.go:31 +0x93
github.com/lxc/incus/v6/internal/server/endpoints.networkServerErrorLogWriter.Write({{0x0?, 0x70?, 0xe0?}}, {0xc0002840e0?, 0x71, 0xc0002840e0?})
        /build/incus/src/incus-6.12/internal/server/endpoints/network_util.go:19 +0x25
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc00003b270)
        /usr/lib/go/src/log/log.go:245 +0x45a
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x562926a4fc48?, {0x562926190597?, 0x562926a50d40?}, {0xc00003b6d8?, 0x562924bcce1b?, 0xc00003b2b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc000420000, {0x562926a4fc48, 0xc002da0cd0}, {0x562926190597, 0x37}, {0xc00003b6d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc000420000, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc00060ebd0, {0x562926a70e70, 0xc00046b508}, 0xc00003baf8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0004a0900, {0x562926a70e70, 0xc00046b508}, {0x562926a503c0, 0xc002d7a3a0}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x56292690c480?, 0xc00003bea8?, {0x562926a503c0?, 0xc002d7a3a0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0000d6f30, {0x562926a63590, 0xc002de6b70})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72083 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc0003ef270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x562926a4fc48?, {0x562926190597?, 0x562926a50d40?}, {0xc0003ef6d8?, 0x562924bcce1b?, 0xc0003ef2b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc00064c5a0, {0x562926a4fc48, 0xc002da0db0}, {0x562926190597, 0x37}, {0xc0003ef6d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc00064c5a0, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc00060ebd0, {0x562926a70e70, 0xc00046bc08}, 0xc0003efaf8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0004a0900, {0x562926a70e70, 0xc00046bc08}, {0x562926a503c0, 0xc00083e4c0}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x56292690c480?, 0xc0003efea8?, {0x562926a503c0?, 0xc00083e4c0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0000d7050, {0x562926a63590, 0xc002de6cf0})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72077 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc000547270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x562926a4fc48?, {0x562926190597?, 0x562926a50d40?}, {0xc0005476d8?, 0x562924bcce1b?, 0xc0005472b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc00064c960, {0x562926a4fc48, 0xc002da10d0}, {0x562926190597, 0x37}, {0xc0005476d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc00064c960, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc00060ebd0, {0x562926a70e70, 0xc000437888}, 0xc000547af8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0004a0900, {0x562926a70e70, 0xc000437888}, {0x562926a503c0, 0xc00083e820}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x56292690c480?, 0xc000547ea8?, {0x562926a503c0?, 0xc00083e820?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0001a4a20, {0x562926a63590, 0xc00059b5c0})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72185 [sync.Mutex.Lock, 100 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc000647b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000224380?, {0x56292615d2b8?, 0xc0006185a0?}, {0xc000647f88?, 0x56292690c720?, 0xc002de7ef0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0001a5ef0, {0x562926a63590, 0xc002de7f50})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72188 [sync.Mutex.Lock, 100 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc000051b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x5629260e66f7?, {0x56292615d2b8?, 0xc0006187e0?}, {0xc000051f88?, 0x56292690c720?, 0xc000618750?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc00066d3b0, {0x562926a63590, 0xc0006187b0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72112 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc00056d270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x562926a4fc48?, {0x562926190597?, 0x562926a50d40?}, {0xc00056d6d8?, 0x562924bcce1b?, 0xc00056d2b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc00064cb40, {0x562926a4fc48, 0xc002da11c0}, {0x562926190597, 0x37}, {0xc00056d6d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc00064cb40, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc00060ebd0, {0x562926a70e70, 0xc002ba0708}, 0xc00056daf8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0004a0900, {0x562926a70e70, 0xc002ba0708}, {0x562926a503c0, 0xc00083e8e0}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x56292690c480?, 0xc00056dea8?, {0x562926a503c0?, 0xc00083e8e0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0001a5950, {0x562926a63590, 0xc000742fc0})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72189 [sync.Mutex.Lock, 100 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc00070bb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x5629260e66f7?, {0x56292615d2b8?, 0xc000618960?}, {0xc00070bf88?, 0x56292690c720?, 0xc0006188d0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc00066d440, {0x562926a63590, 0xc000618930})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72091 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc00063fb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc002aec440?, {0x56292615d2b8?, 0xc002de78f0?}, {0xc00063ff88?, 0x56292690c720?, 0xc002de7860?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000d7290, {0x562926a63590, 0xc002de78c0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72071 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc000037270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x562926a4fc48?, {0x562926190597?, 0x562926a50d40?}, {0xc0000376d8?, 0x562924bcce1b?, 0xc0000372b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc000420960, {0x562926a4fc48, 0xc000659c30}, {0x562926190597, 0x37}, {0xc0000376d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc000420960, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc00060ebd0, {0x562926a70e70, 0xc0008d9508}, 0xc000037af8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0004a0900, {0x562926a70e70, 0xc0008d9508}, {0x562926a503c0, 0xc002d7a7e0}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x56292690c480?, 0xc000037ea8?, {0x562926a503c0?, 0xc002d7a7e0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0001a47e0, {0x562926a63590, 0xc000619e90})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72076 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc0008edb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc002fecb40?, {0x56292615d2b8?, 0xc000215c80?}, {0xc0008edf88?, 0x56292690c720?, 0xc000215260?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0001a4990, {0x562926a63590, 0xc000215590})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72082 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc002af5b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc00071e180?, {0x56292615d2b8?, 0xc002de6c60?}, {0xc002af5f88?, 0x56292690c720?, 0xc002de6bd0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000d6fc0, {0x562926a63590, 0xc002de6c30})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72090 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc002af1b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc002aec340?, {0x56292615d2b8?, 0xc002de7830?}, {0xc002af1f88?, 0x56292690c720?, 0xc002de7770?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000d7200, {0x562926a63590, 0xc002de7800})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72195 [sync.Mutex.Lock, 100 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc00063db58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc002aec1c0?, {0x56292615d2b8?, 0xc0004a8390?}, {0xc00063df88?, 0x56292690c720?, 0xc0004a8090?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000d6120, {0x562926a63590, 0xc0004a8300})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72186 [sync.Mutex.Lock, 100 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc0008ebb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000224580?, {0x56292615d2b8?, 0xc000618660?}, {0xc0008ebf88?, 0x56292690c720?, 0xc0006185d0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc00066d290, {0x562926a63590, 0xc000618630})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72197 [sync.Mutex.Lock, 100 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc000569b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x5629260e66f7?, {0x56292615d2b8?, 0xc0004a8f00?}, {0xc000569f88?, 0x56292690c720?, 0xc0004a89c0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000d66c0, {0x562926a63590, 0xc0004a8ea0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72094 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc002af3b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc002aeca00?, {0x56292615d2b8?, 0xc002de7bc0?}, {0xc002af3f88?, 0x56292690c720?, 0xc002de7b30?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000d7320, {0x562926a63590, 0xc002de7b90})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72095 [sync.Mutex.Lock, 104 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc000649b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc002aecb40?, {0x56292615d2b8?, 0xc002de7c80?}, {0xc000649f88?, 0x56292690c720?, 0xc002de7bf0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000d73b0, {0x562926a63590, 0xc002de7c50})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 72194 [sync.Mutex.Lock, 100 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc000639b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc002aec0c0?, {0x56292615d2b8?, 0xc0004a8030?}, {0xc000639f88?, 0x56292690c720?, 0xc0000e35c0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000d6090, {0x562926a63590, 0xc0004a8000})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 74053 [IO wait]:
internal/poll.runtime_pollWait(0x7f3ce52834c0, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00014c000?, 0xc000614101?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00014c000, {0xc000614101, 0x1, 0x1})
        /usr/lib/go/src/internal/poll/fd_unix.go:165 +0x27a
net.(*netFD).Read(0xc00014c000, {0xc000614101?, 0xc0007284d8?, 0xc000837f70?})
        /usr/lib/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc00081e000, {0xc000614101?, 0xc000837f01?, 0xc000837fb8?})
        /usr/lib/go/src/net/net.go:194 +0x45
net/http.(*connReader).backgroundRead(0xc0006140f0)
        /usr/lib/go/src/net/http/server.go:690 +0x37
created by net/http.(*connReader).startBackgroundRead in goroutine 74052
        /usr/lib/go/src/net/http/server.go:686 +0xb6

goroutine 72198 [sync.Mutex.Lock, 100 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc000615860)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc000615860, 0x0, 0x2, 0xc00070fb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x5629261934ca?, {0x56292615d2b8?, 0xc0002147b0?}, {0xc00070ff88?, 0x56292690c720?, 0xc000450f30?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000d6a20, {0x562926a63590, 0xc000214660})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 46
        /usr/lib/go/src/net/http/server.go:3454 +0x485

May 29 '25 22:05 C0rn3j

@stgraber Would you need any other additional data? I'm sorry to poke you - but it is really consistently, repeatedly overloading the entire server with 100% CPU usage.

Jun 13 '25 10:06 patschi

Saw the other CPU usage thread being resolved with 6.15 kernel, this one unfortunately isn't, happened again fairly quickly on 6.15.2-arch1-1:

log

# curl http://127.0.0.1:8444/debug/pprof/goroutine\?debug\=2

goroutine 17684 [running]:
runtime/pprof.writeGoroutineStacks({0x55970377bb20, 0xc000a7c700})
        /usr/lib/go/src/runtime/pprof/pprof.go:764 +0x6a
runtime/pprof.writeGoroutine({0x55970377bb20?, 0xc000a7c700?}, 0x1e?)
        /usr/lib/go/src/runtime/pprof/pprof.go:753 +0x25
runtime/pprof.(*Profile).WriteTo(0x55970466e3e0?, {0x55970377bb20?, 0xc000a7c700?}, 0xc?)
        /usr/lib/go/src/runtime/pprof/pprof.go:377 +0x14b
net/http/pprof.handler.ServeHTTP({0xc000676c71, 0x9}, {0x55970378b1c8, 0xc000a7c700}, 0xc00047be00)
        /usr/lib/go/src/net/http/pprof/pprof.go:272 +0x52f
net/http/pprof.Index({0x55970378b1c8, 0xc000a7c700}, 0xc00047be00?)
        /usr/lib/go/src/net/http/pprof/pprof.go:389 +0xde
net/http.HandlerFunc.ServeHTTP(0x55970477f600?, {0x55970378b1c8?, 0xc000a7c700?}, 0x559701823d56?)
        /usr/lib/go/src/net/http/server.go:2294 +0x29
net/http.(*ServeMux).ServeHTTP(0x559701586cf9?, {0x55970378b1c8, 0xc000a7c700}, 0xc00047be00)
        /usr/lib/go/src/net/http/server.go:2822 +0x1c4
net/http.serverHandler.ServeHTTP({0xc0006bbb30?}, {0x55970378b1c8?, 0xc000a7c700?}, 0x1?)
        /usr/lib/go/src/net/http/server.go:3301 +0x8e
net/http.(*conn).serve(0xc000194b40, {0x55970378f5c8, 0xc0007003c0})
        /usr/lib/go/src/net/http/server.go:2102 +0x625
created by net/http.(*Server).Serve in goroutine 57
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 1 [select, 923 minutes]:
main.(*cmdDaemon).run(0xc000010d50, 0xc000569a38?, {0xc0005927a0, 0x0, 0x0?})
        /build/incus/src/incus-6.13/cmd/incusd/main_daemon.go:81 +0x5b7
github.com/spf13/cobra.(*Command).execute(0xc0003b4608, {0xc000112400, 0x2, 0x2})
        /build/incus/src/incus-6.13/vendor/github.com/spf13/cobra/command.go:1015 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc0003b4608)
        /build/incus/src/incus-6.13/vendor/github.com/spf13/cobra/command.go:1148 +0x46f
github.com/spf13/cobra.(*Command).Execute(...)
        /build/incus/src/incus-6.13/vendor/github.com/spf13/cobra/command.go:1071
main.main()
        /build/incus/src/incus-6.13/cmd/incusd/main.go:201 +0x17bc

goroutine 227 [syscall, 924 minutes]:
syscall.Syscall6(0xf7, 0x3, 0x1f, 0xc000066e00, 0x4, 0xc00073ca20, 0x0)
        /usr/lib/go/src/syscall/syscall_linux.go:95 +0x39
internal/syscall/unix.Waitid(0x200000000000060?, 0x7f09c6614c18?, 0x7f09c68a0108?, 0x10?, 0xc000071808?)
        /usr/lib/go/src/internal/syscall/unix/waitid_linux.go:18 +0x39
os.(*Process).pidfdWait.func1(...)
        /usr/lib/go/src/os/pidfd_linux.go:106
os.ignoringEINTR(...)
        /usr/lib/go/src/os/file_posix.go:251
os.(*Process).pidfdWait(0xc000066f88?)
        /usr/lib/go/src/os/pidfd_linux.go:105 +0x209
os.(*Process).wait(0x55970465c520?)
        /usr/lib/go/src/os/exec_unix.go:27 +0x25
os.(*Process).Wait(...)
        /usr/lib/go/src/os/exec.go:358
os/exec.(*Cmd).Wait(0xc000745800)
        /usr/lib/go/src/os/exec/exec.go:922 +0x45
github.com/lxc/incus/v6/shared/subprocess.(*Process).start.func3()
        /build/incus/src/incus-6.13/shared/subprocess/proc.go:205 +0x59
created by github.com/lxc/incus/v6/shared/subprocess.(*Process).start in goroutine 1
        /build/incus/src/incus-6.13/shared/subprocess/proc.go:202 +0x6c6

goroutine 7 [syscall, 924 minutes]:
os/signal.signal_recv()
        /usr/lib/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
        /usr/lib/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/lib/go/src/os/signal/signal.go:152 +0x1f

goroutine 8 [select, 924 minutes]:
database/sql.(*DB).connectionOpener(0xc0005b95f0, {0x55970378f600, 0xc000588410})
        /usr/lib/go/src/database/sql/sql.go:1261 +0x87
created by database/sql.OpenDB in goroutine 1
        /usr/lib/go/src/database/sql/sql.go:841 +0x130

goroutine 292 [select, 923 minutes]:
main.deviceEventListener(0xc000781bb0)
        /build/incus/src/incus-6.13/cmd/incusd/devices.go:623 +0x1dc
created by main.(*Daemon).init in goroutine 1
        /build/incus/src/incus-6.13/cmd/incusd/daemon.go:1582 +0x4165

goroutine 293 [chan receive, 924 minutes]:
github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load.func1()
        /build/incus/src/incus-6.13/internal/server/fsmonitor/drivers/driver_fanotify.go:66 +0x2c
created by github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load in goroutine 1
        /build/incus/src/incus-6.13/internal/server/fsmonitor/drivers/driver_fanotify.go:65 +0x28c

goroutine 294 [syscall, 924 minutes]:
syscall.Syscall(0x0, 0x1c, 0xc00010ec00, 0x100)
        /usr/lib/go/src/syscall/syscall_linux.go:73 +0x25
golang.org/x/sys/unix.read(0x559701539a3c?, {0xc00010ec00?, 0x540010?, 0xc0005dac90?})
        /build/incus/src/incus-6.13/vendor/golang.org/x/sys/unix/zsyscall_linux.go:1394 +0x45
golang.org/x/sys/unix.Read(...)
        /build/incus/src/incus-6.13/vendor/golang.org/x/sys/unix/syscall_unix.go:167
github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).getEvents(0xc000759780, {0x55970378f600, 0xc000588370}, 0x1d)
        /build/incus/src/incus-6.13/internal/server/fsmonitor/drivers/driver_fanotify.go:86 +0x78
created by github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load in goroutine 1
        /build/incus/src/incus-6.13/internal/server/fsmonitor/drivers/driver_fanotify.go:71 +0x2fe

goroutine 255 [syscall, 923 minutes]:
syscall.Syscall(0x0, 0x1e, 0xc000801000, 0x1000)
        /usr/lib/go/src/syscall/syscall_linux.go:73 +0x25
golang.org/x/sys/unix.read(0x55970349c720?, {0xc000801000?, 0x559702dae5fe?, 0x6?})
        /build/incus/src/incus-6.13/vendor/golang.org/x/sys/unix/zsyscall_linux.go:1394 +0x45
golang.org/x/sys/unix.Read(...)
        /build/incus/src/incus-6.13/vendor/golang.org/x/sys/unix/syscall_unix.go:167
main.deviceNetlinkListener.func1(0xc0006aff10, 0xc0006cdf80, 0xc000812000)
        /build/incus/src/incus-6.13/cmd/incusd/devices.go:69 +0x8d
created by main.deviceNetlinkListener in goroutine 292
        /build/incus/src/incus-6.13/cmd/incusd/devices.go:66 +0x190

goroutine 359 [select, 923 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc0007391d0, {0x55970378f600, 0xc0007a70e0})
        /build/incus/src/incus-6.13/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x3)
        /build/incus/src/incus-6.13/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.13/internal/server/task/group.go:60 +0x2c5

goroutine 11286 [sync.Mutex.Lock, 343 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000787b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc00071c140?, {0x559702e23970?, 0xc000014120?}, {0xc000787f88?, 0x55970362a760?, 0xc000014030?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000152090, {0x55970378f5c8, 0xc0000140c0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 57 [IO wait]:
internal/poll.runtime_pollWait(0x7f097e5c9b58, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000565680?, 0x900000036?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000565680)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc000565680)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000234dc0)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc000234dc0)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
net/http.(*Server).Serve(0xc0005a1a00, {0x55970378acb8, 0xc000234dc0})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:431 +0x1f
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go.func1()
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:529 +0x50
created by github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:526 +0x73

goroutine 58 [IO wait]:
internal/poll.runtime_pollWait(0x7f097e5c9ea0, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000565300?, 0xc000828000?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000565300)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc000565300)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x7f097e5bd968?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc0005e68a0)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
github.com/lxc/incus/v6/internal/server/endpoints/listeners.(*StarttlsListener).Accept(0xc0005e6900)
        /build/incus/src/incus-6.13/internal/server/endpoints/listeners/starttls.go:36 +0x4a
net/http.(*Server).Serve(0xc0005a1400, {0x55970378b648, 0xc0005e6900})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:431 +0x1f
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go.func1()
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:529 +0x50
created by github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:526 +0x73

goroutine 59 [IO wait, 924 minutes]:
internal/poll.runtime_pollWait(0x7f097e5c9d88, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000565400?, 0x7f09c68a0108?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000565400)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc000565400)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x55970349f101?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc0005e6960)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
net/http.(*Server).Serve(0xc0005a1500, {0x55970378ace8, 0xc0005e6960})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:431 +0x1f
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go.func1()
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:529 +0x50
created by github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:526 +0x73

goroutine 60 [IO wait, 21 minutes]:
internal/poll.runtime_pollWait(0x7f097e5c9c70, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc000565580?, 0xc0007bc008?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc000565580)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc000565580)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000234d80)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc000234d80)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
github.com/lxc/incus/v6/internal/server/endpoints/listeners.(*FancyTLSListener).Accept(0xc0003ab130)
        /build/incus/src/incus-6.13/internal/server/endpoints/listeners/fancytls.go:38 +0x4a
net/http.(*Server).Serve(0xc0005a1400, {0x55970378b5e8, 0xc0003ab130})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:431 +0x1f
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go.func1()
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:529 +0x50
created by github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:526 +0x73

goroutine 61 [runnable]:
internal/poll.runtime_Semacquire(0x0?)
        /usr/lib/go/src/runtime/sema.go:76 +0x25
internal/poll.(*fdMutex).rwlock(0xc000565680, 0x0?)
        /usr/lib/go/src/internal/poll/fd_mutex.go:154 +0xc5
internal/poll.(*FD).readLock(...)
        /usr/lib/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Accept(0xc000565680)
        /usr/lib/go/src/internal/poll/fd_unix.go:602 +0x58
net.(*netFD).accept(0xc000565680)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000234dc0)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc000234dc0)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
net/http.(*Server).Serve(0xc0005a1a00, {0x55970378acb8, 0xc000234dc0})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:431 +0x1f
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go.func1()
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:529 +0x50
created by github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.13/internal/server/endpoints/endpoints.go:526 +0x73

goroutine 62 [select, 924 minutes]:
database/sql.(*DB).connectionOpener(0xc0005668f0, {0x55970378f600, 0xc0003ab310})
        /usr/lib/go/src/database/sql/sql.go:1261 +0x87
created by database/sql.OpenDB in goroutine 1
        /usr/lib/go/src/database/sql/sql.go:841 +0x130

goroutine 12611 [sync.Mutex.Lock, 273 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc0007f7b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000315280?, {0x559702e23970?, 0xc0005b0d20?}, {0xc0007f7f88?, 0x55970362a760?, 0xc0005b0c60?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000708510, {0x55970378f5c8, 0xc0005b0cf0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 9755 [sync.Mutex.Lock, 423 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000633b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d4480?, {0x559702e23970?, 0xc00059ebd0?}, {0xc000633f88?, 0x55970362a760?, 0xc000579f20?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000152e10, {0x55970378f5c8, 0xc000112990})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 356 [select, 924 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000739140, {0x55970378f600, 0xc0007a70e0})
        /build/incus/src/incus-6.13/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x0)
        /build/incus/src/incus-6.13/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.13/internal/server/task/group.go:60 +0x2c5

goroutine 9753 [runnable]:
bytes.indexFunc({0xc000640700, 0x70, 0x70}, 0x5597037704d8, 0x0)
        /usr/lib/go/src/bytes/bytes.go:913 +0xa
bytes.TrimLeftFunc(...)
        /usr/lib/go/src/bytes/bytes.go:852
github.com/lxc/incus/v6/internal/server/endpoints.networkServerErrorLogWriter.stripLog({{0x0?, 0x70?, 0xc000114a90?}}, {0xc000640700?, 0xc00003b080?, 0x5597017f1c8c?})
        /build/incus/src/incus-6.13/internal/server/endpoints/network_util.go:31 +0x93
github.com/lxc/incus/v6/internal/server/endpoints.networkServerErrorLogWriter.Write({{0x0?, 0x559702e595b0?, 0x37?}}, {0xc000640700?, 0x70, 0x2?})
        /build/incus/src/incus-6.13/internal/server/endpoints/network_util.go:19 +0x25
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc00003b270)
        /usr/lib/go/src/log/log.go:245 +0x45a
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x55970377b308?, {0x559702e595b0?, 0x55970377c3e0?}, {0xc00003b6d8?, 0x55970174fc7b?, 0xc00003b2b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc0002d0d20, {0x55970377b308, 0xc000238bb0}, {0x559702e595b0, 0x37}, {0xc00003b6d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc0002d0d20, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc000702000, {0x55970379da10, 0xc00051ea88}, 0xc00003baf8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0005a1400, {0x55970379da10, 0xc00051ea88}, {0x55970377ba80, 0xc0007684a0}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x55970362a4c0?, 0xc00003bea8?, {0x55970377ba80?, 0xc0007684a0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc000152cf0, {0x55970378f5c8, 0xc000579c80})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 11116 [sync.Mutex.Lock, 353 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc0006d7b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc00071c080?, {0x559702e23970?, 0xc000578120?}, {0xc0006d7f88?, 0x55970362a760?, 0xc000578030?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000708090, {0x55970378f5c8, 0xc0005780c0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 357 [select, 924 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000739170, {0x55970378f600, 0xc0007a70e0})
        /build/incus/src/incus-6.13/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x1)
        /build/incus/src/incus-6.13/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.13/internal/server/task/group.go:60 +0x2c5

goroutine 9758 [sync.Mutex.Lock, 423 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc0006e5270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x55970377b308?, {0x559702e595b0?, 0x55970377c3e0?}, {0xc0006e56d8?, 0x55970174fc7b?, 0xc0006e52b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc0007045a0, {0x55970377b308, 0xc0001a55c0}, {0x559702e595b0, 0x37}, {0xc0006e56d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc0007045a0, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc000702000, {0x55970379da10, 0xc000833188}, 0xc0006e5af8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0005a1400, {0x55970379da10, 0xc000833188}, {0x55970377ba80, 0xc0009565c0}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x55970362a4c0?, 0xc0006e5ea8?, {0x55970377ba80?, 0xc0009565c0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc000152fc0, {0x55970378f5c8, 0xc000555b30})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 9754 [sync.Mutex.Lock, 423 minutes]:
internal/sync.runtime_SemacquireMutex(0xc0001e00e0?, 0x0?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000635b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d4380?, {0x559702e23970?, 0xc000579ec0?}, {0xc000635f88?, 0x55970362a760?, 0xc000579d10?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000152d80, {0x55970378f5c8, 0xc000579e00})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 9764 [sync.Mutex.Lock, 423 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000715270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x55970377b308?, {0x559702e595b0?, 0x55970377c3e0?}, {0xc0007156d8?, 0x55970174fc7b?, 0xc0007152b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc000704960, {0x55970377b308, 0xc0001a5740}, {0x559702e595b0, 0x37}, {0xc0007156d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc000704960, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc000702000, {0x55970379da10, 0xc000833888}, 0xc000715af8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0005a1400, {0x55970379da10, 0xc000833888}, {0x55970377ba80, 0xc000956aa0}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x55970362a4c0?, 0xc000715ea8?, {0x55970377ba80?, 0xc000956aa0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0001530e0, {0x55970378f5c8, 0xc0005567e0})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 9757 [sync.Mutex.Lock, 423 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc00062fb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d4680?, {0x559702e23970?, 0xc000555860?}, {0xc00062ff88?, 0x55970362a760?, 0xc0005556b0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000152f30, {0x55970378f5c8, 0xc000555830})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 12602 [sync.Mutex.Lock, 273 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000871b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x559702da9709?, {0x559702e23970?, 0xc000701140?}, {0xc000871f88?, 0x55970362a760?, 0xc0007010b0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000153830, {0x55970378f5c8, 0xc000701110})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 9741 [sync.Mutex.Lock, 423 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000867b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000314a80?, {0x559702e23970?, 0xc0005b17a0?}, {0xc000867f88?, 0x55970362a760?, 0xc0005b1710?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000e93b0, {0x55970378f5c8, 0xc0005b1770})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 9743 [sync.Mutex.Lock, 423 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc00078bb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0003152c0?, {0x559702e23970?, 0xc0005b1a40?}, {0xc00078bf88?, 0x55970362a760?, 0xc0005b19b0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000e94d0, {0x55970378f5c8, 0xc0005b1a10})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 15293 [sync.Mutex.Lock, 125 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc00087db58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000680180?, {0x559702e23970?, 0xc0005b0600?}, {0xc00087df88?, 0x55970362a760?, 0xc0005b0000?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000708120, {0x55970378f5c8, 0xc0005b05d0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 16667 [sync.Mutex.Lock, 51 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc0006d9b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0006802c0?, {0x559702e23970?, 0xc000700210?}, {0xc0006d9f88?, 0x55970362a760?, 0xc000700150?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000e8c60, {0x55970378f5c8, 0xc0007001e0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 12599 [sync.Mutex.Lock, 273 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000037b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x559702da9709?, {0x559702e23970?, 0xc000700ae0?}, {0xc000037f88?, 0x55970362a760?, 0xc000700a50?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000153680, {0x55970378f5c8, 0xc000700ab0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 361 [select]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000739230, {0x55970378f600, 0xc0007a70e0})
        /build/incus/src/incus-6.13/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x5)
        /build/incus/src/incus-6.13/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.13/internal/server/task/group.go:60 +0x2c5

goroutine 358 [select, 25 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc0007391a0, {0x55970378f600, 0xc0007a70e0})
        /build/incus/src/incus-6.13/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x2)
        /build/incus/src/incus-6.13/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.13/internal/server/task/group.go:60 +0x2c5

goroutine 306 [IO wait, 924 minutes]:
internal/poll.runtime_pollWait(0x7f097e5c95e0, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00070b380?, 0x55970159d660?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00070b380)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc00070b380)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x559701520917?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc0007b4900)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
github.com/lxc/incus/v6/internal/server/seccomp.NewSeccompServer.func1()
        /build/incus/src/incus-6.13/internal/server/seccomp/seccomp.go:1084 +0x59
created by github.com/lxc/incus/v6/internal/server/seccomp.NewSeccompServer in goroutine 1
        /build/incus/src/incus-6.13/internal/server/seccomp/seccomp.go:1082 +0x1d5

goroutine 360 [select, 25 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000739200, {0x55970378f600, 0xc0007a70e0})
        /build/incus/src/incus-6.13/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x4)
        /build/incus/src/incus-6.13/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.13/internal/server/task/group.go:60 +0x2c5

goroutine 362 [select]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000739260, {0x55970378f600, 0xc0007a70e0})
        /build/incus/src/incus-6.13/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x6)
        /build/incus/src/incus-6.13/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.13/internal/server/task/group.go:60 +0x2c5

goroutine 363 [select, 924 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000739290, {0x55970378f600, 0xc0007a70e0})
        /build/incus/src/incus-6.13/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x7)
        /build/incus/src/incus-6.13/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.13/internal/server/task/group.go:60 +0x2c5

goroutine 364 [select, 924 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc0007392c0, {0x55970378f600, 0xc0007a70e0})
        /build/incus/src/incus-6.13/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x8)
        /build/incus/src/incus-6.13/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.13/internal/server/task/group.go:60 +0x2c5

goroutine 365 [select]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc0007392f0, {0x55970378f600, 0xc0007a70e0})
        /build/incus/src/incus-6.13/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x9)
        /build/incus/src/incus-6.13/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.13/internal/server/task/group.go:60 +0x2c5

goroutine 11323 [sync.Mutex.Lock, 343 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000785b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0003149c0?, {0x559702e23970?, 0xc000701950?}, {0xc000785f88?, 0x55970362a760?, 0xc000701890?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000708f30, {0x55970378f5c8, 0xc000701920})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 12612 [sync.Mutex.Lock, 273 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc0007f9b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000315500?, {0x559702e23970?, 0xc0005b0de0?}, {0xc0007f9f88?, 0x55970362a760?, 0xc0005b0d50?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0007085a0, {0x55970378f5c8, 0xc0005b0db0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 17011 [sync.Mutex.Lock, 33 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000839b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000680440?, {0x559702e23970?, 0xc0007008a0?}, {0xc000839f88?, 0x55970362a760?, 0xc000700810?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0001538c0, {0x55970378f5c8, 0xc000700870})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 12600 [sync.Mutex.Lock, 273 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc00083db58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x559702da9709?, {0x559702e23970?, 0xc000700f00?}, {0xc00083df88?, 0x55970362a760?, 0xc000700e70?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000153710, {0x55970378f5c8, 0xc000700ed0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 9742 [sync.Mutex.Lock, 423 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000631b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000315180?, {0x559702e23970?, 0xc0005b1980?}, {0xc000631f88?, 0x55970362a760?, 0xc0005b18f0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000e9440, {0x55970378f5c8, 0xc0005b1950})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 9756 [sync.Mutex.Lock, 423 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc0006ddb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d4580?, {0x559702e23970?, 0xc000555170?}, {0xc0006ddf88?, 0x55970362a760?, 0xc00059f0b0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000152ea0, {0x55970378f5c8, 0xc000554e40})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 12592 [sync.Mutex.Lock, 273 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000869b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000681c00?, {0x559702e23970?, 0xc0005b0930?}, {0xc000869f88?, 0x55970362a760?, 0xc0005b0840?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000708360, {0x55970378f5c8, 0xc0005b08d0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 12593 [sync.Mutex.Lock, 273 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000803b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000681e80?, {0x559702e23970?, 0xc0005b0ae0?}, {0xc000803f88?, 0x55970362a760?, 0xc0005b0a50?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0007083f0, {0x55970378f5c8, 0xc0005b0ab0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 17242 [sync.Mutex.Lock, 21 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000837b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000680500?, {0x559702e23970?, 0xc000701cb0?}, {0xc000837f88?, 0x55970362a760?, 0xc000701c20?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000153950, {0x55970378f5c8, 0xc000701c80})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 12591 [sync.Mutex.Lock, 273 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000807b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000681b00?, {0x559702e23970?, 0xc0005b07e0?}, {0xc000807f88?, 0x55970362a760?, 0xc0005b0750?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0007081b0, {0x55970378f5c8, 0xc0005b07b0})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 12610 [sync.Mutex.Lock, 273 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000805b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc000681f80?, {0x559702e23970?, 0xc0005b0bd0?}, {0xc000805f88?, 0x55970362a760?, 0xc0005b0b10?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000708480, {0x55970378f5c8, 0xc0005b0b70})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 9761 [sync.Mutex.Lock, 423 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc0006e1270)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x55970377b308?, {0x559702e595b0?, 0x55970377c3e0?}, {0xc0006e16d8?, 0x55970174fc7b?, 0xc0006e12b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc000704780, {0x55970377b308, 0xc0001a5680}, {0x559702e595b0, 0x37}, {0xc0006e16d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc000704780, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc000702000, {0x55970379da10, 0xc000833508}, 0xc0006e1af8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0005a1400, {0x55970379da10, 0xc000833508}, {0x55970377ba80, 0xc000956800}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x55970362a4c0?, 0xc0006e1ea8?, {0x55970377ba80?, 0xc000956800?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc000153050, {0x55970378f5c8, 0xc00059bda0})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 12601 [sync.Mutex.Lock, 273 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000839b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x559702e5c7b5?, {0x559702e23970?, 0xc000701080?}, {0xc000839f88?, 0x55970362a760?, 0xc000700ff0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0001537a0, {0x55970378f5c8, 0xc000701050})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 11947 [sync.Mutex.Lock, 309 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc000809b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc00071c200?, {0x559702e23970?, 0xc000701980?}, {0xc000809f88?, 0x55970362a760?, 0xc000701860?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc000708fc0, {0x55970378f5c8, 0xc000701950})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 16493 [sync.Mutex.Lock, 61 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc0006bb5f0)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc0006bb5f0, 0x0, 0x2, 0xc00087bb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008d41c0?, {0x559702e23970?, 0xc000700840?}, {0xc00087bf88?, 0x55970362a760?, 0xc0007007b0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0001535f0, {0x55970378f5c8, 0xc000700810})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 17685 [runnable]:
net/http.(*connReader).startBackgroundRead.gowrap2()
        /usr/lib/go/src/net/http/server.go:686
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1
created by net/http.(*connReader).startBackgroundRead in goroutine 17684
        /usr/lib/go/src/net/http/server.go:686 +0xb6

Jun 20 '25 07:06 C0rn3j

Still the case with 6.14, new log:

6.15.5 kernel and 6.14 incus

[0] # curl http://127.0.0.1:8444/debug/pprof/goroutine\?debug\=2

goroutine 354486 [running]:
runtime/pprof.writeGoroutineStacks({0x560497cc33e0, 0xc000700000})
        /usr/lib/go/src/runtime/pprof/pprof.go:764 +0x6a
runtime/pprof.writeGoroutine({0x560497cc33e0?, 0xc000700000?}, 0x15?)
        /usr/lib/go/src/runtime/pprof/pprof.go:753 +0x25
runtime/pprof.(*Profile).WriteTo(0x560498bc24e0?, {0x560497cc33e0?, 0xc000700000?}, 0xc?)
        /usr/lib/go/src/runtime/pprof/pprof.go:377 +0x14b
net/http/pprof.handler.ServeHTTP({0xc000adc191, 0x9}, {0x560497cd2ba0, 0xc000700000}, 0xc000450280)
        /usr/lib/go/src/net/http/pprof/pprof.go:272 +0x52f
net/http/pprof.Index({0x560497cd2ba0, 0xc000700000}, 0xc000450280?)
        /usr/lib/go/src/net/http/pprof/pprof.go:389 +0xde
net/http.HandlerFunc.ServeHTTP(0x560498cd4060?, {0x560497cd2ba0?, 0xc000700000?}, 0x560495d4acf6?)
        /usr/lib/go/src/net/http/server.go:2294 +0x29
net/http.(*ServeMux).ServeHTTP(0x560495aac679?, {0x560497cd2ba0, 0xc000700000}, 0xc000450280)
        /usr/lib/go/src/net/http/server.go:2822 +0x1c4
net/http.serverHandler.ServeHTTP({0xc0006c1050?}, {0x560497cd2ba0?, 0xc000700000?}, 0x1?)
        /usr/lib/go/src/net/http/server.go:3301 +0x8e
net/http.(*conn).serve(0xc0000ef680, {0x560497cd72a8, 0xc0004d4210})
        /usr/lib/go/src/net/http/server.go:2102 +0x625
created by net/http.(*Server).Serve in goroutine 57
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 1 [select, 19138 minutes]:
main.(*cmdDaemon).run(0xc000011ef0, 0xc00055da38?, {0xc0005e6400, 0x0, 0x0?})
        /build/incus/src/incus-6.14/cmd/incusd/main_daemon.go:81 +0x5b7
github.com/spf13/cobra.(*Command).execute(0xc000221208, {0xc00003e4f0, 0x2, 0x2})
        /build/incus/src/incus-6.14/vendor/github.com/spf13/cobra/command.go:1015 +0xaaa
github.com/spf13/cobra.(*Command).ExecuteC(0xc000221208)
        /build/incus/src/incus-6.14/vendor/github.com/spf13/cobra/command.go:1148 +0x46f
github.com/spf13/cobra.(*Command).Execute(...)
        /build/incus/src/incus-6.14/vendor/github.com/spf13/cobra/command.go:1071
main.main()
        /build/incus/src/incus-6.14/cmd/incusd/main.go:201 +0x17bc

goroutine 9 [syscall, 19138 minutes]:
os/signal.signal_recv()
        /usr/lib/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
        /usr/lib/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/lib/go/src/os/signal/signal.go:152 +0x1f

goroutine 12 [select, 19138 minutes]:
database/sql.(*DB).connectionOpener(0xc00060a0d0, {0x560497cd72e0, 0xc0005c4550})
        /usr/lib/go/src/database/sql/sql.go:1261 +0x87
created by database/sql.OpenDB in goroutine 1
        /usr/lib/go/src/database/sql/sql.go:841 +0x130

goroutine 353886 [runnable]:
bytes.indexFunc({0xc0009da310, 0x70, 0x70}, 0x560497cb7bb8, 0x0)
        /usr/lib/go/src/bytes/bytes.go:913 +0x111
bytes.TrimLeftFunc(...)
        /usr/lib/go/src/bytes/bytes.go:852
github.com/lxc/incus/v6/internal/server/endpoints.networkServerErrorLogWriter.stripLog({{0x0?, 0x70?, 0xc0007ae340?}}, {0xc0009da310?, 0xc000449080?, 0x560495d18c2c?})
        /build/incus/src/incus-6.14/internal/server/endpoints/network_util.go:31 +0x93
github.com/lxc/incus/v6/internal/server/endpoints.networkServerErrorLogWriter.Write({{0x0?, 0x56049739ad4e?, 0x37?}}, {0xc0009da310?, 0x70, 0x2?})
        /build/incus/src/incus-6.14/internal/server/endpoints/network_util.go:19 +0x25
log.(*Logger).output(0xc00074ec30, 0x0, 0x2, 0xc000449270)
        /usr/lib/go/src/log/log.go:245 +0x45a
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*http2serverConn).logf(0x560497cc2bc8?, {0x56049739ad4e?, 0x560497cc3ca0?}, {0xc0004496d8?, 0x560495c758bb?, 0xc0004492b0?})
        /usr/lib/go/src/net/http/h2_bundle.go:4799 +0x72
net/http.(*http2serverConn).condlogf(0xc00024c960, {0x560497cc2bc8, 0xc0006fc050}, {0x56049739ad4e, 0x37}, {0xc0004496d8, 0x2, 0x2})
        /usr/lib/go/src/net/http/h2_bundle.go:4853 +0x1fd
net/http.(*http2serverConn).serve(0xc00024c960, {0xfa, 0x1000, 0x1000, 0x100000, 0x100000, 0x100000, 0x0, 0x37e11d600, 0x0, ...})
        /usr/lib/go/src/net/http/h2_bundle.go:5008 +0x7b5
net/http.(*http2Server).serveConn(0xc0004d60e0, {0x560497ce5810, 0xc0006d0008}, 0xc000449af8, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4614 +0xc19
net/http.(*http2Server).ServeConn(...)
        /usr/lib/go/src/net/http/h2_bundle.go:4473
net/http.http2ConfigureServer.func1(0xc0005d9400, {0x560497ce5810, 0xc0006d0008}, {0x560497cc3340, 0xc0005e6640}, 0x0)
        /usr/lib/go/src/net/http/h2_bundle.go:4373 +0x165
net/http.http2ConfigureServer.func2(0x560497b70240?, 0xc000449ea8?, {0x560497cc3340?, 0xc0005e6640?})
        /usr/lib/go/src/net/http/h2_bundle.go:4381 +0x35
net/http.(*conn).serve(0xc0003ea000, {0x560497cd72a8, 0xc0006c0060})
        /usr/lib/go/src/net/http/server.go:2000 +0x1253
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 298 [syscall, 16669 minutes]:
syscall.Syscall(0x0, 0x1c, 0xc00091b600, 0x100)
        /usr/lib/go/src/syscall/syscall_linux.go:73 +0x25
golang.org/x/sys/unix.read(0xc00028f4c0?, {0xc00091b600?, 0xc000865f30?, 0x56049771e6b8?})
        /build/incus/src/incus-6.14/vendor/golang.org/x/sys/unix/zsyscall_linux.go:1394 +0x45
golang.org/x/sys/unix.Read(...)
        /build/incus/src/incus-6.14/vendor/golang.org/x/sys/unix/syscall_unix.go:167
github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).getEvents(0xc000836340, {0x560497cd72e0, 0xc0005c4410}, 0x1e)
        /build/incus/src/incus-6.14/internal/server/fsmonitor/drivers/driver_fanotify.go:86 +0x78
created by github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load in goroutine 1
        /build/incus/src/incus-6.14/internal/server/fsmonitor/drivers/driver_fanotify.go:71 +0x2fe

goroutine 297 [chan receive, 19138 minutes]:
github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load.func1()
        /build/incus/src/incus-6.14/internal/server/fsmonitor/drivers/driver_fanotify.go:66 +0x2c
created by github.com/lxc/incus/v6/internal/server/fsmonitor/drivers.(*fanotify).load in goroutine 1
        /build/incus/src/incus-6.14/internal/server/fsmonitor/drivers/driver_fanotify.go:65 +0x28c

goroutine 251 [syscall, 16669 minutes]:
syscall.Syscall(0x0, 0x1d, 0xc0006d9000, 0x1000)
        /usr/lib/go/src/syscall/syscall_linux.go:73 +0x25
golang.org/x/sys/unix.read(0x5604979e21e0?, {0xc0006d9000?, 0x5604972ef604?, 0x6?})
        /build/incus/src/incus-6.14/vendor/golang.org/x/sys/unix/zsyscall_linux.go:1394 +0x45
golang.org/x/sys/unix.Read(...)
        /build/incus/src/incus-6.14/vendor/golang.org/x/sys/unix/syscall_unix.go:167
main.deviceNetlinkListener.func1(0xc000791d50, 0xc0006a1570, 0xc0006a15e0)
        /build/incus/src/incus-6.14/cmd/incusd/devices.go:69 +0x8d
created by main.deviceNetlinkListener in goroutine 296
        /build/incus/src/incus-6.14/cmd/incusd/devices.go:66 +0x190

goroutine 348 [select, 420 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000655788, {0x560497cd72e0, 0xc0007bc910})
        /build/incus/src/incus-6.14/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x0)
        /build/incus/src/incus-6.14/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.14/internal/server/task/group.go:60 +0x2c5

goroutine 57 [IO wait]:
internal/poll.runtime_pollWait(0x7f0678fb0b68, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00051bb80?, 0x900000036?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00051bb80)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc00051bb80)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000536540)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc000536540)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
net/http.(*Server).Serve(0xc000462600, {0x560497cd2690, 0xc000536540})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:431 +0x1f
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go.func1()
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:529 +0x50
created by github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:526 +0x73

goroutine 58 [IO wait, 3 minutes]:
internal/poll.runtime_pollWait(0x7f0678fb0eb0, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00051b800?, 0xc00081a000?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00051b800)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc00051b800)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x10?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc0006926c0)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
github.com/lxc/incus/v6/internal/server/endpoints/listeners.(*StarttlsListener).Accept(0xc000692720)
        /build/incus/src/incus-6.14/internal/server/endpoints/listeners/starttls.go:36 +0x4a
net/http.(*Server).Serve(0xc0005d9400, {0x560497cd3020, 0xc000692720})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:431 +0x1f
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go.func1()
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:529 +0x50
created by github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:526 +0x73

goroutine 59 [IO wait, 19138 minutes]:
internal/poll.runtime_pollWait(0x7f0678fb0d98, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00051b900?, 0xc000083c70?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00051b900)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc00051b900)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x5604979e4c01?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc000692780)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
net/http.(*Server).Serve(0xc0005d9e00, {0x560497cd26c0, 0xc000692780})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:431 +0x1f
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go.func1()
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:529 +0x50
created by github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:526 +0x73

goroutine 60 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7f0678fb0c80, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc00051ba80?, 0xc0005dc808?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00051ba80)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc00051ba80)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc0005364c0)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc0005364c0)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
github.com/lxc/incus/v6/internal/server/endpoints/listeners.(*FancyTLSListener).Accept(0xc000455310)
        /build/incus/src/incus-6.14/internal/server/endpoints/listeners/fancytls.go:38 +0x4a
net/http.(*Server).Serve(0xc0005d9400, {0x560497cd2fc0, 0xc000455310})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:431 +0x1f
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go.func1()
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:529 +0x50
created by github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:526 +0x73

goroutine 61 [semacquire]:
internal/poll.runtime_Semacquire(0x0?)
        /usr/lib/go/src/runtime/sema.go:76 +0x25
internal/poll.(*fdMutex).rwlock(0xc00051bb80, 0x0?)
        /usr/lib/go/src/internal/poll/fd_mutex.go:154 +0xc5
internal/poll.(*FD).readLock(...)
        /usr/lib/go/src/internal/poll/fd_mutex.go:221
internal/poll.(*FD).Accept(0xc00051bb80)
        /usr/lib/go/src/internal/poll/fd_unix.go:602 +0x58
net.(*netFD).accept(0xc00051bb80)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc000536540)
        /usr/lib/go/src/net/tcpsock_posix.go:159 +0x1b
net.(*TCPListener).Accept(0xc000536540)
        /usr/lib/go/src/net/tcpsock.go:380 +0x30
net/http.(*Server).Serve(0xc000462600, {0x560497cd2690, 0xc000536540})
        /usr/lib/go/src/net/http/server.go:3424 +0x30c
github.com/lxc/incus/v6/internal/server/endpoints.(*Endpoints).serve.func1()
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:431 +0x1f
github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go.func1()
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:529 +0x50
created by github.com/lxc/incus/v6/internal/server/endpoints.(*Tomb).Go in goroutine 1
        /build/incus/src/incus-6.14/internal/server/endpoints/endpoints.go:526 +0x73

goroutine 62 [select, 19138 minutes]:
database/sql.(*DB).connectionOpener(0xc0004c5790, {0x560497cd72e0, 0xc0004554f0})
        /usr/lib/go/src/database/sql/sql.go:1261 +0x87
created by database/sql.OpenDB in goroutine 1
        /usr/lib/go/src/database/sql/sql.go:841 +0x130

goroutine 309 [IO wait, 19138 minutes]:
internal/poll.runtime_pollWait(0x7f0678fb05f0, 0x72)
        /usr/lib/go/src/runtime/netpoll.go:351 +0x85
internal/poll.(*pollDesc).wait(0xc0006b1300?, 0x7f06c023a108?, 0x0)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/lib/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0006b1300)
        /usr/lib/go/src/internal/poll/fd_unix.go:620 +0x295
net.(*netFD).accept(0xc0006b1300)
        /usr/lib/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x560495a45977?)
        /usr/lib/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc0007939b0)
        /usr/lib/go/src/net/unixsock.go:260 +0x30
github.com/lxc/incus/v6/internal/server/seccomp.NewSeccompServer.func1()
        /build/incus/src/incus-6.14/internal/server/seccomp/seccomp.go:1084 +0x59
created by github.com/lxc/incus/v6/internal/server/seccomp.NewSeccompServer in goroutine 1
        /build/incus/src/incus-6.14/internal/server/seccomp/seccomp.go:1082 +0x1d5

goroutine 229 [syscall, 19138 minutes]:
syscall.Syscall6(0xf7, 0x3, 0x1f, 0xc000618600, 0x4, 0xc00073a510, 0x0)
        /usr/lib/go/src/syscall/syscall_linux.go:95 +0x39
internal/syscall/unix.Waitid(0x200000000618688?, 0x7f06761a9b58?, 0x7f06c023a5c0?, 0x10?, 0xc000071008?)
        /usr/lib/go/src/internal/syscall/unix/waitid_linux.go:18 +0x39
os.(*Process).pidfdWait.func1(...)
        /usr/lib/go/src/os/pidfd_linux.go:106
os.ignoringEINTR(...)
        /usr/lib/go/src/os/file_posix.go:251
os.(*Process).pidfdWait(0xc000618788?)
        /usr/lib/go/src/os/pidfd_linux.go:105 +0x209
os.(*Process).wait(0x0?)
        /usr/lib/go/src/os/exec_unix.go:27 +0x25
os.(*Process).Wait(...)
        /usr/lib/go/src/os/exec.go:358
os/exec.(*Cmd).Wait(0xc000740a80)
        /usr/lib/go/src/os/exec/exec.go:922 +0x45
github.com/lxc/incus/v6/shared/subprocess.(*Process).start.func3()
        /build/incus/src/incus-6.14/shared/subprocess/proc.go:205 +0x59
created by github.com/lxc/incus/v6/shared/subprocess.(*Process).start in goroutine 1
        /build/incus/src/incus-6.14/shared/subprocess/proc.go:202 +0x6c6

goroutine 296 [select, 19138 minutes]:
main.deviceEventListener(0xc0008388c0)
        /build/incus/src/incus-6.14/cmd/incusd/devices.go:623 +0x1dc
created by main.(*Daemon).init in goroutine 1
        /build/incus/src/incus-6.14/cmd/incusd/daemon.go:1582 +0x4165

goroutine 349 [select, 420 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc0006557b8, {0x560497cd72e0, 0xc0007bc910})
        /build/incus/src/incus-6.14/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x1)
        /build/incus/src/incus-6.14/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.14/internal/server/task/group.go:60 +0x2c5

goroutine 350 [select, 58 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc0006557e8, {0x560497cd72e0, 0xc0007bc910})
        /build/incus/src/incus-6.14/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x2)
        /build/incus/src/incus-6.14/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.14/internal/server/task/group.go:60 +0x2c5

goroutine 351 [select, 420 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000655818, {0x560497cd72e0, 0xc0007bc910})
        /build/incus/src/incus-6.14/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x3)
        /build/incus/src/incus-6.14/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.14/internal/server/task/group.go:60 +0x2c5

goroutine 352 [select, 58 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000655848, {0x560497cd72e0, 0xc0007bc910})
        /build/incus/src/incus-6.14/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x4)
        /build/incus/src/incus-6.14/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.14/internal/server/task/group.go:60 +0x2c5

goroutine 353 [select]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000655878, {0x560497cd72e0, 0xc0007bc910})
        /build/incus/src/incus-6.14/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x5)
        /build/incus/src/incus-6.14/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.14/internal/server/task/group.go:60 +0x2c5

goroutine 354 [select]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc0006558a8, {0x560497cd72e0, 0xc0007bc910})
        /build/incus/src/incus-6.14/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x6)
        /build/incus/src/incus-6.14/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.14/internal/server/task/group.go:60 +0x2c5

goroutine 355 [select, 420 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc0006558d8, {0x560497cd72e0, 0xc0007bc910})
        /build/incus/src/incus-6.14/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x7)
        /build/incus/src/incus-6.14/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.14/internal/server/task/group.go:60 +0x2c5

goroutine 356 [select, 420 minutes]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000655908, {0x560497cd72e0, 0xc0007bc910})
        /build/incus/src/incus-6.14/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x8)
        /build/incus/src/incus-6.14/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.14/internal/server/task/group.go:60 +0x2c5

goroutine 357 [select]:
github.com/lxc/incus/v6/internal/server/task.(*Task).loop(0xc000655938, {0x560497cd72e0, 0xc0007bc910})
        /build/incus/src/incus-6.14/internal/server/task/task.go:69 +0x14b
github.com/lxc/incus/v6/internal/server/task.(*Group).Start.func1(0x9)
        /build/incus/src/incus-6.14/internal/server/task/group.go:63 +0x65
created by github.com/lxc/incus/v6/internal/server/task.(*Group).Start in goroutine 1
        /build/incus/src/incus-6.14/internal/server/task/group.go:60 +0x2c5

goroutine 354487 [runnable]:
net/http.(*connReader).startBackgroundRead.gowrap2()
        /usr/lib/go/src/net/http/server.go:686
runtime.goexit({})
        /usr/lib/go/src/runtime/asm_amd64.s:1700 +0x1
created by net/http.(*connReader).startBackgroundRead in goroutine 354486
        /usr/lib/go/src/net/http/server.go:686 +0xb6

goroutine 354197 [sync.Mutex.Lock, 11 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x0?, 0x0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc00074ec30)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc00074ec30, 0x0, 0x2, 0xc0004f3b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0x560497384394?, {0x560497364bec?, 0x5604973b73c6?}, {0xc0004f3f88?, 0x560497b704e0?, 0xc000a5a6f0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000eefc0, {0x560497cd72a8, 0xc000a5a750})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 353887 [sync.Mutex.Lock, 27 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc00074ec30)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc00074ec30, 0x0, 0x2, 0xc000b87b58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc00068e200?, {0x560497364bec?, 0xc0006c0150?}, {0xc000b87f88?, 0x560497b704e0?, 0xc0006c00c0?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0003ea3f0, {0x560497cd72a8, 0xc0006c0120})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

goroutine 354252 [sync.Mutex.Lock, 9 minutes]:
internal/sync.runtime_SemacquireMutex(0x0?, 0x71?, 0xe0?)
        /usr/lib/go/src/runtime/sema.go:95 +0x25
internal/sync.(*Mutex).lockSlow(0xc00074ec30)
        /usr/lib/go/src/internal/sync/mutex.go:149 +0x15d
internal/sync.(*Mutex).Lock(...)
        /usr/lib/go/src/internal/sync/mutex.go:70
sync.(*Mutex).Lock(...)
        /usr/lib/go/src/sync/mutex.go:46
log.(*Logger).output(0xc00074ec30, 0x0, 0x2, 0xc0004efb58)
        /usr/lib/go/src/log/log.go:243 +0x405
log.(*Logger).Printf(...)
        /usr/lib/go/src/log/log.go:268
net/http.(*Server).logf(0xc0008c0400?, {0x560497364bec?, 0xc0006c04b0?}, {0xc0004eff88?, 0x560497b704e0?, 0xc0006c0420?})
        /usr/lib/go/src/net/http/server.go:3638 +0x6e
net/http.(*conn).serve(0xc0000ef3b0, {0x560497cd72a8, 0xc0006c0480})
        /usr/lib/go/src/net/http/server.go:1983 +0x130a
created by net/http.(*Server).Serve in goroutine 60
        /usr/lib/go/src/net/http/server.go:3454 +0x485

Jul 23 '25 13:07 C0rn3j

Sorry for the long delay, this issue kinda fell of my list for some reason...

I did look at your most recent goroutine dump and unfortunately there's nothing weird there... the number of routines is reasonable and I'm not seeing anything unexpected for the running ones.

If this is still an issue on current Incus, we're going to have to resort to good old fashion strace to try and figure out what it's doing.

So if this is still happening, next time you see it, please do another goroutine dump and then run strace -fF -o debug.log -s99999 -p PID-OF-INCUSD and capture around 10s of data. Hopefully the excessive CPU usage is because of system calls and strace can tell us what's going on...

Nov 09 '25 05:11 stgraber

I've the daemon with an uptime of 3m22d and it's not doing it anymore, another machine also stopped doing it.

No clue what's up, but it seems to be gone, at least for now, I'll reopen if needed.

Nov 28 '25 08:11 C0rn3j