boundary CPU burn / slow >= v0.14.0

Describe the bug

Running Hashicorp Boundary in a container (at least) burns CPU and each API request from UI is slow (~1s).

Screenshot 2023-10-22 at 11 42 22

To Reproduce

Steps to reproduce the behavior in dev mode:

Order a fresh Linux VM with Ubuntu 22.04 LTS (AMD EPYC 4 cores, 16Gb RAM)
Install Docker
Fix permissions for the test (horrible fix, but it's just to reproduce) : sudo chmod 777 /var/run/docker.sock
Run boundary dev mode : docker run --net=host -v /var/run/docker.sock:/var/run/docker.sock --rm hashicorp/boundary:latest dev
Boundary starts and then CPU burns

Steps to reproduce the behavior in dev mode with external Postgres 15 (another machine):

Order a fresh Linux VM with Ubuntu 22.04 LTS (AMD EPYC 4 cores, 16Gb RAM)
Install Docker
Run boundary dev mode : docker run --rm hashicorp/boundary:latest dev -database-url=XXXXX
Boundary starts (but is really slow, it's about few minutes) and then CPU burns

Steps to reproduce the behavior in production mode with external Postgres 15 (another machine):

Order a fresh Linux VM with Ubuntu 22.04 LTS (AMD EPYC 4 cores, 16Gb RAM)
Install Docker
Run boundary mode with production config enabling worker
Boundary starts (but is really slow) and then CPU burn. Each API request from UI is about 1s

However, in production mode, I found that disabling worker, stops CPU burning but API requests are still really slow from UI.

If I run in dev mode on my laptop (Apple M1 PRO) without Docker, I don't have CPU burn issue and everything is fast. If I run on VMs without container, I also have the issue (Linux Ubuntu 22.04 LTS related issue ?)

It has been tested on two different cloud providers for VMs, both running Ubuntu 22.04 LTS

I don't have the CPU burn issue with v0.13.1 but I've the issue with v0.14.0 and v0.14.1. However, I've really slow API requests and so slow UI with v0.13.1.

Expected behavior

Hashicorp boundary is responsive and no longer burn CPU.

Additional context Add any other context about the problem here.

Oct 22 '23 09:10 guyguy333

Looked into the issue and it seems to be caused by a long-running background check which was using a lot of CPU.

We have started working on a fix for this issue: https://github.com/hashicorp/boundary/pull/3884

Oct 23 '23 21:10 elimt

@guyguy333 The latest Boundary 0.14.2 release has the the fix to address this issue. Let me know if that helps address the issue.

Nov 02 '23 17:11 elimt

Thanks a lot @elimt, I can confirm it solved CPU burning issue.

However, I still have huge slowness in production mode with latency-ms ~900ms for most requests.

Nov 03 '23 10:11 guyguy333

@guyguy333

Could you please provide details about your setup?
Could you also share your system specs? Boundary has a page about system requirements

Nov 03 '23 16:11 elimt

Hey @guyguy333 -- are you still encountering this issue?

Jan 02 '24 23:01 AdamBouhmad

Hi @AdamBouhmad, yes I still have the issue. App is slow and request latency is about 900ms, resulting is really slow UI.

To answer @elimt and provide more details, I run container on a K8S cluster (hashicorp/boundary:0.14.3). There is no CPU or memory limit for this container. I setup these env vars: BOUNDARY_POSTGRES_URL, HOSTNAME (= boundary) and VAULT_TOKEN. Server is started using boundary server -config /boundary/config.hcl

My config:

disable_mlock = true
log_format    = "json"

controller {
  name        = "kubernetes-controller"
  description = "Boundary Controller"
  public_cluster_addr = "boundary-cluster.example.com:443"

  database {
    url = "env://BOUNDARY_POSTGRES_URL"
    max_open_connections = 10
    max_idle_connections = 10
  }
}

# Ingress TCP Route
worker {
  name              = "kubernetes-worker"
  description       = "Boundary Worker"
  address           = "localhost"
  initial_upstreams = ["boundary:9201"]
  public_addr       = "boundary-worker.example.com:443"
}

# Ingress
listener "tcp" {
  address              = "0.0.0.0"
  purpose              = "api"
  tls_disable          = true
  cors_enabled         = true
  cors_allowed_origins = ["https://boundary.example.com"]
}

listener "tcp" {
  address       = "0.0.0.0"
  purpose       = "cluster"
  tls_cert_file = "/certs/tls.crt"
  tls_key_file  = "/certs/tls.key"
}

listener "tcp" {
  address       = "0.0.0.0"
  purpose       = "proxy"
  tls_cert_file = "/certs/tls.crt"
  tls_key_file  = "/certs/tls.key"
}

kms "transit" {
  purpose            = "root"
  address            = "https://vault.example.com"

  // Key configuration
  key_name           = "boundary-root"
  mount_path         = "transit/"
}

kms "transit" {
  purpose            = "recovery"
  address            = "https://vault.example.com"

  // Key configuration
  key_name           = "boundary-recovery"
  mount_path         = "transit/"
}

kms "transit" {
  purpose            = "worker-auth"
  address            = "https://vault.example.com"

  // Key configuration
  key_name           = "boundary-worker-auth"
  mount_path         = "transit/"
}

Certs are mounted from a K8S secret in /certs.

Host machine uses AMD64 arch and has 4 cores and 32Gb RAM

Currently, there is no load, only me testing solution to have something stable before opening it.

Thanks for considering the issue :)

Jan 03 '24 10:01 guyguy333

boundary boundary copied to clipboard

CPU burn / slow >= v0.14.0

boundary
boundary copied to clipboard