bcgov
All health checks return a status of ok or ready even when the service is not functioning
We've run into an issue in a couple of our instances were the agent and/or service database fills up forcing the pods into a crash loop and creating a situation were requests cannot be completed.
When this occurs one or more of the health checks should fail bringing the service outage to the attention of the monitoring solution. However, all of the health checks pass in this scenario, and the issues only surface when interacting with the service to authenticate.
Example application authorization failure messages:
https://vc-authn-oidc.apps.silver.devops.gov.bc.ca/authorize?scope=openid+vc_authn&state=n04bV2BP3ZJvuQCRMDY4a9vUvytm4O9dtsxvMCAnv4k.obdK6SW6E5U.1Ks0kstcS96yQ7oQbRVp5g.eyJydSI6Imh0dHBzOi8vanVzdGljZS5nb3YuYmMuY2EvZGFycy9zaWduaW4tb2lkYyIsInJ0IjoiY29kZSIsInJtIjoiZm9ybV9wb3N0Iiwic3QiOiJDZkRKOEwyRXlWbzlMNTlEaWp5Tmw2QXA2WHBTcXZabEw0ZS1UZUN6dmxMRDItSDhfMkZkbmhrNm1xaDhfMXNsamZtQVlZbV9sMU1nUF9ZczBlcDhUZXJuSlhvaHlwWU8tLWMxM1Q2YS1rVTU2N0VfdVV5UU9RbDVjX3JtN2ZRbE02RDBjamZVdkxqUXF5YXJRNU84T3lIZlNlNXlidjN3Zi0wNTdFYzh1REpDODhiZW5mV3pkN04wQy1fZENWakcxdWkzNmxVd0owbnpPZnFLX0laeV9Ec21MZEhlWHJkeklVdE54bVh5VGVibERDWngxOTBkNU5sbHZrZEJxdGc0c29SaDhyWm01VE1faDdYQkgxeFg2TGJvaUZieU02UGpvS1lZejZ2aWk2d2dxZ3EtVTZzTGpGNU5UUXAtcHB6RjlhRG9WWTdTZlBackNETVRhSkczZnU0UUhBdUVqMi1UcTg0RmU0bHEyN2FfaXNsU3BWdXUzWDhpd2hibDVoWnIwekExNTFwQTZUZkhEQ3RmM3Q5a21ScEdWbFZZS0hXUk01UEZWaFFTT3ZwUEZGeWdVdlNlLXJfYTBwaXJTY2tzWW1wVHZ3In0&response_type=code&client_id=acm-jag-vc&redirect_uri=https%3A%2F%2Floginproxy.gov.bc.ca%2Fauth%2Frealms%2Faccess-to-court-materials-jag%2Fbroker%2Fverifiable-credential%2Fendpoint&pres_req_conf_id=accredited-lawyer-bcpc&nonce=8ljpH6WoNbkEYJiVW05bYA
https://vc-authn-oidc.apps.silver.devops.gov.bc.ca/authorize?scope=openid+vc_authn&state=j0hISkwAxTdgls7xZncQ9l0LipraGCVNdKtosSkJjqs.cRQbpYsjLNk.1Ks0kstcS96yQ7oQbRVp5g.eyJydSI6Imh0dHBzOi8vanVzdGljZS5nb3YuYmMuY2EvZGFycy9zaWduaW4tb2lkYyIsInJ0IjoiY29kZSIsInJtIjoiZm9ybV9wb3N0Iiwic3QiOiJDZkRKOEwyRXlWbzlMNTlEaWp5Tmw2QXA2WHBEc3FDclE2TG96MGFsTWg5ZkpDRGpSWjZtSFNzM1NFdUdzX2E0Rl9MZmFnUDgwT0F0SDlDM2ZTRmp1Y1RsYkdsZE1lc1VNV2xLZ1BDU2pPRXVXb25UbGVnOXhPQ1ZmMXo4WWVOaTZQczZmVm0tc2NCbXdleENYMXg2MnlhVnFQY2xVTDNRSnJrdmdMMmRtYlc4dDdyUWV1OWRxZUZWbWtzcEpnaVZheDhKNmhQZlNXdlNBNTZORVdYdjBTYXdaR2IyQ1NoOEhiSkJBQmFBclBsakVOMUd4UU00X3RKUXIzdkVQdTZEbE40V1NhcktOUmdvSHQ2a3pGWVNIdWVEN08yR21lLVVHTmk3RkNmOVFmY1oyZzBsQkZ6VnNMellOVlpHa1huLW1xdGswUkdKdkhaQmd2MlFHYlBlYVZfNkZhdFN0UjF4ek9BVlVwbGljallud2hSaUk1OHN4WGZSMEZFblgwY0I3TTNWS0JxSk9JUmI5NEJsd3N0NWpwSFZ4eUFUa1dyNnNIU0d1SFQwb3ZwS2tPTm5GOGVlUm5RVGlBOEUtS3hIRUp3eW5kemJTeVViMmtSWHpIeWlsMTRqTlM3NFRtUTFfU3pIOXByak1obVRIQUJvIn0&response_type=code&client_id=acm-jag-vc&redirect_uri=https%3A%2F%2Floginproxy.gov.bc.ca%2Fauth%2Frealms%2Faccess-to-court-materials-jag%2Fbroker%2Fverifiable-credential%2Fendpoint&pres_req_conf_id=csb-transcriber&nonce=Wt8Ty97EAqyOIGtWZboHOQ
