Pega pods connection to SRS with an external elastic fails

[arun-022]

Describe the bug I have provisioned Pega 8.8.3 on AKS with a Postgres DB using the pega-helm-charts. I started with using the deprecated internal search service and am now switching to the SRS service with an external elastic cluster. My external elastic cluster is also provisioned in AKS and the SRS pods can connect to this cluster. The SRS pods comes up fine and I see the message ["Elasticsearch cluster 'elasticsearch' contacted successfully, server version: '7.17.14' (compatible), node number: '1', cluster health: 'GREEN']. Now as a last step to externalizing search I changed the values file for the Pega deployment to using the externalSearchService and configured the endpoint where SRS is exposed. I see these lines now in the PegaRULES log and the search functionality doesnt work

2023-11-23 10:28:35,200 [sse-nio-8443-exec-24] [ STANDARD] [ ] [ PegaRULES:8] ( internal.srs.SRSConnectorImpl) WARN |10.148.194.18 HFM9NO04R5FTF25GDU9HHUE73L0GTYAJWA [email protected] - [SRS][Response][Track: 2f9edaff] Response status code from SRS is invalid and it won't be parsed, response status code: 404, body: {"message":"Isolation Context doesn't exist, this possibly means that the datamodel hasn't been published"} 2023-11-23 10:28:35,300 [sse-nio-8443-exec-24] [ STANDARD] [ ] [ PegaRULES:8] ( internal.srs.SRSConnectorImpl) WARN |10.148.194.18 HFM9NO04R5FTF25GDU9HHUE73L0GTYAJWA [email protected] - [SRS][Response][Track: 5b9e76ae] Response status code from SRS is invalid and it won't be parsed, response status code: 404, body: {"message":"Isolation Context doesn't exist, this possibly means that the datamodel hasn't been published"} 2023-11-23 10:28:53,739 [sse-nio-8443-exec-16] [TABTHREAD0] [ ] [ PegaRULES:8] (rulebridge.SRSStatusEngineImpl) ERROR |10.148.194.206 HFM9NO04R5FTF25GDU9HHUE73L0GTYAJWA [email protected] - SRS Indexing error: missing indexable group for SRS: SRS$Deployment

The SRS Pods in turn throw this error

"Security Alert: Request attempted with invalid isolation ID","logger":"com.pega.fnx.search.common.exceptions.handler.TenantNotFoundExceptionHandler","thread":"RxCachedThreadScheduler-1","level":"ERROR","stack_trace":"com.pega.fnx.search.common.exceptions.TenantNotFoundException: null\n\tat io.reactivex.rxjava3.internal.operators.maybe.MaybeErrorCallable.subscribeActual(MaybeErrorCallable.java:41)\n\tat io.reactivex.rxjava3.core.Maybe.subscribe(Maybe.java:5377)\n\tat io.micronaut.rxjava3.instrument.RxInstrumentedMaybe.subscribeActual(RxInstrumentedMaybe.java:53)\n\tat io.reactivex.rxjava3.core.Maybe.subscribe(Maybe.java:5377)\n\tat io.reactivex.rxjava3.internal.operators.maybe.MaybeSwitchIfEmpty$SwitchIfEmptyMaybeObserver.onComplete(MaybeSwitchIfEmpty.java:88)\n\tat io.micronaut.rxjava3.instrument.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:72)\n\tat io.reactivex.rxjava3.internal.operators.maybe.MaybeFlatten$FlatMapMaybeObserver.onComplete(MaybeFlatten.java:107)\n\tat io.micronaut.rxjava3.instrument.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:72)\n\tat io.reactivex.rxjava3.internal.operators.maybe.MaybeMap$MapMaybeObserver.onComplete(MaybeMap.java:100)\n\tat io.micronaut.rxjava3.instrument.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:72)\n\tat io.reactivex.rxjava3.internal.operators.maybe.MaybeFlatten$FlatMapMaybeObserver.onComplete(MaybeFlatten.java:107)\n\tat io.micronaut.rxjava3.instrument.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:72)\n\tat io.reactivex.rxjava3.internal.operators.maybe.MaybeFlatten$FlatMapMaybeObserver.onComplete(MaybeFlatten.java:107)\n\tat io.micronaut.rxjava3.instrument.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:72)\n\tat io.reactivex.rxjava3.internal.operators.maybe.MaybeSubscribeOn$SubscribeOnMaybeObserver.onComplete(MaybeSubscribeOn.java:101)\n\tat io.micronaut.rxjava3.instrument.RxInstrumentedMaybeObserver.onComplete(RxInstrumentedMaybeObserver.java:72)\n\tat io.reactivex.rxjava3.internal.operators.maybe.MaybeObserveOn$ObserveOnMaybeObserver.run(MaybeObserveOn.java:105)\n\tat com.pega.fnx.search.model.CommonRequestFilter.lambda$new$0(CommonRequestFilter.java:50)\n\tat io.reactivex.rxjava3.core.Scheduler$DisposeTask.run(Scheduler.java:644)\n\tat com.pega.fnx.search.model.CommonRequestFilter.lambda$new$0(CommonRequestFilter.java:50)\n\tat io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:65)\n\tat io.reactivex.rxjava3.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:56)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n","isolationID":"pega-tst","X-Correlation-ID":"2f9edaff"}

To Reproduce Deploy an elastic cluster, then deploy the SRS backing service using the helm chart. Deploy Pega and connect to the backing service

Expected behavior Pega pods should be able to connect successfully to the SRS pods.

Chart version 3.4.0

Server (if applicable, please complete the following information):

• OS: Ubuntu 18.04
• Environment: [AKS]
• Database: [PostgreSQL]

Additional context Add any other context about the problem here.

[maracle6]

Please add a customerDeploymentID to the global section of your values file. For example:

---
global:
  # This values.yaml file is an example.  For more information about
  # each configuration option, see the project readme.

# Enter your Kubernetes provider.
  provider: "YOUR_KUBERNETES_PROVIDER"
  customerDeploymentID: application_environment

The customerDeploymentID is used for multi tenant services (currently only for SRS) and it will be the prefix used for the Elasticsearch indexes. It must be a unique string for every environment using SRS. If you enable OAuth for SRS, your token must provide this string in the guid claim.

However, this should default to the name of the namespace pega was installed to. If this doesn't resolve your problem try ensuring you're using the latest helm chart release and check that the ElasticSearch cluster has auto_create_index set to false.

[MadhuriArugula]

@arun-022 Are you still seeing the issue?

[kishorv10]

@arun-022 Are you still seeing the issue?

[arun-022]

No, I initiated a reindex from the Pega web pods which solved the issue

[kishorv10]

Client confirmed that issue is no more seen.