ERROR There was an issue loading ingest pipeline into OpenSearch [503]

[michburger]

Dear sassoftware support,

during installation I get this error message:

INFO Loading Content into OpenSearch INFO The index management policy [viya_logs_idxmgmt_policy] already exist in OpenSearch; skipping load and using existing policy. ERROR There was an issue loading ingest pipeline into OpenSearch [503] ERROR Exiting script [deploy_logging.sh] due to an error executing the command [logging/bin/deploy_opensearch_content.sh].

Where can I find more details and how can I solve this?

[gsmith-sas]

@michburger The problem you've encountered is not one we've seen before, so I cannot point you to any documentation or notes about it. Let me ask you a few questions and see if we can figure out what's going on.

• From the messages you shared, it appears you were updating an existing deployment of SAS Viya Monitoring. Please clarify the versions involved; i.e. what version was in-place and what version were you attempting to upgrade to. Please share the output from running the command helm -n logging list.
• Were there any WARNING messages generated during the deployment process prior the set of messages you shared? These errors were generated in step where we are loading content into OpenSearch, there is a previous step where OpenSearch is deployed (or upgraded) and some configuration tweaks are made. I'm wondering if that step produced any messages that might be helpful.
• The 503 response code in the ERROR message suggests that OpenSearch was not ready when the deployment script attempted to load some content in OpenSearch. Please run a kubectl get pods command to check on the status of the pods in the "logging" namespace and share the results.
• If the pods in the 'logging' namespace appear to be healthy now, it's possible that this is/was a transient problem. Have you tried re-running the deployment scripts?

Regards, Greg

By the way, the SAS Viya Monitoring project documentation is available in the SAS Help Center, here.

[gsmith-sas]

@michburger Were you able to everything working or are you still having trouble?

[michburger]

Dear Greg,during my investigation, I found an out of space issue with the underlying database. Currently, I try to find out why old data was not deleted. I assume that these data are leftovers from previous viya logging setups.Many thanks for the hints!Michael Burger SoftMatch.net -------- Ursprüngliche Nachricht --------Von: Greg Smith @.> Datum: 03.05.24 16:18 (GMT+01:00) An: sassoftware/viya4-monitoring-kubernetes @.> Cc: michburger @.>, Mention @.> Betreff: Re: [sassoftware/viya4-monitoring-kubernetes] ERROR There was an issue loading ingest pipeline into OpenSearch [503] (Issue #634) @michburger Were you able to everything working or are you still having trouble?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[gsmith-sas]

@michburger When OpenSearch runs out of space, it isn't able to do the index management processing that would (normally) delete "old" data and free up space. The easiest (and most effective) solution at that point is to remove and redeploy after deleting the PVCs. The logging/bin/remove_logging.sh script does NOT delete the OpenSearch PVCs by default. However, you can have the script delete PVCs by setting the environment variable LOG_DELETE_PVCS_ON_REMOVE to 'true'.

It is also a good idea to try and identify why you ran out of space. For example, had you recently turned on a lot of DEBUG level messaging? Did you deploy some new application onto the cluster that is especially verbose? Are you aware of a problem that may have caused a spike in log volumes? Was there especially heavy end-user activity?

If you determine that the spike in log message volume was a temporary event, you may decide to continue with the default size (30GB) for the OpenSearch PVCs. However, if you are not sure or if you feel the end-user activity prior to the problem was "typical" for you environment, you may want to increase the size of these PVCs. To increase the size of these PVCs before (re)deploying, you need to specify a larger value for the persistences.size key in the logging/user-values-opensearch.yaml file in your $USER_DIR directory. More information about customizing your deployment of SAS Viya Monitoring can be found in the section The Customization Process in our documentation. You may want review the sections Adjust the Retention Period for Log Messages and Increase Storage for Log Messages as well. The Increase Storage for Log Message section focuses on how to increase storage in a running system (rather than adjusting things during the pre-deployment phase). I would encourage you to keep an eye on the size of the OpenSearch PVCs and make additional adjustments as-needed after you have things (re)deployed.