Shouldn't exceptions be logged? REST service was unable to handle this query

[ILyaCyclone]

We have 2 WebLogic 12.2.1.3 servers, that we believe are identical. But on freshly deployed wls-exporter.war with same configuration server1 is showing needed allmetrics, while server2 has error in datasource area:

# REST service was unable to handle this query
# {
#   "links": [],
#   "fields": [],
#   "children": {
#     "JDBCServiceRuntime": {
#       "links": [],
#       "fields": [],
#       "children": {
#         "JDBCDataSourceRuntimeMBeans": {
#           "links": []
#         }
#       }
#     }
#   }
# }

I could not find any indication of this error in server logs and started googling error message. And found this section: https://github.com/oracle/weblogic-monitoring-exporter/blob/master/src/main/java/io/prometheus/wls/rest/ExporterServlet.java#L70 ... which catces exception and doesn't print anything to the log.

Two questions:

1. Why is it done this way, shouldn't errors be logged?
2. How can I find the cause of this error?

Thanks.

[russgold]

If it were logged, it would appear every time you do a query, which would likely overload the server log. It might be possible to cache them and avoid duplicates, but I'd be concerned about small differences failing the duplicate check. It's worth exploring a better way to handle it, though. Thanks for the suggestion.

[ILyaCyclone]

Got your point. Something like a buffer would fit probably. Kind of Map<Exception, Long> exceptionCount (key obviously needs more thinking) and dump them to log when count hits some configured value.

Could you suggest how do I debug error happening on my server?

[russgold]

The first thing I would try is to look at the "messages" page. It is not linked, but you can find it at http:/messages (as opposed to http:/metrics, which is where you should have found the reply above). That will show you what is being sent to WLS and back by the exporter. If it's not obvious what is going on, paste the result here.

[ILyaCyclone]

server1/wls-exporter/messages has like 10 screens of text and has entry about datasource:

REQUEST:
{"links":[],"fields":[],"children":{"JDBCServiceRuntime":{"links":[],"fields":[],"children":{"JDBCDataSourceRuntimeMBeans":{"links":[]}}}}}
REPLY:
{"JDBCServiceRuntime": {"JDBCDataSourceRuntimeMBeans": {"items": [
...

server2/wls-exporter/messages looks alike but does not mention anything about datasource, even though both servers' exporters are queried and configured in the same way.

---

I found out the error cause using WebLogic rest management: server2:adminport/management/weblogic/latest/domainRuntime/serverRuntimes/wld1ms1/JDBCServiceRuntime/JDBCDataSourceRuntimeMBeans showed an error message which read smth like "hr@test is suspended and cannot allocate resource". There is no such datasource in console Services>Data Sources, but I found there is an old ADF test application that was turned off (state suspended) and it had an inner JDBC module (created by JDeveloper deployment) named "hr@test". After I deleted that application the problem on server2/metrics lifted.

That whole thing feels terrible to me. A whole management section goes down just if I have a suspended application (which is not wrong by itself)? Of cause it is not directly connected to wls-exporter, but... I might not be understanding the oracle ways.

---

Should I close this issue or leave it be or recreate it just about logging? I still believe there should be some error cause indication in wls-exporter.

[satyam031291]

Hi All,

I am also facing similar issue in which all other metrics in Prometheus are getting executed but getting a run time error only for JDBC Data sources.

I have also attached my wls-exporter file which I have deployed on weblogic server.

wls-exporter.txt

Below is the error from my wls-exporter metrics.

wls_persistentstore_update_count{domain="asapdomain",name="_WLS_MS1"} 45

REST service was unable to handle this query

{

"links": [],

"fields": [],

"children": {

"JDBCServiceRuntime": {

"links": [],

"fields": [],

"children": {

"JDBCDataSourceRuntimeMBeans": {

"links": []

}

}

}

}

}

[ddsharpe]

The exporter needs to do a better job of reporting that things are not working. The user needs to be able to see what failed. The suggestion above regarding caching might be a viable alternative. Throwing away the log messages completely is not helpful to the user.

[russgold]

There have been a number of usability enhancements added to the exporter. Closing now - if the latest is still a problem, I'll expect a new issue to be raised.