Health check on my openscoring server

[Tsachi-Simplex]

I would like to perform a health check on my openscoring server. This is required for Amazon services (such as load balancing), as well as other features I need. My health check includes: Checking that I can send commands to the server (such as query the server for models). More importantly, I need a good indication that at least one model is deployed. Is there any built in way to do it?

Thanks!

[vruusmann]

You could perform a GET request against the default "Get the summaries of all models" endpoint /model.

This endpoint is guaranteed to be available at all times. The only problem that I see is that perhaps this endpoint is too "expensive", because the size of the response body is proportional to the number (and complexity) of deployed models.

It would be possible to create a dedicated "health check" REST API endpoint. Do you have a formal specification for designing it (eg. easy integration with AWS)?

[Tsachi-Simplex]

Yes. I wish to ask for an easy integration with AWS.

Specifically for a health check URL that will be used by auto-scaling groups or Elastic Load Balancers. What they need is a way to know when:

1. the application server is ready to accept traffic. Only when that happens will an application server start to get traffic from the elastic load balancer.
2. similarly, that the application server is still healthy. If not - then the auto-scaling group, after a couple of attempts to call the health check (configurable on the auto-scaling group side) will shutdown the machine of the application server and start a new one instead.

This health check URL will be called, indeed, pretty frequently, and as such should be pretty lightweight.

The above are the requirements. I believe the best implementation would be if the application war, would be able to be configured with a directory of the models, and upon initialisation will install all of them. One health check URL will return "healthy=true" only when all the models in the directory will be successfully deployed. Another possible health check URL, that may be used in other use-cases, might be a simpler one, that would just return "health=true" always and would be used for other use-cases.

We need the first use-case though.

Makes sense ?

[vruusmann]

The Openscoring web service relies on proven Java technologies, so the risk of it dying "out of the blue" is effectively zero. You would need to stop/restart Openscoring web service only in relation to planned version updates.

It seems to me that your use case can be characterized as "developing a health check, which ensures that the specified PMML filesystem directory is in full sync with the Openscoring instance".

It should be implemented as a separate JAX-RS component:

1. Create a POJO that holds a reference to the singleton org.openscoring.service.ModelRegistry instance. The reference will be provided by the JAX-RS framework using dependency injection.
2. On the first access, this component should start a PMML filesystem directory "watchdog" thread. If this thread detects a filesystem event, then it will invoke appropriate ModelRegistry methods such as #put(String, Model), #replace(String, Model, Model) or #remove(String, Model).
3. This component keeps track of the "health" of the watchdog thread, which is exposed via a dedicated REST endpoint as {status=in sync} or {status=off sync}. It is possible that the watchdog thread may die independently of the Openscoring web service.
4. This component is registered with the Openscoring web service by listing the name of its class in Openscoring configuration file under the componentClasses section (together with the name of the watchable PMML filesystem directory etc).

[vruusmann]

This issue is related to issue #9.

The existing auto-deployment functionality is a client-side solution. It means that the auto-deployer is started as a separate process, which communicates with the Openscoring web service over the standard REST API interface. A client may die (or be killed) at any time, and the Openscoring instance wouldn't know about it. Also, it is possible that there are several clients (eg. one client process per watchable PMML filesystem directory) updating a single Openscoring instance.

However, this use case is totally different, because the auto-deployer will live inside the Openscoring instance and will communicate with it using internal Java APIs. I did research possible communication options a bit, and the above "custom JAX-RS component that maintains a watchdog thread" seems the only reasonable solution.

I don't like the idea of making this functionality part of the Openscoring "core" codebase at this point (breaks the golden rule that Java EE applications should not start their own service threads, plus unnecessary maintenance burden). It should be implemented by users themselves when needed.