coherence
coherence copied to clipboard
oracle
Failing to list/create/delete caches through the REST API on Coherence v14.1.1.0
Describe the bug Attempting to create/list/delete caches through the REST API results in 404 error code.
To Reproduce I've installed this in my K8s cluster using the Coherence Kubernetes Operator as follows:
helm repo add coherence https://oracle.github.io/coherence-operator/charts
helm repo update
helm install \
--namespace coherence \
coherence-operator \
coherence/coherence-operator
apiVersion: coherence.oracle.com/v1
kind: Coherence
metadata:
name: storage-cluster
spec:
replicas: 3
role: storage
image: ghcr.io/oracle/coherence-ce:14.1.1-0-20
# Coherence settings
coherence:
cacheConfig: /coherence/config/cache-config.xml
localPort: 7575
localPortAdjust: 7576
metrics:
enabled: true
port: 9612
ssl:
enabled: false
management:
enabled: true
port: 30000
ssl:
enabled: false
# Mount the cache config
configMapVolumes:
- mountPath: /coherence/config
name: cache-config
# JVM settings
jvm:
memory:
heapSize: 4g
args:
- "-XX:+UseG1GC"
- "-XX:+AlwaysPreTouch"
- "-Dcoherence.rest.config=/coherence/config/coherence-rest-config.xml"
- "-Dcoherence.management=all"
# Ports configuration
ports:
- name: management
port: 30000
- name: metrics
port: 9612
serviceMonitor:
enabled: true
- name: extend
port: 20000
- name: rest
port: 8080
# Container resources
resources:
requests:
memory: "6Gi"
cpu: "2"
limits:
memory: "8Gi"
cpu: "4"
# Volume claim for persistence
volumeClaimTemplates:
- metadata:
name: storage
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
storageClassName: "gp3"
# Pod management
readinessProbe:
initialDelaySeconds: 30
periodSeconds: 10
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 10
# Service monitoring annotations
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9612"
apiVersion: v1
kind: ConfigMap
metadata:
name: cache-config
namespace: coherence
data:
cache-config.xml: |
<?xml version="1.0"?>
<cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config coherence-cache-config.xsd">
<caching-scheme-mapping>
<cache-mapping>
<cache-name>*</cache-name>
<scheme-name>distributed-scheme</scheme-name>
</cache-mapping>
</caching-scheme-mapping>
<caching-schemes>
<distributed-scheme>
<scheme-name>distributed-scheme</scheme-name>
<service-name>PartitionedCache</service-name>
<backing-map-scheme>
<local-scheme/>
</backing-map-scheme>
<autostart>true</autostart>
</distributed-scheme>
<proxy-scheme>
<service-name>ExtendHttpProxyService</service-name>
<acceptor-config>
<http-acceptor>
<local-address>
<address>localhost</address>
<port>8080</port>
</local-address>
</http-acceptor>
</acceptor-config>
<autostart>true</autostart>
</proxy-scheme>
</caching-schemes>
</cache-config>
coherence-rest-config.xml: |
<?xml version="1.0"?>
<rest-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://xmlns.oracle.com/coherence/coherence-rest-config"
xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-rest-config coherence-rest-config.xsd">
<resources>
<resource>
<cache-name>*</cache-name>
<key-class>java.lang.String</key-class>
<value-class>example.Person</value-class>
<direct-query enabled="true"/>
</resource>
</resources>
</rest-config>
I've followed the configuration instructions from the following documentation: https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.0/develop-remote-clients/building-your-first-coherence-rest-application.html#GUID-45D33C82-D9DA-455D-8691-FA341D86CF81 https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.0/develop-remote-clients/performing-grid-operations-rest.html#GUID-5C807FF2-5F0F-4F70-8665-1741BF7A22B0
Expected behaviour I expect to successfully retrieve, create and delete the caches in the cluster through the REST API. Instead, I get the following:
~ $ curl -v -X PUT http://storage-cluster-rest:8080/api/dist-http-example/1 \
> -H "Content-Type: application/json" \
> -d '{"value": "test1"}'
* Host storage-cluster-rest:8080 was resolved.
* IPv6: (none)
* IPv4: 172.20.51.129
* Trying 172.20.51.129:8080...
* Connected to storage-cluster-rest (172.20.51.129) port 8080
* using HTTP/1.x
> PUT /api/dist-http-example/1 HTTP/1.1
> Host: storage-cluster-rest:8080
> User-Agent: curl/8.12.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 18
>
* upload completely sent off: 18 bytes
< HTTP/1.1 404 Not Found
< Date: Wed, 19 Feb 2025 09:15:56 GMT
< Content-length: 0
< X-content-type-options: nosniff
<
* Connection #0 to host storage-cluster-rest left intact
~ $ curl -v http://storage-cluster-rest:8080/api/dist-http-example/1.json
* Host storage-cluster-rest:8080 was resolved.
* IPv6: (none)
* IPv4: 172.20.51.129
* Trying 172.20.51.129:8080...
* Connected to storage-cluster-rest (172.20.51.129) port 8080
* using HTTP/1.x
> GET /api/dist-http-example/1.json HTTP/1.1
> Host: storage-cluster-rest:8080
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 404 Not Found
< Date: Wed, 19 Feb 2025 09:16:11 GMT
< Content-length: 0
< X-content-type-options: nosniff
<
* Connection #0 to host storage-cluster-rest left intact
~ $ curl -v http://storage-cluster-rest:8080/api/caches
* Host storage-cluster-rest:8080 was resolved.
* IPv6: (none)
* IPv4: 172.20.51.129
* Trying 172.20.51.129:8080...
* Connected to storage-cluster-rest (172.20.51.129) port 8080
* using HTTP/1.x
> GET /api/caches HTTP/1.1
> Host: storage-cluster-rest:8080
> User-Agent: curl/8.12.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 404 Not Found
< Date: Wed, 19 Feb 2025 09:16:15 GMT
< Content-length: 0
< X-content-type-options: nosniff
<
* Connection #0 to host storage-cluster-rest left intact
According to the logs, the acceptor is successfully listening for connections:
MacBook-Pro:~ georgecristiancimpoies$ kubectl logs storage-cluster-0 | grep -i acceptor
Defaulted container "coherence" out of: coherence, coherence-k8s-utils (init)
2025-02-19 09:09:55.849/5.541 Oracle Coherence CE 14.1.1.0.20 <Info> (thread=NameService:TcpAcceptor, member=n/a): TcpAcceptor now listening for connections on storage-cluster-0:7575.3
2025-02-19 09:09:56.371/6.062 Oracle Coherence CE 14.1.1.0.20 <Info> (thread=Proxy:ManagementHttpProxy:HttpAcceptor, member=1): HttpAcceptor now listening for connections on 0:0:0:0:0:0:0:0:30000
2025-02-19 09:09:56.932/6.623 Oracle Coherence CE 14.1.1.0.20 <Info> (thread=Proxy:ExtendHttpProxyService:HttpAcceptor, member=1): HttpAcceptor now listening for connections on 10.203.30.168:8080
2025-02-19 09:09:57.008/6.699 Oracle Coherence CE 14.1.1.0.20 <Info> (thread=Proxy:MetricsHttpProxy:HttpAcceptor, member=1): HttpAcceptor now listening for connections on 0:0:0:0:0:0:0:0:9612
Environment (please complete the following information):
- Coherence CE version: 14.1.1-0-20
- Java version and Java vendor: Not applicable
- Environment: Kubernetes Client Version: v1.32.0 Kustomize Version: v5.5.0 Server Version: v1.29.12-eks-2d5f260
Additional context Add any other context about the problem here.
Hi @thegridman, I would appreciate it immensely if you could please have a look over my configuration, there's obviously something wrong that I'm doing in configuring the REST API...
Just for context, we are currently using Coherence version 3 and due to us being limited to Java 8, we are currently experimenting with Coherence CE v14.1.1.0 (AFAIK this is the last version supporting Java 8, please correct me if I'm wrong). I'm trying to do a simple PoC by installing it on our EKS cluster, enabling the REST API and performing some simple operations through the REST API before integrating it with our apps.
Thank you!
I'm no expert in Coherence REST and quite frankly that part of Coherence has kind of fallen out of use as most people just build REST applications on top of Coherence using frameworks like Helidon, Spring or Micronaut. Coherence REST is not ideal for real production quality REST applications.
Having said that, it should still work. Have you actually tried running all this outside of Kubernetes by just starting Coherence locally with the same config and trying the http requests.
Hi @thegridman , thank you for the quick response! Unfortunately no, I wasn't able to test it with a sample Java app either.
Could you please confirm my understanding that there are actually 2 REST APIs, one for general cluster management and monitoring exposed over port 30000 and an Extend proxy defined according to the following acceptor:
<proxy-scheme>
<service-name>ExtendHttpProxyService</service-name>
<acceptor-config>
<http-acceptor>
<local-address>
<address>localhost</address>
<port>8080</port>
</local-address>
</http-acceptor>
</acceptor-config>
<autostart>true</autostart>
</proxy-scheme>
exposed on port 8080, which I should be able to use in order to interact with the data in the cache?
All I want to do is create/write to/read from and delete a cache over REST, that's all.
Yes, there is a REST API for Coherence Management, which basically gives all the same sort of thing you can get from Coherence JMX MBeans.
What you are using is Coherence REST where you can access cache data as documented here for 14.1.1.0 https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.0/develop-remote-clients/building-your-first-coherence-rest-application.html#GUID-45D33C82-D9DA-455D-8691-FA341D86CF81
I tried grpc as an alternative but that doesn't seem to be available in the version I'm using (as I'm stuck with Java 8). Would you be able to recommend a simple example on how to test the cluster deployed on K8s with either REST or a Java client?
Thanks!
Following the documentation at https://docs.oracle.com/en/middleware/standalone/coherence/14.1.1.0/develop-remote-clients/building-your-first-coherence-rest-application.html#GUID-E2C5134A-FF8D-44B5-8777-3FE67424FAA6
and using the following:
java -cp DEV_ROOT\config;DEV_ROOT;DEV_ROOT\libs\dependencies;
COHERENCE_HOME\coherence-rest.jar -Dcoherence.clusterport=8090
-Dcoherence.ttl=0
-Dcoherence.cacheconfig=DEV_ROOT\config\example-server-config.xml
com.tangosol.net.DefaultCacheServer
and the coherence.jar (standalone install) https://www.oracle.com/middleware/technologies/coherence-archive-downloads.html
seems to be missing this class:
Error: Could not find or load main class com.tangosol.net.DefaultCacheServer
so I suspect something is definitely wrong.
You are missing the Coherence jar from the class path, it appears you only have coherence-rest.jar
@george-c29 @thegridman Is there anything else we need to do here, or can this issue be closed?
@aseovic I wasn't able to get the REST api working so I dropped this initiative completely for now...if you happen to have a straightforward guide on how to interact with the cache cluster solely through the REST API, that would be great. However, I tried the documentation examples listed above with no success.
Thanks!
@george-c29 What error did you get? Did you check the console output from the Coherence server to see that the Http server (HttpAcceptor service) is started and use the IP address and port the HttpAcceptor listens on in your Http request?
@lsho yes, I've posted a log extract in the first commend of all the acceptors that started and their listening port, so the answer is yes. However, using the examples above I wasn't able to reach it (I was getting a 404 no matter what I tried).
@george-c29 Your Http server is listening on: (thread=Proxy:ExtendHttpProxyService:HttpAcceptor, member=1): HttpAcceptor now listening for connections on 10.203.30.168:8080
But your http request uses: 172.20.51.129:8080