api-gateway icon indicating copy to clipboard operation
api-gateway copied to clipboard

Published 20 hours ago • verified publisher icon membrane

The proxy sporadically returns an empty response

Open 700grm opened this issue 3 years ago • 0 comments

System:

Ubuntu 18.4 LTS
membrane 4.7.3
Java Service Wrapper: 3.5.44
ClamAV from the oficial ubuntu repo

In some very rare cases service-proxy may return an empty resposne, and it is not clear why. We are using the proxy sometime once a day, sometime once a week. We have configuration where the proxy is using different paths to forward requests to relevant servers. It may work with one path but it fail with another. Config file

<spring:beans xmlns="http://membrane-soa.org/proxies/1/"
    xmlns:spring="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.2.xsd
                        http://membrane-soa.org/proxies/1/ http://membrane-soa.org/schemas/proxies-1.xsd">
    <router>
        <soapProxy port="1553" wsdl="./conf.d/server1.wsdl">
            <ssl>
                <keystore location="./conf.d/key-server.jks" password="<PASSWORD>" keyPassword="<PASSWORD>" />
            </ssl>
            <clamav host="localhost" port="8888" />
            <request>
                <validator wsdl="./conf.d/server1.wsdl" />
                <limit maxBodyLength="32000" />
                <rateLimiter requestLimit="100" requestLimitDuration="PT1S"/>
                <xmlProtection removeDTD="true"/>
            </request>
            <log category="info" headerOnly="false" />
        </soapProxy>
        <soapProxy port="2553" wsdl="./conf.d/server2.wsdl">
            <ssl>
                <keystore location="./conf.d/key-server.jks" password="<PASSWORD>" keyPassword="<PASSWORD>" />
            </ssl>
            <clamav host="localhost" port="8888" />
            <request>
                <validator wsdl="./conf.d/server2.wsdl" />
                <limit maxBodyLength="32000" />
                <rateLimiter requestLimit="100" requestLimitDuration="PT1S"/>
                <xmlProtection removeDTD="true"/>
            </request>
            <log category="info" headerOnly="false" />
        </soapProxy>
        <soapProxy port="3553" wsdl="./conf.d/server3.wsdl">
            <!-- http://192.168.0.99:80/services/ShService -->
            <ssl>
                <keystore location="./conf.d/key-server.jks" password="<PASSWORD>" keyPassword="<PASSWORD>" />
            </ssl>
            <clamav host="localhost" port="8888" />
            <request>
                <validator wsdl="./conf.d/server3.wsdl" />
                <limit maxBodyLength="32000" />
                <rateLimiter requestLimit="100" requestLimitDuration="PT1S"/>
                <xmlProtection removeDTD="true"/>
            </request>
            <log category="info" headerOnly="false" />
        </soapProxy>
    </router>
</spring:beans>

For example by executing the command from the proxy itself (aso from the remote server) # wget --post-file=test.xml --header="Content-Type: text/xml" --no-check-certificate https://localhost:3553/services/ShService -O response.xml

it may return 502 Bad Gateway:

# wget --post-file=test.xml --header="Content-Type: text/xml" --no-check-certificate https://localhost:3553/services/ShService  -O response.xml

--2021-08-25 13:32:34--  https://localhost:3553/services/ShService
Resolving localhost (localhost)... 127.0.0.1, ::1
Connecting to localhost (localhost)|127.0.0.1|:3553... connected.
WARNING: cannot verify localhost's certificate, issued by ‘<I REMOVED CERT ISSUER PART>’:
  Unable to locally verify the issuer's authority.
WARNING: no certificate subject alternative name matches
        requested host name ‘localhost’.
HTTP request sent, awaiting response... 502 Bad Gateway
2021-08-25 13:32:35 ERROR 502: Bad Gateway.

Where the membrane wrapper log shows the successful response HTTP/1.1 200 OK from the server3, but it is not received on the initiator end. Appears as an empty response.

INFO   | jvm 1    | 2021/08/25 13:32:34 | 13:32:34,921  INFO info:127 - ==== Request ===
INFO   | jvm 1    | 2021/08/25 13:32:34 | 13:32:34,926  INFO info:127 - POST /services/ShService HTTP/1.1
INFO   | jvm 1    | 2021/08/25 13:32:34 |
INFO   | jvm 1    | 2021/08/25 13:32:34 | 13:32:34,926  INFO info:127 - Headers:
INFO   | jvm 1    | 2021/08/25 13:32:34 | 13:32:34,926  INFO info:127 - User-Agent: Wget/1.19.4 (linux-gnu)
INFO   | jvm 1    | 2021/08/25 13:32:34 | Accept: */*
INFO   | jvm 1    | 2021/08/25 13:32:34 | Accept-Encoding: identity
INFO   | jvm 1    | 2021/08/25 13:32:34 | Host: localhost:3553
INFO   | jvm 1    | 2021/08/25 13:32:34 | Connection: Keep-Alive
INFO   | jvm 1    | 2021/08/25 13:32:34 | Content-Type: text/xml
INFO   | jvm 1    | 2021/08/25 13:32:34 | Content-Length: 2103
INFO   | jvm 1    | 2021/08/25 13:32:34 | X-Forwarded-For: 127.0.0.1
INFO   | jvm 1    | 2021/08/25 13:32:34 | X-Forwarded-Proto: https
INFO   | jvm 1    | 2021/08/25 13:32:34 | X-Forwarded-Host: localhost:3553
INFO   | jvm 1    | 2021/08/25 13:32:34 |
INFO   | jvm 1    | 2021/08/25 13:32:34 | 13:32:34,926  INFO info:127 - Body:
INFO   | jvm 1    | 2021/08/25 13:32:34 | 13:32:34,926  INFO info:127 - <?xml version="1.0"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:u="http://www.metaswitch.com/ems/soap/sh/userdata" xmlns:sh="http://www.metaswitch.com/ems/soap/sh" xmlns:s="http://www.metaswitch.com/ems/soap/sh/servicedata">
INFO   | jvm 1    | 2021/08/25 13:32:34 |   <soapenv:Header></soapenv:Header>
INFO   | jvm 1    | 2021/08/25 13:32:34 |   <soapenv:Body>
INFO   | jvm 1    | 2021/08/25 13:32:34 |     <sh:ShUpdate>
........
REMOVED BODY PART FOR SEC REASONS
........
INFO   | jvm 1    | 2021/08/25 13:32:34 |     </sh:ShUpdate>
INFO   | jvm 1    | 2021/08/25 13:32:34 |   </soapenv:Body>
INFO   | jvm 1    | 2021/08/25 13:32:34 | </soapenv:Envelope>
INFO   | jvm 1    | 2021/08/25 13:32:34 | 13:32:34,926  INFO info:127 - ================
INFO   | jvm 1    | 2021/08/25 13:32:35 | 13:32:34,935  INFO info:127 - ==== Response ===
INFO   | jvm 1    | 2021/08/25 13:32:35 | 13:32:34,935  INFO info:127 - HTTP/1.1 200 OK
INFO   | jvm 1    | 2021/08/25 13:32:35 |
INFO   | jvm 1    | 2021/08/25 13:32:35 | 13:32:34,935  INFO info:127 - Headers:
INFO   | jvm 1    | 2021/08/25 13:32:35 | 13:32:34,935  INFO info:127 - Server: Apache-Coyote/1.1
INFO   | jvm 1    | 2021/08/25 13:32:35 | Content-Type: text/xml;charset=UTF-8
INFO   | jvm 1    | 2021/08/25 13:32:35 | Transfer-Encoding: chunked
INFO   | jvm 1    | 2021/08/25 13:32:35 | Date: Wed, 25 Aug 2021 13:32:34 GMT
INFO   | jvm 1    | 2021/08/25 13:32:35 |
INFO   | jvm 1    | 2021/08/25 13:32:35 | 13:32:34,935  INFO info:127 - Body:
INFO   | jvm 1    | 2021/08/25 13:32:35 | 13:32:34,936  INFO info:127 - <REMOVED BODY PART FOR SEC REASONS>
INFO   | jvm 1    | 2021/08/25 13:32:35 | 13:32:34,936  INFO info:127 - ================

The next consecutive attemps will work fine, for me it looks like some processes are idling inside membrane

700grm avatar Aug 26 '21 14:08 700grm