Obtaining raw XML request and response in addition to "processed" response from the server

[vpogrebi]

Version: 2.4.0

I have developed a generic SOAP client interface capable of executing web service requests against different target systems. In addition to obtaining response from the server (which normally comes in a form of a dictionary-like object), we must maintain raw XML request and response (stored in the database) - in order to be able to troubleshoot production issues and perform all sorts of analysis/research/investigation. Problem is: "zeep" package does not provide an easy way to obtain raw XML request/response when SOAP operation is executed.

After spending a long time researching, debugging and stepping through the code - we finally found an easy way to do this. Our solution involves overriding Transport class (Transport.post() method to me precise) to store both XML request and response that comes back from "requests" package - as additional attributes of Transport class:

from zeep import transports


class Transport(transports.Transport):
    def post(self, address, message, headers):
        self.xml_request = message.decode('utf-8')
        response = super().post(address, message, headers)
        self.response = response
        
        return response

This simple solution (class override) is easily available - due to the fact that our code instantiates Transport class directly (before instantiating Client class), unlike my other issue (#601: https://github.com/mvantellingen/python-zeep/issues/601) where luxury of direct class instantiation is not available to client code.

What this "override" does - is that it intercepts raw XML request ('message') and 'response' received from "requests" package, and stores them as self.xml_request and self.response attributes. Initially, I stored just "clear" XML response part of the 'response' (as response.content.decode('utf-8'), but later decided to store entire 'response' instead - to also enable use of other 'response' attributes (response.status_code and response.reason) in order to process response (check for failure).

I would like to request that "zeep" developers implement same enhancement to make it available to everyone - as it is a very common requirement to have both raw XML data and "processed" response.

[mvantellingen]

Isn't http://docs.python-zeep.org/en/master/plugins.html#historyplugin what you are looking for?

[vpogrebi]

I have not known about plugins before (I am pretty new to "keep"), but I guess it "can" be used for this purpose. Question is: why would a special plugin need to be used to do this - while it's just literally two-line (of code) change? Personally, I much prefer that Transport class includes "xml_request", "xml_response" and "response" attributes - to provide a simple, direct access to this data. Besides... "HistoryPlugin" does not provide direct access to raw 'response' from "requirements" module.

I think what I offered - is a better alternative...

[fatestapestry]

Alternatively, can the HistoryPlugin not be extended to include an option for xml in both last_sent and last_received?

I see this is included in your custom plugin sample, but this is such a common request, should it's worth including this in the HistoryPlugin itself?

[fredcwbr]

Howdy, Could you show how you did the overloading? Got the same situation, but not as proficient in python nor Zeep...

[fatestapestry]

@fredcwbr what are you trying to do exactly? Not sure which message you are responding to in the thread...

[geolobrad]

I think I understand fredcwbr's question because I am in the same situation. I am a Python newbie working on a Python script to access an application using SOAP API. I use SOAPUI to test the requests so I know what the raw XML should look like.

Looks like HistoryPlugin does not record the raw XML requests, I see a mention by jonathanelscpt about 'custom plugin sample' but don't see this sample, is that available and I am missing it?

I can include the 'class Transport' code from vpogrebi in my script with no error, but looks like more is needed to get it to work, I'm guessing post() need to be invoked somewhere else in order to populate the values of xml_request and response?

Thanks!

[fatestapestry]

Set up your client as per the HistoryPlugin guide:

from zeep import Client
from zeep.plugins import HistoryPlugin

history = HistoryPlugin()
client = Client(wsdl=WSDL, transport=transport, plugins=[history])

Then, you can extract the xml message using etree.

from lxml import etree

try:
    for hist in [history.last_sent, history.last_received]:
        print(etree.tostring(hist["envelope"], encoding="unicode", pretty_print=True))
except (IndexError, TypeError):
    # catch cases where it fails before being put on the wire
    pass

[geolobrad]

works for me, thank you!

[fredcwbr]

Howdy,

Got the Transport Override working as suggested by **vpogrebi ** , and did get the XML Copy on a file with everything that I wanted,.

Did use the HistoryPlugin too, but got some issues with threading. Understanding both scenarios, IMPO it would be interesting to have **vpogrebi's ** suggestion considered a little further.

[fatestapestry]

out of interest, can you share the issues that you had with HistoryPlugin when threaded? Is there a bug for this?

[fredcwbr]

Howdy,

The problem that I'm resolving needs a multi worker thread (at this moment 30) , each one, which does its own request, and with the response, spawn other 3 requests in behalf of the recalled info. For some reason the HistoryPlugin mixes up the responses., At this point I would not regard it to a bug or some issue on zeep, even because I don't have neither the experience nor the expertise to fully understand .

The Transport override solution did address my issue, and solved the problem without any other effort,

[ghost]

Guys, I have raw xml that i need to send as request and get the raw xml in response. i have used "requests" library, but can't i use zeep? i don't have wsdl url. Please help

[bitsofinfo]

agree w/ the OP, I had a need to get the raw XML back in the response.. I looked at history plugin, but the docs are pretty incomplete as to "what" exact kind of object this Envelope that is returned is. only the comment by @jonathanelscpt above helped. Some docs https://docs.python-zeep.org/en/master/api.html and https://docs.python-zeep.org/en/master/internals.html would be great.

Also @fredcwbr comment re: concurrency is concerning

[do-rtk]

I just had a similar need. We pickle python requests to be able to replay transactions later and it looks like the current plugins architecture won't really support that. ref: https://github.com/mvantellingen/python-zeep/blob/377d9313b1b4807a31a5ee42227f1dc7e7e0471e/src/zeep/wsdl/bindings/soap.py#L221

Any plans to add support to have plugins at the request level i.e. capture and modify ingress and egress requests? Happy to send a PR with something like request_plugins if there's interest

[Zyten]

+1 for this. Currently working on a 3rd party integration where I am expected to log the xml_request, xml_response, status_code and message (reason). Without this override I was stuck with using raw_response=True, just to get http_status and reason before parsing the response again

With the override it's a breeze