[Feature]: See the formatted (JSON or XML) response in the trace viewer Network tab

[eveltman-twinfield]

🚀 Feature Request

Add some way to quickly see the formatted JSON or XML response in the Network tab of the trace viewer. It could be a button "Format response" in the Body (sub) tab, or a "Preview" tab similar to the Chrome Developer Tools where you can expand and collapse sections of the response, or some other solution that makes unformatted responses easier to read.

Example

JSON

Suppose we receive a response like:

{"_links":{"self":{"href":"https://api.accounting.master.dev.twinfield.com/api/users"},"current":{"href":"https://api.accounting.master.dev.twinfield.com/api/users/user?code=PAYABNAMRO"},"summaries":{"href":"https://api.accounting.master.dev.twinfield.com/api/users/summaries"},"userbasicinfo":{"href":"https://api.accounting.master.dev.twinfield.com/api/users/userbasicinfo?code={code}"},"user":{"href":"https://api.accounting.master.dev.twinfield.com/api/users/user?code={code}"},"formpostlogindeadline":{"href":"https://api.accounting.master.dev.twinfield.com/api/users/formpostlogindeadline"}}}

It would be great if there was a quick way (button, a dedicated preview tab?) to see it like:

{
    "_links": {
        "self": {
            "href": "https://api.accounting.master.dev.twinfield.com/api/users"
        },
        "current": {
            "href": "https://api.accounting.master.dev.twinfield.com/api/users/user?code=PAYABNAMRO"
        },
        "summaries": {
            "href": "https://api.accounting.master.dev.twinfield.com/api/users/summaries"
        },
        "userbasicinfo": {
            "href": "https://api.accounting.master.dev.twinfield.com/api/users/userbasicinfo?code={code}"
        },
        "user": {
            "href": "https://api.accounting.master.dev.twinfield.com/api/users/user?code={code}"
        },
        "formpostlogindeadline": {
            "href": "https://api.accounting.master.dev.twinfield.com/api/users/formpostlogindeadline"
        }
    }
}

XML

Original:

<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><ProcessXmlStringResponse xmlns="http://www.twinfield.com/"><ProcessXmlStringResult>SNIP</ProcessXmlStringResult></ProcessXmlStringResponse></soap:Body></soap:Envelope>

Reformatted/preview version:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema">
    <soap:Body>
        <ProcessXmlStringResponse xmlns="http://www.twinfield.com/">
            <ProcessXmlStringResult>SNIP</ProcessXmlStringResult>
        </ProcessXmlStringResponse>
    </soap:Body>
</soap:Envelope>

Motivation

REST APIs or SOAP web services don't always format the responses nicely with newlines and other whitespace. It would save some time troubleshooting test failures when we could easily see the formatted (JSON or XML) responses in the Network tab of the trace viewer.

Then we don't have to copy-paste it to (for example) vscode and use some formatting tool there.

[pavelfeldman]

@cpAdm do you want to do it?

[cpAdm]

@cpAdm do you want to do it?

@pavelfeldman Sure. However, I don't think this feature is trivial to add. Formatting is not something CodeMirror offers anymore.

For JSON, it would be trivial to add via JSON.stringify(JSON.parse(value), null, 2), but for XML (and others) it would require prettier (and probably also some plugins). This significantly increases the bundle size.

So sadly, I don't think we should add this feature in CodeMirror. Note that monaco-editor does offer such features out-of-the-box.

[pavelfeldman]

Sounds reasonable. Might still be worth it for JSON?

[eveltman-twinfield]

Hi, yes it would already be an improvement to have this for JSON only at first. Yesterday I was also thinking maybe I should create two separate issues for this, but eventually went for creating a single issue.

But the XML formatting would be very valuable for us too, since our tests use SOAP web services a lot, maybe even more than the REST APIs.

For implementing the XML formatting perhaps you could use https://www.npmjs.com/package/xmldom-format or https://www.npmjs.com/package/@xmldom/xmldom ? Maybe some library with a Sax parser could do the trick too.

Or would that increase the bundle size too much also?

Now I was looking in the trace viewer, I just had an idea: Maybe a nice UI solution would be to have another dropdown next to the 'Copy request' dropdown. Something like 'View' with the options 'Raw' and 'Reformatted' (and only do the actual reformatting when you switch to that view).

[cpAdm]

If we want to implement this I suggest a toggle button for pretty printing. Similar to what chrome devtools has:

One of the disadvantages is that the line numbering for the pretty print will no longer correlate with the original line numbering if we implement it with the JSON.stringify. We will probably also have to hide this button/bar whenever the response is not JSON. And what will we do if the response doesn't turn out to be valid JSON?

@pavelfeldman What is the Playwright team's stance on switching to Microsoft's monaco-editor? I understand that this would be a big change. But it does align nicely with VS-code (basically same editor) and offers more modern functionality out-of-the-box.

[pavelfeldman]

monaco is good, but it is too big. we bundle trace viewer in the reports, so we need something much more lightweight. devtools approach sgtm (i think i implemented it 15 years ago there :)). Line numbers mismatch is acceptable.

[cpAdm]

@pavelfeldman Makes sense. It would probably increase with couple of MB it seems, unfortunately. https://www.agenthicks.com/research/codemirror-vs-monaco-editor-comparison

It turns out that if the response is JSON, Playwright already formats it: https://github.com/microsoft/playwright/blob/7249e2068ac847884c461a8fda5b844a40875cf0/packages/trace-viewer/src/ui/networkResourceDetails.tsx#L242

So, not sure if I can contribute here.

[eveltman-twinfield]

Hmm as far as I can see our API supports different response content types (application/vnd.twinfield+json, application/hal+json, application/json). Our UI and our EndToEndTests request application/vnd.twinfield+json mostly, so that is what we get as our content type.

Our API is built on the HATEOAS/Hypermedia principles. I think that's also where the somewhat odd content types are coming from. Not sure if we can switch our Accept headers to application/json. I doubt it would get priority here if it's to achieve a small benefit in troubleshooting failed e2e tests :-(

Any chance the format could be detected based on content instead of based on the response header? Or that the user selects the format they want to see it autoformatted in?

And is XML (SOAP response) formatting out of scope for this issue/something I should create a separate issue for?

[cpAdm]

Playwright currently only formats here if the media type is application/json, but as per RFC 6839 I think we should also try to JSON-format if if the type ends with +json.

Looking at the codebase, we should probably do that at more places. E.g. when filtering on Fetch in the network tab, I think you don't see these '+json' request here? https://github.com/microsoft/playwright/blob/5b04cf56c41727dc279acc7a10d008da08b6117d/packages/trace-viewer/src/ui/networkTab.tsx#L365

I guess we could use mimeType.ts::isJsonMimeType in these (two) places. https://github.com/microsoft/playwright/blob/5b04cf56c41727dc279acc7a10d008da08b6117d/packages/playwright-core/src/utils/isomorphic/mimeType.ts#L17-L19

@pavelfeldman What do you think?

[pavelfeldman]

This all code does not make sense to me, we should figure it out based on the request type, not the contentType. Why would Fetch imply application/json.

[cpAdm]

This all code does not make sense to me, we should figure it out based on the request type, not the contentType. Why would Fetch imply application/json.

Good observation. I agree.

I am just unsure how to proceed here. Because I don't think we have such a network 'request type' available here in the (HAR) Entry?

I did have a look how this behaviour is implemented at chrome devtools. They do have such a NetworkRequestTypes available. https://github.com/ChromeDevTools/devtools-frontend/blob/bb82faf6ad6e0da0471a4487fe02dcdd3db25bb4/front_end/models/trace/lantern/types/Lantern.ts#L7

And their logic on determining the type: https://github.com/ChromeDevTools/devtools-frontend/blob/bb82faf6ad6e0da0471a4487fe02dcdd3db25bb4/front_end/panels/utils/utils.ts#L120-L136

Playwright does have a network.Request.resourceType() available, but this does not seem to get saved into the HAR. https://github.com/microsoft/playwright/blob/5620cc7722e5818111792499f7ceac4409f9b6db/packages/playwright-core/src/server/network.ts#L168

So, I think the solution here would be to to save this resourceType inside the HAR and use that to filter here. Maybe I am overseeing something here. But considering the complexity its probably better that I leave this one to the Playwright team. @pavelfeldman @Skn0tt

[pavelfeldman]

They do have such a NetworkRequestTypes available.

First of all, not they, but we - Playwright folks here are the original DevTools team :P.

Playwright does have a network.Request.resourceType() available, but this does not seem to get saved into the HAR.

We can put it on har as an underscore property, not big deal. Or we can store it outside har entry in the tracing network entry. Making it a part of the har entry is simpler though.

[cpAdm]

Bit too much complexity for my liking at this point in time, so not picking this one up.