Feature request: support for binary response body in recording

[Jinming-Hu]

[Jinming-Hu]

Hi @vhvb1989 , correct me if I'm wrong. The recording-playback framework doesn't support binary body, right?

So I'm working on Blob Query feature in storage SDK, and the response body is avro/binary. Can we add support for binary body so that testcases for this feature can still be recorded/playedback?

Here's is an example of HTTP response

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: avro/binary
Last-Modified: Wed, 11 May 2022 11:02:42 GMT
Accept-Ranges: bytes
ETag: "0x8DA333DC536DC03"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-creation-time: Wed, 11 May 2022 11:02:42 GMT
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-blob-type: BlockBlob
x-ms-server-encrypted: true
x-ms-version-id: 2022-05-11T11:02:42.1074947Z
x-ms-is-current-version: true
x-ms-request-id: d4ddfe09-901e-0054-7826-659595000000
x-ms-version: 2020-08-04
x-ms-client-request-id: 93d648e0-95c1-411d-578d-c0da884b0e28
Date: Wed, 11 May 2022 11:02:44 GMT

[vhvb1989]

@Jinming-Hu , the test recorded writes the response to json for responses that are not greater than 2k.

I don't see a body in your HTTP response example.

We could use the Content-Type and if it is avro/binary, we can write a second file with the binary content, and the recorded response would use it. However, I am a little worried about adding big files to the repo (if responses are big).

Also, the C++ playback recorded is going EOL soon. We need to migrate to use the cross-language playback recorded https://github.com/Azure/azure-sdk-tools/tree/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy

@scbedd , does the test-proxy already support recording avro/binary responses?

[Jinming-Hu]

Hi @vhvb1989 , I wanted to upload a sample response. But it turned out GitHub won't let me, even if I save binary file in .txt.

Size of the response body for test cases is about 3KB. If we use base64 encoding, it will 4KB. It's not a lot. I guess it's acceptable.

But since you mentioned the playback framework is about to reach its EOL, maybe we can just wait.

[Jinming-Hu]

@vhvb1989 Feel free to close this PR

[scbedd]

@scbedd if we don't recognize the Content-Type to be "text" one, we will fall back base64 encoding the bytes to store in the recording. I believe avro/binary should fall into that. Willing to give it a shot if you can pass me a test to run though!

[Jinming-Hu]

body.txt

Hi @scbedd , here's an example of avro/binary response body. Does that work for you?

[scbedd]

I need to see the actual request itself. A fiddler of one of these request/response pairs is basically exactly what I'm looking for. We look at headers to determine the type, which is what will control the fallback mechanisms.

Short of a fiddler, if this repo has tests integrated with the proxy, I can physically run one of the tests against the proxy and tell you what the result is.

[Jinming-Hu]

Hi @scbedd , here is a request:

POST https://account.blob.core.windows.net/blockblobclienttestqueryjsoninputcsvoutput/QueryJsonInputCsvOutput?comp=query HTTP/1.1
Connection: Keep-Alive
Content-Type: application/xml; charset=UTF-8
Authorization: SharedKey euap:QM5dOfq1qHuajxEc2V9pHxsMebnRCP4Hl/sxl1icQAA=
User-Agent: azsdk-cpp-storage-blobs/12.5.0-beta.1 (Windows 10 Pro 6.3 19044 19041.1.amd64fre.vb_release.191206-1406)
x-ms-client-request-id: 453a1109-31a5-4913-6af8-97c03b9a280c
x-ms-date: Sat, 14 May 2022 03:51:37 GMT
x-ms-version: 2020-08-04
Content-Length: 552
Host: account.blob.core.windows.net

<?xml version="1.0" encoding="utf-8"?><QueryRequest><QueryType>SQL</QueryType><Expression>SELECT * from BlobStorage WHERE id &gt; 101 AND price &lt; 100;</Expression><InputSerialization><Format><Type>json</Type><JsonTextConfiguration><RecordSeparator/></JsonTextConfiguration></Format></InputSerialization><OutputSerialization><Format><Type>delimited</Type><DelimitedTextConfiguration><ColumnSeparator/><FieldQuote/><RecordSeparator/><EscapeChar/><HasHeaders>false</HasHeaders></DelimitedTextConfiguration></Format></OutputSerialization></QueryRequest>

and response headers:

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: avro/binary
Last-Modified: Sat, 14 May 2022 03:51:37 GMT
Accept-Ranges: bytes
ETag: "0x8DA355D0C0028A4"
Server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
x-ms-creation-time: Sat, 14 May 2022 03:51:37 GMT
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-blob-type: BlockBlob
x-ms-server-encrypted: true
x-ms-version-id: 2022-05-14T03:51:37.5914148Z
x-ms-is-current-version: true
x-ms-request-id: 3dcaf035-601e-000d-0d45-671216000000
x-ms-version: 2020-08-04
x-ms-client-request-id: 453a1109-31a5-4913-6af8-97c03b9a280c
Date: Sat, 14 May 2022 03:51:48 GMT

[scbedd]

@Jinming-Hu this is exactly what I'm looking for.

Early answer, I believe this should work. We will simply take your content bytes -> base 64 encode them for storage -> restore to original binary on playback. I'll ping back a bit later today after I have time to trace the code paths and be absolutely sure.

[RickWinter]

This is supported in the new test-recorder integration.