Support for testing chunked transfer encoding response bodies

[ocramz]

Chunked response bodies have a specific format, which prepends chunk length as a hex integer,.e.g

4␍␊            (chunk size is four octets)
Wiki           (four octets of data)
␍␊             (end of chunk)

7␍␊            (chunk size is seven octets)
pedia i        (seven octets of data)
␍␊             (end of chunk)

B␍␊            (chunk size is eleven octets)
n ␍␊chunks.    (eleven octets of data)
␍␊             (end of chunk)

0␍␊            (chunk size is zero octets, no more chunks)
␍␊             (end of final chunk with zero data octets)

https://en.wikipedia.org/wiki/Chunked_transfer_encoding

It would be nice to support this in hspec-wai. It would be sufficient to implement a MatchBody that does the chunking (and optional validation of the chunk lengths) and applies a predicate to the chunks.

[ocramz]

A simple implementation of the internals using ReadP from base and aeson (with no length validation):

-- | assumes that each chunk can be decoded as a separate JSON object
chunkedP :: A.FromJSON a => RP.ReadP [a]
chunkedP = mconcat <$> RP.manyTill chunkP RP.eof

chunkP :: A.FromJSON a => RP.ReadP [a]
chunkP = do
  _ <- hexP >> RP.skipSpaces
  cs <- RP.option [] jsonString -- NB last chunk in the stream is a 0-length string
  RP.skipSpaces
  pure cs

jsonString :: A.FromJSON a => RP.ReadP [a]
jsonString = do
  chars <- RP.munch1 (\c -> not $ c `elem` ['\n', '\r'])
  pure $ maybeToList $ A.decode $ LBS8.pack chars


-- | read a hexadecimal number as a string
hexP :: RP.ReadP [Char]
hexP = RP.munch1 predicate
  where
    predicate c = isDigit c || c `elem` ("abcdef" :: [Char])

[sol]

I'm not sure if I understand what you are asking for?

Do you want to test a WAI application? Or do you want to test that a WAI server (e.g. warp) produces the correct wire format?

[ocramz]

I originally thought of addressing the latter scenario, testing a server, which would fit with the use cases of this library.

[sol]

I originally thought of addressing the latter scenario, testing a server, which would fit with the use cases of this library.

I'm still not sure if I understand you correctly. Do you want to write more tests for warp, or are you working on an alternative WAI server implementation?

[ocramz]

Hi, neither of the two :) I call a "server" any wai Application.

I am proposing here to add a helper to test the serialization produced as response by Applications that comply with the chunked transfer format.

Hence my notes above, regarding MatchBody.

[sol]

I'm still not sure if I understand your use case. Naïvely, I assume that it is the responsibility of the server (read warp) to create the wire format.

If you have a specific use case that is currently not addressed, then please provide a concrete and minimal code example that illustrates your use case.