Get '[JSON] does not imply a header Accept: application/json

[alt-romes]

I am using servant-client to request JSON information from the hackage API, but it turns out that specifying '[JSON] in the Get request will not add a header Accept: application/json to the Request, even though requestAccept seems to have the correct application types.

Here's a reproducer:

{-# LANGUAGE DataKinds #-}
{-# LANGUAGE TypeOperators #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DerivingStrategies #-}
{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE DeriveAnyClass #-}
module Main where

import GHC.Generics
import Data.Proxy
import Data.Aeson
import Servant.API
import Network.HTTP.Client (newManager, defaultManagerSettings, requestHeaders)
import Servant.Client

data Package
  = Package
    { packageName :: String
    , downloads :: Int
    }
    deriving stock Show
    deriving stock Generic
    deriving anyclass FromJSON

type API = "top" :> Get '[JSON] [Package]

getTop :: ClientM [Package]
getTop = client (Proxy @API)

main :: IO ()
main = do
  manager' <- newManager defaultManagerSettings
  let clientEnv = mkClientEnv manager' (BaseUrl Http "hackage.haskell.org" 80 "packages")
  res <- runClientM getTop (clientEnv{makeClientRequest = \b r -> makeClientRequest clientEnv b r >>=
                                                            \r' -> return r'{requestHeaders = [("Accept", "application/json")]}})
  case res of
    Left err -> putStrLn $ "Error: " ++ show err
    Right packages -> print packages

with build-depends: base, servant, servant-client, aeson, http-client

Expected behaviour

I was expecting the Accept: application/json to be automatically added to the request, since I'm using a Get request and JSON as the decoding format.

[tchoutri]

@alt-romes Quite unfortunately your code works perfectly fine on my machine, which confirms my theory that Hackage is a bad HTTP server to query. Not too sure what can be done at Servant's level, since we put the right headers (https://hackage.haskell.org/package/servant-0.20.1/docs/src/Servant.API.ContentTypes.html#line-134).

PS: download numbers on Hackage are unreliable because it doesn't use the data from its CDN, so the numbers are not representative at all of any kind of trend (except perhaps that a package has been downloaded at least once).

[alt-romes]

I don't recall if I analyzed the request generated by servant. Perhaps one could look to see whether Get '[JSON] [Package] is sufficient to add the correct header to the request.

I see now that my reproducer may be wrong in the sense that it is the one that works. If you remove the part where requestHeaders is manually edited to introduce the needed header, does it still work?

[tchoutri]

Ah, how quaint! I'll toy with your code and hit httpbin.org to see where it fails.

[tchoutri]

Hmm I'm not seeing anything explicitly wrong in this result:

{
  "headers": {
    "Accept": "application/json;charset=utf-8,application/json", 
    "Accept-Encoding": "gzip", 
    "Host": "httpbin.org", 
    "X-Amzn-Trace-Id": "Root=1-660c6e2c-75bb6dfc08b3f13056f4c4d0"
  }
}

[tchoutri]

@alt-romes In my personal experience, hackage-sever is more often faulty than servant-client. If you can trace the requests as they are understood and processed by hackage-server and show that it's servant-client's fault I'd be more than happy to reopen, but in this case it might be a case of bad behaviour from the server.