How to design a type-safe JSON endpoint where output fields depend on incoming request?

[saurabhnanda]

Note: This may be closely related to #9

Examples:

GET /products/1?fields=name,currency,advertised_price

{
  "name": "Ceramic mug"
  ,"currency": "INR"
  ,"advertised_price": "129.50"
}

GET /products/1?fields=name,currency,advertised_price,description,comparison_price

{
  "name": "Ceramic mug"
  ,"currency": "INR"
  ,"advertised_price": "129.50"
  ,"description": "Shatter resistant ceramic mug with lovely flower prints"
  ,"comparison_price": "12"
}

GET /products/1?fields=name,currency,advertised_price,variants.name,variants.sku

{
  "name": "Ceramic mug"
  ,"currency": "INR"
  ,"advertised_price": "129.50"
  ,"variants": [
    {
      "name": "Red color"
      ,"sku": "MUGRED"
    }
    ,{
      "name": "Blue color"
      ,"sku": "MUGBLUE"
    }
  ]
}

[sudhirvkumar]

@saurabhnanda

This immediately brings up horror stories...

1. Security issues
2. Performance
3. Complexity

I believe if you need flexibility in selecting the fields then we can use GraphQL (http://graphql.org/)

http://graphql.org/blog/rest-api-graphql-wrapper/

http://stackoverflow.com/questions/38339442/json-schema-to-graphql-schema-converters

Let me know your thoughts

[saurabhnanda]

Wrt GraphQL, yes, this is an ad-hoc way of implementing the same ideas that GraphQL is trying to solve.

However, I haven't researched GraphQL deeply. Can you help me with the following questions:

• Is GraphQL just a protocol or does it have server-side & client-side implementation?
• Is any server-side implementation available in Haskell?
• Is any client-side implementation available for Purescript, Elm, or Haskell?
• If one is using GraphQL, does it mean that one doesn't need to write JSON API endpoints at all? You just configure the GraphQL server to hook up to your DB (Postgres in our case) and let it do it magic?

[sudhirvkumar]

I mentioned GraphQL as it might be useful for your use cases.

• GraphQL has a server side which in turns fetches data from API endpoints
• Haskell server not yet (https://hackage.haskell.org/package/graphql)
• Purescript client not yet (https://github.com/jqyu/purescript-graphql)
• We will be writing API endpoints for mutations. We can avoid a lot of boilerplate with regard to reading data. We can use relay (https://facebook.github.io/relay/)

Having said all that, I am not going to jump into GraphQL right away. I am still evaluating as I want to avoid using JavaScript... if I have time, then I might try to build a library in PureScript (node.js).

Even though I can use GraphQL rightaway too... then I will need to use Haskell for REST API Endpoints and GraphQL server (node.js) and use that in PureScript client with FFI.

Right now Haskell API end points and then in the future GraphQL / Apollo (http://www.apollostack.com/)

[sudhirvkumar]

@saurabhnanda

if you really want to implement this, then I can think of a way.

1. Limit the fields that can be mentioned in the "fields" QueryParams using ADT?
2. use a [(key,value)] type and store the fields manually by filtering the fields
3. use custom ToJson instance to render the [(key, value)] type to JSON.

doable... a little messy and no guarantees! I am not sure if I will want to do something like this

[mpdairy]

I think if you want to just send certain fields of a record, you would want to box every field value of the record with something like a Maybe. So you'd still send back, for instance, a whole product record every time, but all the values of fields you don't care about get set to Nothing and all the ones you do care about are wrapped in Just.

To specify, in the query request, which fields you want to get back in the response, you'd send a whole record (i.e. a product record) with every field having a Bool value to indicate whether to fetch or ignore that field. That same record of bools could probably be used to limit the database query as well.

[saurabhnanda]

Hey @mpdiary, thanks for chiming-in. If the whole product record gets sent with missing fields as Nothing, it is still quite an overhead. Also, we would be force-fitting three states into two: value present, value absent, and value omitted, into just the two. What we need is something like:

data Omittable a = Absent | Present a
-- non nullable field 
Omittable String
-- Nullable field
Omittable (Maybe String)

So, a custom toJSON function should be able to take a record and NOT emit the fields with an Omittable type having an Absent value.

This problem is similar to how most DB libraries need to deal with missing columns in SQL queries.

[mpdairy]

Oh, you're right, that would be a lot of overhead. Well it looks like you can use the alternative <|> operator to give an option for fields that were not found in the JSON. For example:

data Person = Person
              { name :: Bool
              , age  :: Bool
              } deriving (Show)

instance FromJSON Person where
     parseJSON (Object v) = Person <$>
                            (v .: "name" <|> pure False)
                             <*>
                            (v .: "age" <|> pure False)
     parseJSON _          = mzero

Then you can do:

λ> decode ((Data.ByteString.Lazy.Char8.pack "{\"name\":true}")) :: Maybe Person
Just (Person {name = True, age = False})

And the omitted fields are just given a False value. This would require you to request which fields you wanted in a JSON body rather than in the GET url.