RFC for embedded static resource support

[wuzzeb]

Servant already uses wai-app-static for the very simple serveDirectory which just produces a Server Raw. This isn't ideal because the routes for individual files are not represented in the type. Also, wai-app-static has the more sophisticated ability to embed resources into the executable at compile time (full disclosure, I was the original author of this code in wai-app-static although since then several people have contributed to wai-app-static).

This pull request is my RFC for an approach to extend serveDirectory to be able to specify individual resources in the API type for better exposure of the static resources that are available. Along the way, I also add support for embedding the static content at compile time.

type MyAPI = "static" :> "js" :> "bootstrap.js" :> EmbeddedContent "application/javascript"
        :<|> "static" :> "css" :> "bootstrap.css" :> EmbeddedContent "text/css"
        :<|> "static" :> "css" :> "mysite.css" :> EmbeddedContent "text/css"

At the moment I added all this into a new project, but that was just for ease of testing. If you like this approach, some of the types should be moved into Servant.API, other code merged into servant-server, and perhaps the more exotic generators such as compiling with lesscss and ghcjs moved into a new project. I can morph this code around once I get some feedback.

I went through about 4 iterations on the design before I found one I was happy with. My first iteration used wai-app-static directly but it had to do A LOT with template haskell and as always using as little template haskell as possible is good. I then realized that we only need a few internal functions from wai-app-static, so the later evolutions of the design don't depend on wai-app-static anymore. I then went through several iterations on how the combinators look. I have started using it in my own servant-based application and the combinators are working well, but I'm still open to improvements.

I wrote haddock documentation, so for more details you should see the documentation, especially the haddock comment in Servant.Server.Embedded.hs

[3noch]

:+1: Really cool feature! Having totally self-contained web-apps is a very rare thing indeed!

[jkarni]

Wow, great work!

An issue with having it as a separate package is that, if it's to have instances for all the core classes, it'll need to depend on all the core packages, making e.g. wai a transitive dependency of client code that uses it.

One thing we've been talking about is starting a servant-contrib set of packages. @haskell-servant/maintainers would we want it there or in the core packages? I personally think this will be useful enough to have in the core ones. On the other hand, it's perhaps good practice to start the -contrib packages and have new combinators go there, and only then move them, so we have (a) some time to be more aggressive about release cycles and breaking API changes as we try things out, and (b) give people who are writing other interpreter classes (e.g. servant-swagger) some time to know what's in the pipeline.

[utdemir]

+1 for me for self-contained web apps, and compile-time file embedding would certainly be very useful in servant-server. But I'm not sure if it belongs to the API definition, because it looks like an implementation detail of server (I don't think clients or documentation generators care about if it's generated on-the-fly or on compile time.).

But, I think the part minifiers, compressors, CSS transpilers etc. belong to a different package, as you said.

And maybe it's worth to talk about the general idea of "serving static files" here. For example, on our API's, we were serving the JS client of our server on some endpoint. But in order to set Content-Type header for them, we had to define the type as Raw:

myApi :: "client.js" :> Raw
serve :: Application
serve req resp = resp $ responseLBS status200 [("Content-type", "text/javascript")]
   (BL.fromStrict . T.encodeUtf8 $ jsForApi myApi)

In order to simplify this, I wrote these instances (simplified):

data ContentType (typeName :: Symbol) (subtypeName :: Symbol)

instance Accept (ContentType typeName subtypeName) 
instance MimeRender (ContentType t s) ByteString 
instance MimeRender (ContentType t s) Text 
...

And then I was able to define API's like this

myApi :: "client.js" :> Get '[ContentType "application" "javascript"] Text
serve :: EitherT ServantError IO Text
serve = return $ jsForAPI myApi

I told you about this, because I think the TH file embedding also fits here as a separate type, something like:

instance MimeRender (ContentType t s) EmbeddedContent where
myApi :: "somejs" :> Get '[ContentType "application" "javascript"] EmbeddedContent
serve = embedFile "./somejs.js"

Maybe this RFC can cover serving static content (but not from an external file) with a specific content-type too.

[soenkehahn]

@wuzzeb: I very much like the idea of embedding static files into the executable. I've been using that feature from wai-app-static and it's awesome.

I do wonder however: what is the advantage of having all the static files represented in the type of the API? What serveDirectory was aimed at is use-cases that follow this pattern:

type MyJsonRestAPI =
  ... -- lots of JSON REST routes

type MyServerAPI =
  MyJsonRestApi :<|>
  Raw -- to serve some html, css and javascript that uses the API

I don't see how this would benefit from having all the static files explicitly represented in the type? I.e. I probably don't need to be able to generate type safe client functions to fetch all the static assets individually. Also I don't want all the css files to show up in generated documentation, be it by servant-docs or servant-swagger. Am I overlooking an important use-case?

These static file combinators would also force you to manually keep them in sync with the files on your disk. What I like very much about serveDirectory is that you can just throw a new file in the directory and it'll be automatically served.

Generally I'm wondering if it wouldn't be good to solve these things purely in wai. serveDirectory does not depend on anything in servant at all. (The implementation is just based on wai-app-static and although the type is given as FilePath -> Server Raw that's just fancy talk for FilePath -> Application.) Maybe it should be moved to wai-app-static (and re-exported in servant?). And maybe we do want something like serveDirectoryEmbedded. That way not only servant users, but all wai users would benefit.

[soenkehahn]

What I could imagine is a combinator like this:

type MyApi =
  MyJsonRestApi :<|>
  "static" :> StaticAssets

instance HasServer StaticAssets where
  type Server StaticAssets = StaticAssetsConfig
  route = ...

data StaticAssetsConfig
  = FromDisk FilePath
  | Embedded Embedded -- from wai-app-static

If I think about the different interpretations, that feels like the right amount of information that should be in the API specification. So generated documentation can say something like:

/static (and subpaths) - static assets like html, css, javascript, images, etc.

And we could have:

instance HasClient StaticAssets where
  type Client StaticAssets = FilePath -> ExcepT ServantError IO ByteString

So that you can easily generate a client function that allows to fetch assets like this: getStaticAssets "index.html".

[jkarni]

One obvious advantage is type-safe links to static assets.

Maybe the solution is explicit routes, but with TH to generate them from a directory?

[soenkehahn]

But where would you want to have type-safe links to static assets? I guess if you're generating html files within haskell... Personally I would prefer to have this functionality in a templating library or an html generation library and de-coupled from servant. Maybe that also already exists.

[codedmart]

I tend to agree with @soenkehahn on this. Awesome work from @wuzzeb, but I don't see the advantages of this yet to be in core. Seems more like a specified use case that would be great as a contrib lib that someone can pull in when needed.

[wuzzeb]

Sorry, was busy the past few days.

My first approach was much closer to serveDirectory. I didn't call it serveDirectoryEmbedded but that was essentially what it was. I still have the code around in my git history (not on github, for github I did a rebase) so we could go to it if you like. The main reason I changed to include it all in the route is ETags and the HasLink instance.

For static resources, the recommendation is to tell the browser to cache them and then change the URL used to access. For example, see here for example. This means that when the server refers to a static resource, it must change the URL each time the content changes. wai-app-static does that via including the ETag as a query-param. What I mean is the server must generate a link such as

<link rel="stylesheet" href="/static/bootstrap.css?etag=12345678"/>

and the etag must change each time the content changes. So in order for the server to generate such links, it needs to know the computed etag. With my first approach which used serveDirectoryEmbedded, the template haskell had to create a variable containing a string with the etag, but then that string had to be combined with the full path. Bringing each resource out into the actual route allows HasLink instances which easily allow the server to generate the correct link with the correct etag.

I agree, if you didn't care about etags and proper caching, a solution which hid all the resources behind a single type in the servant type would be the best approach.

[jkarni]

Changing the url on file changes for better caching would be amazing - I've been impressed by that feature of yesod/wai-app-static, and would be pretty happy to see it in servant too.

[soenkehahn]

@wuzzeb: Btw, I would love to see something like serveDirectoryEmbedded as a contribution to wai-app-static.

[soenkehahn]

It seems to me that perfect caching (with etag query params, like @wuzzeb describes) is ideally done in a separate library on top of wai. I think the fact that HasLink does not provide perfect caching out of the box is not worth the complexity at the type-level.

I'm in favor of not merging (and closing) this PR.

If however HasLink doesn't play nice with an external library that supports perfect caching, then we should fix that, of course.

[rashadg1030]

Will this ever be merged into master? Seems like something I could use on a project I'm working on.