koala
koala copied to clipboard
arsduo
ETags
Hey @arsduo, I'm trying to get ETags to work via Koala and not having any success.
Intro
Facebook supports ETags, as of January 2012.
I can fetch and check ETags via curl as explained in the docs linked above.
curl -i https://graph.facebook.com/84398256230_10151738382101231/comments
However, handling them via Koala is failing. The issue is if you want to be able to process the headers you need to send options.merge(http_component: :headers), which lops off the body. That means to use ETags to check if anything is new means making two calls. However, the use case for ETags in the first place is fewer calls.
More details
This will return the headers but not the body:
api.graph_call(path, args, "get", options.merge(http_component: :headers))
These two calls return the headers and the body but .graph_call turns the response into a GraphCollection immediately if the collection is pageable: ("object_id/comments" for example.)
api.graph_call(path, args, "get", options.merge(headers: etag_header).merge(http_components: [:headers, :body]))
api.graph_call(path, args, "get", {})
What's happening is while the response itself does include the headers, they get tossed when turned into a GraphCollection – which happens whenever there is a collection to paginate. That means you can't snag the ETag in the same request as the collection itself. Fetching the ETag will take an extra call with graph_call, eliminating the efficiencies gained from using ETags.
What would be great is to pull the ETag from the headers on every graph_call before any processing and return that as a part of the object.
The ETag header can be passed to future requests via the options hash, but may be worth eventually integrating into the various method calls, as everyone should be using them.
Action
This is something I can take on in the next couple of weeks. Do you have any preliminary suggestions or work arounds that make the work unnecessary?
@jbielick we implemented Faraday middleware to deal with etags. Works well with Koala. I found this as a jumping off point and went from there. YMMV.
@garrettd714 wow, that's interesting —a very holistic approach. Thanks for sharing. I think I envisioned persisting e-tags to my database so that fetching the objects a great deal later (syncing remote / local objects) could use the etag and the 304 would be an indicator if there were any changes to sync since the last fetch. That sound crazy?
@jbielick doesn't sound crazy. That is actually what we do. We store the :request_url => etag in short-term storage. We have the middleware scoped to only check certain urls and :get requests, so not every request is evaluated. If etag matches, then we stop processing (no need to sync). So we don't employ a "standard" etag implementation; we don't store the result. We simply want to know early if there is a change or not. Works well for syncing objects and fetching/backfilling stats.
I ran into this issue as well. Opened a PR that returns a specific response when etags passed in get a cache hit: https://github.com/arsduo/koala/pull/453
Also working on a PR that will return the etags as a headers attribute on a GraphCollection instance, since it seems like any time you are working with etags you'd want to be able to access etags for a request in addition to just passing them in. Will open that PR soon, too.
