Adds URLRequestOption

[danthorpe]

Allows the framework consumer to attach (on a per request basis) "options" to requests which can be accessed inside the client.

For example, you might want to build a network client which supports advanced features such as de-duplication, throttling, caching, authentication, retrying. Typically these features would require some kind of configuration, for example, perhaps not all requests should be cached, or the retry strategy might be different for some endpoints. Therefore, we require a mechanism to specify request options to each Route, which we can retrieve inside the client. This is inspired by SwiftUI's EnvironmentValues, and from Dave de Long's blog post on Request Options.

First, framework consumers would define an option, lets say we want our client to cache response on a per request basis, we could define an option as follow:

enum CacheOption: URLRequestOption {
  static var defaultValue: Self = .always
  case always, never
}

enum ThrottleOption: URLRequestOption {
  static var defaultValue: Self = .always
  case always, never
}

The protocol, URLRequestOption is provided by the URLRouting library. Similar to how we use EnvironmentValues in SwiftUI, we can provide a convenience accessor:

extension ParserPrinter where Input == URLRequestData {
    func cacheOption(for route: Output) -> CacheOption {
        option(CacheOption.self, for: route)
    }

    func throttleOption(for route: Output) -> ThrottleOption {
        option(ThrottleOption.self, for: route)
    }
}

Inside our router, we can then specify any options, e.g.

static let router = OneOf {
  Route(.case(.dashboard)) {
    Path { "dashboard" }
    Options {
        CacheOption.never
        ThrottleOption.never
    }
  }
}

Finally, inside a custom URLClient, we can now query the value of this option for every request.

extension URLRoutingClient {
    public static func connection<R: ParserPrinter>(
        router: R,
        session: URLSession = .shared,
        decoder: JSONDecoder = .init()
    ) -> Self
    where R.Input == URLRequestData, R.Output == Route {
        Self.init(
            request: { route in
                // Check request options as part of fetching this route
                let cacheOption = router.cacheOption(for: route)

                // etc
            },
            decoder: decoder
        )
    }
}

This PR provides the mechanism to specify user-defined options as part of a Route, and then access them later via the router, or URLRequestData. It does not provide any built in URLRequestOption types, nor does it consume any inside the default URLClient which ships with the framework.

It is intended that framework consumers would likely define their own Route options for their own purposes, and so would likely have their own network stack/client.

Very happy to take any feedback or thoughts on this - especially how it works! And of course, the naming. Also, if there is a way for me to achieve this without changing the current framework - that would be great too. I have already tried composing URLRequestData inside my own URLRequest container which handles the options. This does not work very well, because ultimately, we need to specify the options as part of the Route.

[stephencelis]

@danthorpe Thanks for taking the time to discuss and submit this! I hadn't heard of this kind of abstraction before. I wonder, though, if we can achieve the same thing without changing the underlying data types.

For example, you can use baseRequestData to selectively apply headers and other request data to certain routes. If you want home and login routes to cache differently, for example:

func testOptions() throws {
  enum MyRoute {
    case home
    case login
  }

  let r = OneOf {
    Route(MyRoute.home) {
      Path { "home" }
    }
    .baseRequestData(.init(headers: ["Cache-Control": ["max-age=604800"]]))

    Route(MyRoute.login) {
      Path { "login" }
    }
    .baseRequestData(.init(headers: ["Cache-Control": ["no-cache"]]))
  }
  XCTAssertEqual(
    try r.request(for: .home).allHTTPHeaderFields,
    ["Cache-Control": "max-age=604800"]
  )
  XCTAssertEqual(
    try r.request(for: .login).allHTTPHeaderFields,
    ["Cache-Control": "no-cache"]
  )
}

This gives you a direct influence over the request data for printing, and no need to hold onto options just to be applied later.

Does this solve the problem for you?

[danthorpe]

👋 @stephencelis - thanks for the response!

Taking advantage of the base request data certainly merits exploring so I've had a play using that. A couple of points spring to mind..

1. For "options" which are well understood or defined by the HTTP specification, and are client/server agnostic, such as cache-control - this actually makes a lot of sense. 👍

2. But, for options which are not well defined, or specific to a client's requirements there are a couple of issues. Primarily that using HTTP Headers means losing any type information, and working with only Strings. While we could require every option value to be Codable, and put JSON in the HTTP Headers. But, I don't think this is what headers are intended to be used for. Secondly, options which only make sense to a client network stack, should not necessarily form part of the URLRequest and sent to the server.

For example, I've used option to define request numbers and identifiers which are private to my network stack, and used to trace requests in client side logs. (You've spoken about this recently discussing Thread dictionaries). Now, while that concept is likely something which a server would also want - it doesn't necessarily follow that the numbers or identifiers would be the same header key or value for both client and server.

Another example, which is pretty typical in a client-side network stack would be retrying failed requests. This is usually done with a configurable with a backoff strategy. This is a good example where the "option" attached to the routes would likely be more than just string value - but actually some kind of type, in this example it would determine the backoff interval. While we could encode this as some JSON into the request headers, should this be something that gets sent over to the server?

Does this make sense?

[stephencelis]

@danthorpe Sorry, but I'm still not sure I fully see things 😅 The examples you mention are all things I think I might handle outside of routing. I'm also having a hard time connecting server-side request identifiers to these "options" since this PR seems to only attach data to outgoing requests. How do you see these request identifiers being used once they are printed to URLRequestData? Could you sketch out a bit more code to show the problem being solved? As for retry strategies, that also seems like something that would be done at the API client level and not the router level.

One thing I'm really not understanding is that at the end of the day, the "options" attached to a printed request need to be transformed into actual request data, right? In fact, right now there are several helpers that print routes into more ergonomic types:

router.request(for:)  // (MyRoute) -> URLRequest
router.url(for:)      // (MyRoute) -> URL
router.path(for:)     // (MyRoute) -> String

These are helpers for Foundation types that simplify interfacing with URLSession, etc., but you can also imagine helpers for NIO and other libraries that have their own URL request types.

All of these helpers implicitly lose any options attached to a route, right?

[stephencelis]

@danthorpe Curious if you have any updates here? Are you still interested in pitching this addition? Or did you find it wasn't needed after all?

[danthorpe]

Hi @stephencelis - sorry for the delayed response, not really had much time to address this, so I will close this PR.

---

I still think there is an aspect of client side configurability missing from this package. Perhaps when you come to integrate it into Isowords you may come back to these ideas. When it comes down to practical usage of this framework, I think the "client" side of things might be a little bit too simplistic for real-world usage.

Before a request reaches a URLSession, the application would likely want to do things like check caches, remove any duplicate in-flight requests, throttle requests, retry failed requests, authentication etc. Each of these features has the same (Request) -> async throws Response shape. And if we define that shape through a protocol - say, NetworkComponent, we can build up "network stack" of components - similar to what you've shown with the Parser library, ReducerProtocol etc.

However, each of these components likely need their own "options" associated with a Request. Because, let's say, that some requests should never be throttled, or retried. But, we cannot pass arbitrary additional arguments using the (Request) -> async throws Request shape, unless we can somehow embed them into the Request.

Some component options might well translate into the HTTP spec naturally, and so we could encode them into the Headers. But, it doesn't actually make much sense that this information gets baked into the URLRequest and sent to the server, as it's purely a client side concern. Hence this PR just adds a type-safe way of associating options through some storage in URLRequestData.

I have a project which uses your URLRouting library in conjunction with the network stack described above, see how a high level Connection type is created here: https://github.com/danthorpe/danthorpe-networking/blob/main/Sources/Networking/Connection.swift#L55

At the call site, using this approach works something like this...

let network = NetworkStack
        .use(session: .shared)
        .throttle(max: 3)
        .retry()
        .cached(size: 3, fileName: "Network-Response-Cache")
        .removeDuplicates()
        .use(logger: logger)
let connection = Connection(router: myRouter, with: network)
let result = try await connection.value(for: .search(keywords: "foo"), as: SearchResult.self).body