Handle multiple sets of credentials with a single ApnsClient instance

[jchambers]

Now that we've introduced connection pooling in #492, the next big step will be to get clients to choose connections from different sub-pools depending on the topic to which a notification should be sent (and therefore which credentials the client should use). To get there, we'll probably need to introduce some abstract notion of ApnsCredentials that has subclasses/implementations for X509 certificates and PKCS#8 signing keys, and then extend the existing connection pool infrastructure to add "keyed" pool functionality.

The end result will be that callers will only need to have a single ApnsClient instance for all of their destination apps, which will probably be a big win for high-volume users.

[justinpermar]

+1 from a user who would benefit from this enhancement!

[ChunBoL]

I also encountered the problem of handling multiple sets of documents, when can I support multiple sets of documents?

[jchambers]

I'm starting in on some initial design work for this feature, and we're going to have to work around a few obstacles. The main ones right now are:

1. We have two possible types of credentials: X509 certificates for TLS-based authentication and EC private keys for token-based authentication.
2. Based on an X509 certificate's metadata, we can identify the APNs topics for which it can be used. We can't tell which topics a signing key is good for, though, and have to rely on callers to tell us.

The general plan in my mind is to expand PooledObjectFactory (which creates connections to the APNs server) to be a KeyedPooledObjectFactory that would look something like this:

interface KeyedPooledObjectFactory <K, T> {
    Future<T> create(K key, Promise<T> promise);
    Future<Void> destroy(K key, T object, Promise<Void> promise);
}

Elsewhere, we'd have an ApnsCredentialProvider:

public interface ApnsCredentialProvider {
    Future<TlsApnsCredentials> getTlsCredentials(String topic);
    Future<ApnsSigningKey> getSigningKey(String topic);
}

I'm trying to decide what we should use for keys in the KeyedPooledObjectFactory. We could build everything around topics, which would be reasonably simple and straightforward, but maaaaay introduce some unnecessary overhead in cases where people are sending to lots of topics with the same credentials. If we keyed based on credentials instead, we'd have connections allocated per credential set rather than per topic.

In other words, it might be the difference between this:

• com.example.app
  • connection 1
  • connection 2
• com.example.app.voip
  • connection 3
  • connection 4
• com.example.app.complication
  • connection 5
  • connection 6

…and this:

• Signing key for com.example.app.*
  • connection 1
  • connection 2

My question to you folks: is the many-topics-with-one-set-of-credentials scenario a use case you actually have, or is the overhead of channels-per-topic tolerable?

Separately, what kinds of pool configuration are you folks hoping for? The simplest thing would be to just use a single limit like maxConnectionsPerTopic or maxConnectionsPerCredentialSet, but that might not be ideal in cases where some topics/credentials are much busier than others.

Cheers!

[jchambers]

Oh, right, and to close out this thought:

Based on an X509 certificate's metadata, we can identify the APNs topics for which it can be used. We can't tell which topics a signing key is good for, though, and have to rely on callers to tell us.

…right now, the situation is that the caller gives us a signing key and is free to at least try to send notifications to whichever topic they'd like, even if that topic isn't covered by the signing key. We rely on the APNs server to tell us if it's not the right key. In a world where we might have multiple credentials associated with a client, we couldn't get away with that any more because the client would need some mechanism to decide which credentials to use based on the topic.

The only real option I can think of in the general case is to make callers tell us which topics are associated with which keys. To be fair, we already do this for key IDs and team IDs, but at least those are clearly spelled out in the Apple Developer console. To my knowledge, there's no way to get a full list of all of the topics a key supports. Does anybody have other suggestions?

For a backward-compatibility hack, I'm thinking that we could have ApnsSigningKey implement ApnsCredentialProvider and have getSigningKey return the signing key itself unconditionally. That would allow callers who want to use a single key (I'm guessing that's a very common use case) to avoid listing out all of the topics to which they might ever want to send notifications.

[jacobwallace]

Any rough idea when this might be available @jchambers? Huge thank you for all your work on Pushy.

[jchambers]

It's the next major feature planned, but there's isn't a specific timeline for release.

[qiankunli]

+1 from a user who would benefit from this enhancement! I am looking forward to it.

[aptituz]

Just some cents from a user about the KeyedPooledObjectFactory topic above:

We could build everything around topics, which would be reasonably simple and straightforward, but maaaaay introduce some unnecessary overhead in cases where people are sending to lots of topics with the same credentials. If we keyed based on credentials instead, we'd have connections allocated per credential set rather than per topic.

We are one of those users having several topics with the same keys. Thus, I think it wouldn't be ideal to have connection pools per topic (but of course I may be biased here). In case its of any worth for you: in our service we have the concept of an app-group and topics are mapped to app-groups. Maybe a similar concept would be helpful for pushy?

[jchambers]

Yikes. Hard to believe this issue is now more than six years old!

I've made a lot of progress toward actually implementing this in #948 (though there's a good deal of in-progress work I haven't pushed, too), and that process has given me a much clearer picture of what this will all look like in its final form. And with that clearer picture, I'm no longer convinced this is a good idea. In short, I think implementing multi-credential support in Pushy itself will introduce new complexity and performance penalties for simple use cases while simultaneously failing to satisfy everybody who wants this feature for industrial use.

On performance/complexity: I think we'd ultimately need to do an internal topic -> credentials -> channel pool -> channel chain for every notification. Much of that would be hidden from end users, but they'd still need to pay the cost of the additional lookups on each notification. Honestly, I suspect this wouldn't be that bad at steady-state, but it'd still be there. All that indirection means that there's more that can go wrong in more places, though, and some things we can check at construction time now would wind up getting deferred to "send notification" time in a multi-credential world. The API surface area would expand with a lot of stuff that most users don't need, too.

For industrial users, I think there are enough tradeoffs to make that nothing I could build into Pushy would be the right answer for everybody. For example: is it necessary to provide affordances for "unloading" credentials? What should happen when we do so? Should all topics/credential sets get the same number of open connections? Do we need to build in global connection limiting?

We could probably build something for power users that has tons and tons of configuration options, but at that point, I worry that working with Pushy would become a major chore. And, ultimately, I think we have to weigh that against just letting folks do something like, say:

Map<String, ApnsClient> apnsClientsByCustomerId;

Does that shift some burden to industrial users? Yes, for sure. But at the same time, that seems pretty okay to me. That keeps Pushy simple and puts the burden for customization on the users best-equipped to do that work.

Another way I'm thinking about things: I don't think there's anything in Pushy right now that prevents industrial users from doing what they want/need to do now. Adding multi-credential support wouldn't introduce any fundamentally new capabilities even if it might make some things more convenient for a relatively small number of users. It's absolutely valid for a project to try to tackle convenience features for power users, but at some point, I think that's where projects cross the line from being "libraries" to being "products." Personally, I think Pushy makes a lot more sense as a library.

My concrete proposal, then, is to close this issue as "won't do" and instead:

1. Bolster documentation about best practices for multi-credential use cases
2. Clear out the backlog of issues that have been blocked on multi-credential support for some time

I recognize this may be controversial, and I welcome feedback. If I don't hear anything from anybody in the next week or two, I'll move forward with the "won't fix" plan.

Thank you all for your patience and input so far!

[jchambers]

If I don't hear anything from anybody in the next week or two, I'll move forward with the "won't fix" plan.

Okay; let's move forward with this plan, then. I'll start moving in the direction of "make it more obvious how to use Pushy in its current form for multi-credential scenarios."