ably
Experimental v2/channel and consumer group
Labelled draft as not intended to merge. We will create a special release tag from this branch for customers requiring preview access to this functionality.
Adds a client-side simulation of channel + consumer groups functionality.
A channel group identifies a set of channels using a regular expression. The client SDK is responsible for subscribing to channels that match the regex as they become active and unsubscribing as they become inactive. Active channels are made known to the client via a dedicated channel named $ably:active which for the POC is published to by a lambda integration rule.
The set of channels in the channel group can be partitioned across a set of consumers that share the same consumerGroup.name. Consumers become aware of one another via presence on a channel with the same name as the consumer group. Channels are partitioned according to a simple modulo hash based partitioning scheme.
// subscribes to all active channels matching `foo:*`
const group = client.channelGroups.get('foo:.*', { consumerGroup: { name: 'group' } });
await group.subscribe((channel, msg) => console.log(`received message ${msg.name} on ${channel}`));
This implementation is intended to demonstrate the channel/consumer groups functionality and is not intended for usage in production. In a production-ready implementation, channel and consumer groups will be implemented in the backend realtime system and the client implementation will be much simpler. However this PR is indicative of what the public API would look like (except for some clearly-labelled simulation-specific configuration).
@VeskeR I pushed up some changes to tests. They pass reliably for me locally so not sure why we're getting issues in CI. Will wait for the run to finish and inspect the logs
Still failed after 100+ minutes unfortunately. Hope you will be able to load some logs, there are a lot.
Quickly looked at previous PRs: seems like it started happening with this PR and onwards: https://github.com/ably/ably-js/pull/1655 (see its CI result https://github.com/ably/ably-js/actions/runs/8109535381/job/22164928744). Probably something was added to the tests that is causing an infinite loop of some sorts.
PRs before that had their tests passing in CI. For example, https://github.com/ably/ably-js/pull/1651 precedes the PR above and its tests either pass, or fail in a reasonable time (~10 minutes)
From the logs, there's a lot of
2024-03-05T13:47:13.6564779Z 13:47:13.656 Ably: Connection state: failed; reason: [ErrorInfo: account restricted (connection limit exceeded); statusCode=401; code=40111; see https://help.ably.io/error/40111 ]
So I have bumped the default connection limits for the test account from 100 to 200, and will see what effect that has.
Actually, looks like there's a lot of
3-06T10:36:33.3326370Z 10:36:33.332 Ably: Transport.onProtocolMessage(): received on web_socket: [ProtocolMessage; action=DISCONNECTED; error=[ErrorInfo: Key/token status changed (expire); statusCode=401; code=40142; see https://help.ably.io/error/40142 ]]; connectionId = IK5jju0wpn
2024-03-06T10:36:33.3330234Z 10:36:33.332 Ably: Transport.onDisconnect(): err = [ErrorInfo: Key/token status changed (expire); statusCode=401; code=40142; see https://help.ably.io/error/40142 ]
Which is causing a large number of connections to be terminated and reopened. Looking into it
@ttypic @VeskeR Tests are now passing. The problem was due to some shared auth state causing conflicts between simultaneous usage of the two BaseRealtime instances. This was causing a large amount of connection churn due to auth issues. Either way, having separate ConnectionManager instances meant double the volume of connections even when not using channel groups. So, I've made changes to use the same Channels object from the ChannelGroup as well as the base client and prevent conflicts using a dummy channel option so that the library treats channels created by the channel group independently.