Worker pool unification, iteration 2

[fenek]

Hi everyone,

I was thinking about unifying all outgoing connection pools in terms of configuration and reliability. Bartek Gorny already did the unified supervision and access parts. :)

Configuration

Schema

Current configuration model would be replaced with following schema:

{outgoing_pools, [
  {redis, host, default, PoolOpts, ConnOpts},
  {rdbms, host, read, PoolOpts, ConnOpts},
  {rdbms, host, [write, default], PoolOpts, ConnOpts},
  {riak, global, default, PoolOpts, ConnOpts},
  {cassandra, global, default, PoolOpts, ConnOpts},
  {elasticsearch, global, default, PoolOpts, ConnOpts},
  {http, host, auth, PoolOpts, ConnOpts},
  {http, global, default, PoolOpts, ConnOpts}
]}.

• First tuple element is the pool type. Used by mongoose_wpool to automatically call proper mongoose_wpool_[type] modules.
• Second element states whether one pool should be created per vhost or per whole node.
• Third element is the pool tag. Could be any atom with some limitations:
  • default - default (:)) pool used by various modules if not specified otherwise.
  • read and write are not reserved but by convention would be used to separate types of operations by various modules, e.g. mod_roster.
• PoolOpts are the pool parameters (size, strategy etc.)
• ConnOpts are the parameters used by the connection library itself.

Pool choice strategy

For now modules will expect precise configuration of pools. It means that if an admin configures mod_roster to use {global, default} for writes and {host, read} for reads, no other pools will be attempted automatically (could be a nice feature in future iterations though).

Automatic reconnection

Worker pool should automatically perform reconnections with various backoff strategies. Reconnection events would be recorded in metrics. Reconnection mechanism in actual connection library, if present, should be disabled.

Misc

Consistent naming scheme

All pools will have following names:

mongoose_wpool_[type]_[scope]_[tag] where [scope] is global or specific hostname.

Metrics

mongoose_wpool must provide a default set of metrics for every running pool:

Their names would be [global | Host, mongoose_wpool, type, Metric].

Following stats should be exposed:

• size
• total_message_queue_len
• connections - bumped by successful reconnection
• disconnections - not bumped by failed connection
• failed_connections - bumped by both ordinary connection and reconnection failure
• reconnection_attempts
• successful_requests
• failed_requests

Config reload

mongoose_wpool will expose a function that ejabberd_config will call upon config reload. It will delegate a resposibility of closing/opening proper pools in a smart way (no crude stop & start of everything).

supervisor2

supervisor2 would become obsolete, as its main benefit is an option of endless restarts with some cooldown.

[michalwski]

It all makes sens to me. I'd like to see this idea implemented (sooner than later). I have one comment for now regarding host specific pools. I'm not sure if the following is covered by what you described.

From my experience if a pool is host specific it also often means that the connection goes to different endpoints for some or every host. I imagine the config would look like the following then:

{redis, "domain1.my-service.com", default, PoolOpts, ConnOptsForDomain1},
{redis, "domain2.my-service.com", default, PoolOpts, ConnOptsForDomain2},

Where "domain1.my-service.com" and "domain2.my-service.com" are XMPP hosts served by MongooseIM.

[fenek]

Nope, I haven't covered it. Definitely a good idea. :) I think that for sake of flexibility, it should be possible to provide generic config as well and config handler should merge these automatically, so:

{redis, "domain1.my-service.com", default, PoolOpts, ConnOptsForDomain1},
{redis, host, default, PoolOpts, ConnOpts},
{redis, "domain2.my-service.com", default, PoolOpts, ConnOptsForDomain2},

Will start default redis pools for every served domain but for domain1 and domain2 they will have different configs. Does it make sense?

[michalwski]

Yes it does! Good idea.

[michalwski]

As far as I can tell only metrics and config reload are to be done. All the other parts of the idea were already implemented.