discord.js-modules icon indicating copy to clipboard operation
discord.js-modules copied to clipboard

feat: added `@discordjs/sharder` module

Open kyranet opened this issue 3 years ago • 1 comments

Please describe the changes this PR makes and why it should be merged:

I heard you wanted a nice and complete sharder. :eyes:

Sadly, this system is not compatible with [email protected], but it will be for a future version! This sharder draws inspiration from the strategy and plug-in system Sapphire has largely adopted over the months, the way Kurasuta handles messages and shards separately, and the structure of Discord.js's built-in sharding manager to make the update easy, alongside a few sprinkles of my own based on my experience when building a cross-process intercommunication system.

:warning: This PR is under heavy WIP: docs and tests are missing, they'll be done as I progress. The API may also change as new needs arise when building more code.

Let's talk about this sharder's structure! It is divided in three components:

  • The Sharding Manager: it's the central class that handles the shards, from starting them up, to closing.
  • The Shards: they're strategies that define how the "channel" should be handled from the sharding manager, they can be of any kind, from workers (WorkerShardHandler) and forks (ForkProcessShardHandler) to cross-server clusters going thru local clusters (ClusterProcessShardHandler). Yes, you heard that. Although we'll most likely ship only the 3 local ones. You can make your own custom shard strategies.
  • The Messengers: they're also strategies, but they define how data must be serialized and/or deserialized as well as handling message queues, we'll provide RawMessageHandler (data as-is, without converting to a Buffer or a string), JsonMessageHandler (JSON), and BinaryMessageHandler (V8 serder)... You can plug in your own ones, such as the Erlang External Term Format (ETF), YAML, TOML, etc.

This clear separation between responsibilities allow for greater customization of the core functionality of the library, reducing the need to use external libraries.

And coming in the future, we'll probably release The Message Mappers, which give a final touch-up to a message, such as appending a header/footer or even, using OpenSSL to encrypt messages over the network.

Issues to handle:

  • DONE https://github.com/discordjs/discord.js/issues/2882: I'll change respawn: boolean to respawn: number, accepting -1 and Infinity for unlimited respawns, but a number that is decreased by one on each shard until it's ready, in which case it'll reset the counter.
  • https://github.com/discordjs/discord.js/issues/3560: I had an eureka moment here, it'll require a small refactor in Discord.js's WebSocket system but... turns out we can query /gateway/bot in the sharder, only once, and pass the data to the shards during spawn. We could also implement a query system so we can get that data from the ShardingManager without querying Discord more than once a day by caching and updating the result accordingly.
  • https://github.com/discordjs/discord.js/issues/3890: See above.
  • https://github.com/discordjs/discord.js/issues/4095: Status unknown.
  • https://github.com/discordjs/discord.js/issues/5364: The new ready handler should be a lot cleaner and far more efficient, but we'll see.
  • https://github.com/discordjs/discord.js/issues/5581: Can it happen with this sharder? Let's find out.

Todo:

  • [x] Consolidate the sharder's API, make everything more consistent.
  • [x] Make ProcessShardHandler abstract, for the two classes below.
  • [x] Implement ForkProcessShardHandler.
  • [x] Implement CusterProcessShardHandler.
  • [ ] Implement WorkerShardHandler.
  • [x] Implement RawMessageHandler.
  • [x] Implement BinaryMessageHandler.
  • [x] Implement built-in cluster sharding (e.g. 2 processes with 5 shards each to have 10 shards) support.
  • [x] Implement fetchRecommendedShards.
  • [ ] Implement shard client to interact with the shard manager, ShardTunnel? ShardClient?
  • [ ] Type the events.
  • [ ] Add documentation.
  • [ ] Add tests.

Status and versioning classification:

  • I know how to update typings and have done so, or typings don't need updating
  • This PR changes the library's interface (methods or parameters added)

kyranet avatar Dec 02 '21 23:12 kyranet

Codecov Report

Merging #87 (daf9d67) into main (c22081d) will decrease coverage by 0.08%. The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #87      +/-   ##
==========================================
- Coverage   89.03%   88.95%   -0.09%     
==========================================
  Files           9        9              
  Lines        1487     1566      +79     
  Branches      153      162       +9     
==========================================
+ Hits         1324     1393      +69     
- Misses         12       13       +1     
- Partials      151      160       +9     
Impacted Files Coverage Δ
packages/rest/src/lib/REST.ts 95.57% <0.00%> (-0.98%) :arrow_down:
packages/rest/src/lib/errors/DiscordAPIError.ts 81.81% <0.00%> (-0.66%) :arrow_down:
packages/rest/src/lib/RequestManager.ts 93.46% <0.00%> (-0.50%) :arrow_down:
packages/rest/src/lib/utils/constants.ts 100.00% <0.00%> (ø)
...ackages/rest/src/lib/handlers/SequentialHandler.ts 82.98% <0.00%> (+0.54%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c22081d...daf9d67. Read the comment docs.

codecov[bot] avatar Dec 02 '21 23:12 codecov[bot]