web3.js: (draft) Principles for a rewrite

[steveluscher]

Problem

I believe that the appetite for a rethink of the web3.js library's architecture has reached a tipping point.

• @armaniferrante: (tweet)
• @0xCryptoSheik: (tweet, draft pr)
• @jacobcreech (tweet)
• @johnrees (tweet)

These are some of the problems that a rewrite would aim to solve:

1. The current monolithic design means that the library delivers poorly on functionality-per-byte. We force users to download code that they are likely never to need, and can't tree-shake away.
2. Opaque helpers like sendTransaction make it difficult for people to customize behavior. They both assume too much responsibility (ie. do too much, like fetching recent blockhashes and signing transactions) and mutate their inputs (eg. calling sendTransaction overwrites a transaction's ‘recent blockhash’).
3. The use of JavaScript classes to encapsulate state thwarts our best efforts to create typesafe applications. For example, what's the difference between a signed transaction, an unsigned transaction, and a nonce-based transaction? In the current library, these are just Transaction instances that may or may not be in the configuration you hope, and may throw runtime errors if they're not.

Proposed Solution

This isn't even a bad idea.

I've experienced a lot of ground-up rewrites in my time, and the approach that I've known to consistently end in success is:

1. Create the ideal API.
2. Replace the implementation of the existing API using the new one under the hood, incrementally.
3. Once you've proven that you can reimplement the legacy API using the modern one, freeze development on the legacy API and focus on giving people tooling and tutorials to migrate to the modern one.

Principles

What follows are a set of principles that I believe should inform and guide the rewrite.

Principle 1: Let data be data

Unless a parcel of data needs to mutate itself in response to events, do not wrap it in a JavaScript class.

1. We should take pains to treat data as immutable. Prefer the use of the Readonly and ReadonlyArray types in TypeScript.
2. Operations over data should take immutable data structures as input and produce, as output, immutable data structures that share as much structure with the originals as possible.
3. Data structures should be well typed. The type of a data structure should tell you everything you need to know, statically, about which operations it's compatible with.

Principle 2: Perform work by transforming data using functions

Data should not be mutated in place nor should it be expressed as a JavaScript class that mutates its own private state. Instead, operations (eg. signing a transaction, serializing a transaction, adding an instruction to a transaction, converting a public key between formats) should be performed by invoking functions over immutable data to produce new immutable data.

// Instead of mutating the internal state of JavaScript class instances...
const transaction = new Transaction(/* ... */);
transaction.sign(/* ... */);

// Perform operations by transforming data using functions.
const transaction = createTransaction(/* ... */);
const signedTransaction: ISignedTransaction = signTransaction(transaction, /* ... */);

Principle 3: Importing a module must never produce a side effect

The top level of an ESModule is called its ‘module factory.’ Anything at the top level runs immediately the moment the module is imported from another module.

We must never do any work in a module factory other than defining and exporting types, primitive values, JavaScript classes, and functions. We must never call a function at the top level, nor access a property of an object. We must enforce this with lint rules and with a check at the CI step.

1. This will ensure that the library is fully tree-shakeable (ie. unused code can be deleted by an optimizing compiler)
2. This will make the library more ‘lazy,’ which is to say code will compile and run as it's invoked rather than as it's imported.

Principle 4: Use opaque types to guarantee runtime contracts

After making certain assertions about a value, return it in its most primitive form but cast to an opaque type

For instance, once you've asserted that an array of numbers is a valid public key, cast it to an opaque PublicKey type. This makes it so that you can pass around a primitive value while at the same time enforcing runtime guarantees about its compatibility with various operations.

Imagine asserting that a string is definitely a base58 encoded pubkey, then casting it to an opaque Base58EncodedPublicKey TypeScript type. Now that value comes with a guarantee that you can deserialize it back into a array that conforms to the PublicKey type, or use it as an input to an RPC call – and all without using JavaScript classes.

Principle 5: Extreme modularization

Break the library up into as many ES modules as practical. Test them independently.

End users may choose a monolithic import style compatible with a tree-shaking compiler (eg. import {createPublicKey} from '@solana/web3.js';) or they may craft a custom build by importing ES modules directly (eg. import createPublicKey from '@solana/web3.js/modules/createPublicKey.js';)

Principle 6: Program wrappers belong in their own packages

Wherever possible, kick program wrappers (eg. vote-program) into their own npm modules that are tested and published separately.

1. This should reduce CI time in cases where only the changed package needs to be retested.
2. People who don't use a program wrapper don't have to download it.

Principle 7: All errors must be coded

Never throw errors with freeform text. Always throw typed or coded errors, so that people can make airtight assertions about the nature of what went wrong in their catch blocks.

Principle 8: All asynchronous operations and subscriptions must be cancellable

Never offer an asynchronous operation without also offering a way to cancel it.

1. Subscriptions must always return a dispose handle.
2. Asynchronous operations must always accept and respond to an AbortController.

// You can later call `dispose()` to cancel this subscription.
const {dispose} = makeSubscription('accountChange', /* ... */);

// You can at any time call `abortController.abort()` to cancel this promise.
const abortController = new AbortController();
await confirmTransaction(/* ... */, {abortController});

Principle 9: Minimize over-requires in function inputs

A function should make as efficient the use of its input arguments as possible. That is to say that it should not require as input anything that it doesn't make use of.

Instead of this:

function doThing(
  connection: Connection  // Way more information than we need.
) {
  const commitment = connection.commitment;
  // Do something with `commitment`
}

Do this:

function doThing(commitment: Commitment) {
  // Do something with `commitment`
}

h/t @tg44

Principle 10: Debuggability

Produce a debug build that produces warnings and messages that otherwise get stripped out in production mode through dead-code elimination.

if (__DEV__) {
  log(...);  // Gets stripped out when `__DEV__` is `false` in production mode.
}

Under consideration: Allow developers to inject a custom logger, in development or production. Supply a default implementation of this logger that logs to the console. Examples of logs this logger might produce include outgoing RPC calls and incoming RPC subscription notifications.

h/t @swertin

Principle 11: Avoid JavaScript numbers

JavaScript numbers can express integer values up to 2^53 - 1. If you want to express a value higher than that you must express it either as a bigint or a string.

• When communicating with a web API (eg. the JSON RPC) always use strings to avoid truncation
• When storing and performing arithmetic on values in memory always use bigint