Dialer v2: modular and composable dialer

[raulk]

This PR deprecates #88.

Description

Here we introduce a Pipeline construction for the dialer. Copied from the godoc:

A Pipeline assembles a set of modular components, each with a clearly-delimited responsibility, into a dialing engine. Consumers can replace, delete, add components to customize the dialing engine's behavior.

The Pipeline comprises five components. We provide brief descriptions below. For more detail, refer to the respective godocs of each interface:

• Preparer: runs preparatory actions prior to the dial execution. Actions may include: deduplicating, populating timeouts, circuit breaking, validating the peer ID, etc.
• AddressResolver: populates the set of addresses for a peer, either from the peerstore and/or from other sources, such as a discovery process.
• Planner: schedules dials in time, responding to events from the environment, such as new addresses discovered, or dial jobs completing.
• Throttler: throttles dials based on resource usage or other factors.
• Executor: actually carries out the network dials.

Recommendations for review

Here's the recommended walkthrough to review this PR:

1. Read the package-level godocs on dial/doc.go for an overview of the solution.
2. Go through the godocs of the interfaces on dial/interfaces.go.
3. Explore the Request and Job entities in dial/request.go and dial/job.go.
4. Review the godocs on the Pipeline, as well as the pipeline logic itself in dial/pipeline.go.
5. Review the implementations of the interfaces in the rest of the files.

[raulk]

@stebalien @vyzo we probably want to start moving on this soon.

[raulk]

Calling out @vyzo @bigs specifically for a first round of review. A bunch of things are now depending on the dialler changes (self-dial, prioritising non-relay addresses, etc.), so we should move on this urgently to avoid dropping the ball.

---

@magik6k thanks for the review! 🎉

Would be nice to test in go-ipfs (sharness / real network / gateways)

Agree. I've so far tested with the DHT crawler, which is the most dial-intensive thing we have, and it looks good memory and CPU wise. But I agree we should integrate into IPFS, and test.

If we'd have a way to suspend pipelines, and an additional transport layer, we might be able to implement connection migration quite easily. (leave this for the future)

Could you elaborate on this? Suspending pipelines was on my mind but for a different use case (e.g. activating transports only if we need to dial peers with those transports). It allows for a more dynamic environment, but not an immediate requirement.

I like this, it seems to provide a good base for further development

Yeah, optional self-dial can now be added by removing the default Validator, and adding one that recognises self-dials and returns a singleton in-memory net.Pipe (see https://github.com/libp2p/go-libp2p/pull/638#pullrequestreview-242664051). Also, we can order, prioritise and stagger dials for transports we'd like to avoid (e.g. relay).

[raulk]

@bigs has stepped up to move this PR over the finish line :-)

[bigs]

@raulk @Stebalien just rebased this and updated the branch to reflect the core refactor, amongst other things. tests passing, this should be ready for a review.

[raulk]

Resuming this thread.

@vyzo:

I would like to have a mechanism to supply at a minimum the address resolver for individual dial jobs.

IMO, that would be too complex. We really don't want users of the swarm/network providing address resolvers per-dial; that leaks an abstraction. Instead, the address resolver can take contextual decisions based on the dial that it's processing.

[raulk]

If this becomes a common case, we introduce an higher-level AddressResolverDispatcher that conforms to the AddressResolver interface and proxies a bunch of address resolvers behind a set of predicates. The dispatcher evaluates predicates sequentially against the incoming dial request, and chooses the address resolver that matches, or errors if none match.