workerd icon indicating copy to clipboard operation
workerd copied to clipboard
verified publisher icon cloudflare

Implement built-in node:net impl

Open jasnell opened this issue 1 year ago • 5 comments

Implement a subset of the node:net and node:tls modules supporting net.Socket, tls.TLSSocket, net.BlockList, and net.SocketAddress. The net.Server and tls.TLSServer types are explicitly unsupported and won't be implemented.

Update: The key next step on this is implementing tests... Folks should feel free to start performing code review on the main piece. If folks would like a walkthrough to help with the code review, let me know

Very rough todo list:

  • [x] Get it basically working ... ;-) ... basic implementation of the core functions. Difficult to delineate all the specific tasks involved here.
    • [x] net.connect(...) creates the Socket, normalizes the arguments, and forwards the args to socket.connect(...)
    • [x] socket.connect(...) validates the input arguments, initiates the underlying socket connection
    • [x] new Socket([options]) works
      • [x] options.allowHalfOpen works and passes through to the underlying Socket
      • [x] options.fd throws an error if set
      • [x] options.readable is ignored since options.fd is unsupported
      • [x] options.writable is ignored since options.fd in unsupported
      • [x] options.signal sets an AbortSignal that destroys the net.Socket when triggered
      • [x] ~~Setting options.handle to an existing standard Socket instance should wrap that socket~~ Will implement this separately as it's not critical for immediate compat in the short term
    • [x] Initialize and establish the connection
    • [x] connect and ready events are emitted when the connection is opened
    • [x] close event is emitted when destroy is called
    • [x] close event hadError argument is true when destroy is called with an error
    • [x] error event is emitted when destroy is called with an error
    • [x] destroy is called with an error when attempt to establish a connection fails
    • [x] connectionAttempt event is emitted when initializing a connection
    • [x] connectionAttemptFailed event is emitted when initializing a connection fails
    • [x] connectionAttemptTimeout event will not be emitted since the family autoselection feature is not implemented
    • [x] The addressType argument in the connectAttempt and connectionAttmptFailed events is the correct type
    • [x] Writes while connecting are buffered until connection established, then buffered writes are flushed
    • [x] Writev works
    • [x] Writes can be corked/uncorked any time
    • [x] Writes can be encoded strings or TypedArrays, other types throw
    • [x] Individual write callbacks are called. Error argument is set appropriately when write fails
    • [x] Calling end() closes the connection
    • [x] The drain event is emitted when the write buffer is empty
    • [x] When end() is called, buffered data in the readable side is still avaialble
    • [x] If options.lookup is provided and destination address is not an IP address, the options.lookup function should be called to "resolve" the address. (optional... we could just as easily choose not to support options.lookup)
    • [x] The lookup event is emitted after the options.lookup is called and returns a result
    • [x] Underlying reader on the socket is a BYOB reader reusing the user provided buffer or our auto allocated buffer
    • [x] An error while reading from the underlying socket causes the net.Socket to be destroyed with an error
    • [x] The net.Socket can have a read event attached putting the stream in flowing mode
    • [x] If the read event is attached while the connection is still pending, the read will automatically start when the connection is established.
    • [x] The stream can be paused/resumed to stop/restart the flow of data
    • [x] Attempting to connect to a pipe path fails
    • [x] The data event is emitted when a chunk of data has been read
    • [x] The end event is emitted when the underlying stream readable side is done
    • [x] Reading honors backpressure signaling by pausing
    • [x] Receiving and EOS from the remote should end the readable side of the stream. If allowHalfOpen is false, this should close and destroy the socket ONLY once the write buffer is drained. If allowHalfOpen is true, it should not.
    • [x] socket.setTimeout(...) sets an activity timer on the socket.
    • [x] The socket timeout is reset on writes, reads, and when the connection is established.
    • [x] The timeout event is emitted when the activity timeout expires.
    • [x] socket.address() returns the local address if the socket is connected, undefined otherwise... we hard code this to 0.0.0.0:0 since we have no notion of a local bound address in workers
    • [x] socket.autoSelectFamilyAttemptedAddresses is always an array, when connect is called, it always just contains the one address we're connecting to. This is generally non-op since we do not implement family autoselection
    • [x] socket.bufferSize returns the number of bytes buffered
    • [x] socket.bytesRead returns the number of bytes read
    • [x] socket.bytesWritten returns the number of bytes written
    • [x] socket.connect(options[, connectListener]) works
      • [x] Passing an ipv6 address works
      • [x] options.autoSelectFamily only accepts a falsy value. truthy values throw
      • [x] options.autoSelectFamilyAttemptTimeout is validated to be a number but is otherwise ignored
      • [x] options.family is verified to match the host specified
      • [x] options.hints is ignored unless options.lookup is given, then it is passed to that function when called
      • [x] options.host is the address to connect to. Defaults to localhost.
      • [x] options.keepAlive only falsy values are accepted. Truthy values throw
      • [x] options.keepAliveInitialDelay is validated to be a number but is otherwise ignored
      • [x] options.localAddress is ignored
      • [x] options.localPort is ignored
      • [x] options.lookup is used if the host is not a valid IP address
      • [x] options.noDelay falsy values are accepted. Truthy values throw
      • [x] options.port is validated to be a proper port
      • [x] options.path throws
      • [x] options.onread is used if specified
    • [x] socket.connect(path[, connectListener]) throws
    • [x] socket.connect(port[, host[, connectListener]) works
    • [x] socket.connecting is true while the connection is being established. Is false otherwise
    • [x] socket.destroy(error) immediately destroys the stream and closes the connection
      • [x] All underlying resources are freed
    • [x] ~socket.connect(...) can be called again immediately after socket.destroy(...)~ Currently not planning to implement this for now. We can do this in a separate PR.
    • [x] socket.destroyed is true after calling socket.destroy()
    • [x] socket.destroySoon() destroys the socket after all data is written. If the finish event already emitted, the socket is destroyed immediately. If the socket is still writable, end() is called.
    • [x] socket.end(data[, encoding[, callback]]) writes the given chun and ends the writable size and half-closes the socket
    • [x] socket.localAddress always returns 0.0.0.0
    • [x] socket.localPort always returns 0
    • [x] socket.localFamily always returns 0
    • [x] socket.pause() pauses reading of data on the stream. Pauses the underlying read loop
    • [x] socket.pending is true if the socket is not connected yet at all
    • [x] socket.ref() and socket.unref() are non-op
    • [x] socket.remoteAddress is the remote host
    • [x] socket.remoteFamily is the remote host IP type
    • [x] socket.remotePort is the remote port
    • [x] socket.resetAndDestroy() closes the connection and destroys the stream.
    • [x] socket.resume() resumes reading after a call to pause
    • [x] socket.setEncoding(...) sets the text encoding for the readable side of the socket
    • [x] socket.setKeepAlive() accepts falsy values, throws on truthy values
    • [x] socket.setNoDelay() accepts falsy values, throws on truthy values
    • [x] socket.readyState accurately reflects the ready status of the connection
    • [x] The timeout timer is reset after each read from the underlying socket
  • [x] Add the compat flag / date that will signal the availability of the net API`
  • [x] Tests test and more tests ... The plan is to port as many of the Node.js tests as possible but testing of the Socket API within workerd is fairly limited currently
  • [ ] Documentation

Future work items (to come in separate follow-up PRs

  • tls.TLSSocket is implemented
  • net.BlockList is implemented
  • net.SocketAddress is implemented
  • Socket supports diagnostics channel
  • AsyncLocalStorage context is correctly propagated such that it matches Node.js' behavior (will needs tests to verify this)

jasnell avatar Jun 05 '24 00:06 jasnell

This is not 100% complete but it's far enough along that we ought to be able to get the initial bits landed. Follow on PRs will add tests and missing pieces.

jasnell avatar Jun 11 '24 16:06 jasnell

Rebased the PR to build on top of the tcp-ingress PR so I can start working on tests

jasnell avatar Jun 13 '24 22:06 jasnell

Tests from Node.js to port ... not all of these will be relevant. Will check them off as they are ported or determined not to be relevant:

  • [x] test/parallel/test-net-access-byteswritten.js
  • [x] test/parallel/test-net-after-close.js
  • [x] test/parallel/test-net-allow-half-open.js
  • [x] test/parallel/test-net-better-error-messages-port-hostname.js
  • [x] test/parallel/test-net-binary.js
  • [x] test/parallel/test-net-buffersize.js
  • [x] test/parallel/test-net-bytes-read.js
  • [x] test/parallel/test-net-bytes-stats.js
  • [x] test/parallel/test-net-bytes-written-large.js
  • [x] test/parallel/test-net-can-reset-timeout.js
  • [x] test/parallel/test-net-connect-abort-controller.js
  • [x] test/parallel/test-net-connect-after-destroy.js
  • [x] test/parallel/test-net-connect-buffer.js
  • [x] test/parallel/test-net-connect-destroy.js
  • [x] test/parallel/test-net-connect-immediate-destroy.js
  • [x] test/parallel/test-net-connect-immediate-finish.js
  • [x] test/parallel/test-net-connect-keepalive.js
  • [x] test/parallel/test-net-connect-memleak.js
  • [x] test/parallel/test-net-connect-no-arg.js
  • [x] test/parallel/test-net-connect-nodelay.js
  • [x] test/parallel/test-net-connect-options-allowhalfopen.js
  • [x] test/parallel/test-net-connect-options-fd.js
  • [x] test/parallel/test-net-connect-options-invalid.js
  • [x] test/parallel/test-net-connect-options-ipv6.js
  • [x] test/parallel/test-net-connect-options-path.js
  • [x] test/parallel/test-net-connect-options-port.js
  • [x] test/parallel/test-net-connect-paused-connection.js
  • [x] test/parallel/test-net-dns-custom-lookup.js
  • [x] test/parallel/test-net-dns-error.js
  • [x] test/parallel/test-net-dns-lookup-skip.js
  • [x] test/parallel/test-net-dns-lookup.js
  • [x] test/parallel/test-net-during-close.js
  • [x] test/parallel/test-net-end-close.js
  • [x] test/parallel/test-net-end-destroyed.js
  • [x] test/parallel/test-net-end-without-connect.js
  • [x] test/parallel/test-net-isip.js
  • [x] test/parallel/test-net-isipv4.js
  • [x] test/parallel/test-net-isipv6.js
  • [x] test/parallel/test-net-keepalive.js
  • [x] test/parallel/test-net-large-string.js
  • [x] test/parallel/test-net-local-address-port.js
  • [x] test/parallel/test-net-localerror.js
  • [x] test/parallel/test-net-normalize-args.js
  • [x] test/parallel/test-net-onread-static-buffer.js
  • [x] test/parallel/test-net-options-lookup.js
  • [x] test/parallel/test-net-pause-resume-connecting.js
  • [x] test/parallel/test-net-perf_hooks.js
  • [x] test/parallel/test-net-persistent-keepalive.js
  • [x] test/parallel/test-net-persistent-nodelay.js
  • [x] test/parallel/test-net-persistent-ref-unref.js
  • [x] test/parallel/test-net-pingpong.js
  • [x] test/parallel/test-net-pipe-connect-errors.js
  • [x] test/parallel/test-net-pipe-with-long-path.js
  • [x] test/parallel/test-net-reconnect.js
  • [x] test/parallel/test-net-remote-address-port.js
  • [x] test/parallel/test-net-remote-address.js
  • [x] test/parallel/test-net-settimeout.js
  • [x] test/parallel/test-net-socket-byteswritten.js
  • [x] test/parallel/test-net-socket-close-after-end.js
  • [x] test/parallel/test-net-socket-connect-invalid-autoselectfamily.js
  • [x] test/parallel/test-net-socket-connect-invalid-autoselectfamilyattempttimeout.js
  • [x] test/parallel/test-net-socket-connect-without-cb.js
  • [x] test/parallel/test-net-socket-connecting.js
  • [x] test/parallel/test-net-socket-constructor.js
  • [x] test/parallel/test-net-socket-destroy-send.js
  • [x] test/parallel/test-net-socket-destroy-twice.js
  • [x] test/parallel/test-net-socket-end-before-connect.js
  • [x] test/parallel/test-net-socket-end-callback.js
  • [x] test/parallel/test-net-socket-local-address.js
  • [x] test/parallel/test-net-socket-no-halfopen-enforcer.js
  • [x] test/parallel/test-net-socket-ready-without-cb.js
  • [x] test/parallel/test-net-socket-reset-send.js
  • [x] test/parallel/test-net-socket-reset-twice.js
  • [x] test/parallel/test-net-socket-setnodelay.js
  • [x] test/parallel/test-net-socket-timeout-unref.js
  • [x] test/parallel/test-net-socket-timeout.js
  • [x] test/parallel/test-net-socket-write-after-close.js
  • [x] test/parallel/test-net-socket-write-error.js
  • [x] test/parallel/test-net-stream.js
  • [x] test/parallel/test-net-sync-cork.js
  • [x] test/parallel/test-net-throttle.js
  • [x] test/parallel/test-net-timeout-no-handle.js
  • [x] test/parallel/test-net-writable.js
  • [x] test/parallel/test-net-write-after-close.js
  • [x] test/parallel/test-net-write-after-end-nt.js
  • [x] test/parallel/test-net-write-arguments.js
  • [x] test/parallel/test-net-write-cb-on-destroy-before-connect.js
  • [x] test/parallel/test-net-write-connect-write.js
  • [x] test/parallel/test-net-write-fully-async-buffer.js
  • [x] test/parallel/test-net-write-fully-async-hex-string.js
  • [x] test/parallel/test-net-write-slow.js
  • [x] test/parallel/test-net-connect-reset-after-destroy.js
  • [x] test/parallel/test-net-connect-reset-before-connected.js
  • [x] test/parallel/test-net-connect-reset-until-connected.js
  • [x] test/parallel/test-net-connect-reset.js

test-net-* tests we won't be implementing now:

  • The autoselectfamily tests are a happy eyeballs implementation in Node.js that we are not implementing directly
    • test/parallel/test-net-autoselectfamily-attempt-timeout-cli-option.js
    • test/parallel/test-net-autoselectfamily-attempt-timeout-default-value.js
    • test/parallel/test-net-autoselectfamily-commandline-option.js
    • test/parallel/test-net-autoselectfamily-default.js
    • test/parallel/test-net-autoselectfamily-ipv4first.js
    • test/parallel/test-net-autoselectfamily.js
  • test/parallel/test-net-better-error-messages-listen-path.js
  • test/parallel/test-net-better-error-messages-listen.js
  • test/parallel/test-net-better-error-messages-path.js
  • test/parallel/test-net-bind-twice.js
  • test/parallel/test-net-child-process-connect-reset.js
  • test/parallel/test-net-client-bind-twice.js
  • test/parallel/test-net-listen-after-destroying-stdin.js
  • test/parallel/test-net-listen-close-server-callback-is-not-function.js
  • test/parallel/test-net-listen-close-server.js
  • test/parallel/test-net-listen-error.js
  • test/parallel/test-net-listen-exclusive-random-ports.js
  • test/parallel/test-net-listen-fd0.js
  • test/parallel/test-net-listen-handle-in-cluster-1.js
  • test/parallel/test-net-listen-handle-in-cluster-2.js
  • test/parallel/test-net-listen-invalid-port.js
  • test/parallel/test-net-listen-ipv6only.js
  • test/parallel/test-net-listen-twice.js
  • test/parallel/test-net-listening.js
  • test/parallel/test-net-connect-buffer2.js
  • test/parallel/test-net-connect-call-socket-connect.js
  • test/parallel/test-net-server-async-dispose.mjs
  • test/parallel/test-net-server-call-listen-multiple-times.js
  • test/parallel/test-net-server-capture-rejection.js
  • test/parallel/test-net-server-close-before-calling-lookup-callback.js
  • test/parallel/test-net-server-close-before-ipc-response.js
  • test/parallel/test-net-server-close.js
  • test/parallel/test-net-server-drop-connections.js
  • test/parallel/test-net-server-keepalive.js
  • test/parallel/test-net-server-listen-handle.js
  • test/parallel/test-net-server-listen-options-signal.js
  • test/parallel/test-net-server-listen-options.js
  • test/parallel/test-net-server-listen-path.js
  • test/parallel/test-net-server-listen-remove-callback.js
  • test/parallel/test-net-server-max-connections-close-makes-more-available.js
  • test/parallel/test-net-server-max-connections.js
  • test/parallel/test-net-server-nodelay.js
  • test/parallel/test-net-server-options.js
  • test/parallel/test-net-server-pause-on-connect.js
  • test/parallel/test-net-server-reset.js
  • test/parallel/test-net-server-simultaneous-accepts-produce-warning-once.js
  • test/parallel/test-net-server-try-ports.js
  • test/parallel/test-net-server-unref-persistent.js
  • test/parallel/test-net-server-unref.js
  • test/parallel/test-net-deprecated-setsimultaneousaccepts.js
  • test/parallel/test-net-eaddrinuse.js
  • test/parallel/test-net-error-twice.js

jasnell avatar Jun 13 '24 22:06 jasnell

This PR can be reviewed. Landing is blocked on the pending code review of https://github.com/cloudflare/workerd/pull/1429

jasnell avatar Jun 17 '24 22:06 jasnell

Note that this PR is still blocked pending review of https://github.com/cloudflare/workerd/pull/1429

jasnell avatar Jun 20 '24 20:06 jasnell