WASI: Fine-grained network policies

[badeend]

#7662 is a good first step towards letting users of the wasmtime library customize network behavior. After that change, library users have two options: either use inherit_network which grants access to everything, or use socket_addr_check which allows the user to define arbitrarily complex rules. The upside of that last one is also its downside: the user must define everything themself. On the command-line, the available choices are quite limited: everything or nothing at all. Guess which one users will pick :P

I would like to add a middle ground option for both library & CLI users. This option should provide a default "good enough" option for the majority of users, while still allowing to fall back to fully customized control. My objective is to be able to declare network policies based on:

• Domain name
• Initiator (client vs server)
• Protocol (TCP, UDP, HTTP(S))

I tried to capture the gist of it in pseudo code:

// A single network permission
enum Grant {

    // Allow TCP sockets to connect to remote_host/port, optionally using a specific local_interface/port
    TcpOutbound {
        remote_host: RemoteHostPattern,
        remote_port: RemotePortPattern,
        local_interface: LocalInterfacePattern, // default = LocalInterfacePattern::Any
        local_port: LocalPortPattern, // default = LocalPortPattern::Ephemeral
    },

    // Allow TCP sockets to listen on a local port, optionally using a specific local_interface too
    TcpInbound {
        local_port: LocalPortPattern,
        local_interface: LocalInterfacePattern, // default = LocalInterfacePattern::Any
    },

    // Allow UDP sockets to initiate flows to remote_host/port, optionally using a specific local_interface/port
    UdpOutbound {
        remote_host: RemoteHostPattern,
        remote_port: RemotePortPattern,
        local_interface: LocalInterfacePattern, // default = LocalInterfacePattern::Any
        local_port: LocalPortPattern, // default = LocalPortPattern::Ephemeral
    },

    // Allow UDP sockets to handle incoming flows on a local port, optionally using a specific local_interface too
    UdpInbound {
        local_port: LocalPortPattern,
        local_interface: LocalInterfacePattern, // default = LocalInterfacePattern::Any
    },

    // Allows the component to make outgoing HTTP connections
    HttpOutbound {
        scheme: Http | Https,
        host: RemoteHostPattern,
        port: RemotePortPattern,
    },
}


// "*"             -> RemoteHostPattern::Any
// "localhost"     -> RemoteHostPattern::Loopback
// "example.com"   -> RemoteHostPattern::Domain(DomainPattern::Single("example.com"))
// "*.example.com" -> RemoteHostPattern::Domain(DomainPattern::Wildcard("example.com"))
// "192.0.2.0"     -> RemoteHostPattern::Ip(IpRange("192.0.2.0".into())
// "192.0.2.0/24"  -> RemoteHostPattern::Ip(IpRange("192.0.2.0/24".into())
enum RemoteHostPattern {
    Any,
    Loopback, // Effectively `127.0.0.0/8` and `::1`, but without committing to a specific address family.
    Ip(IpRange),
    Domain(DomainPattern),
}


// "*"           -> RemotePortPattern::Range(1..=u16::MAX))
// "0"           -> invalid
// "80"          -> RemotePortPattern::Range(80..=80))
// "35000-35999" -> RemotePortPattern::Range(35000..=35999))
enum RemotePortPattern {
    Range(PortRange),
}


// "*"         -> LocalInterfacePattern::Any
// "localhost" -> LocalInterfacePattern::Loopback
// "192.0.2.0" -> LocalInterfacePattern::Ip(IpRange("192.0.2.0".into())
// "::"        -> LocalInterfacePattern::Ip(IpRange("::".into())
enum LocalInterfacePattern {
    Any, // Effectively `0.0.0.0` and `::`, but without committing to a specific address family.
    Loopback, // Effectively `127.0.0.0/8` and `::1`, but without committing to a specific address family.
    Ip(IpRange),
}


// "*"           -> LocalPortPattern::Range(0..=u16::MAX))
// "0"           -> LocalPortPattern::Ephemeral
// "80"          -> LocalPortPattern::Range(80..=80))
// "35000-35999" -> LocalPortPattern::Range(35000..=35999))
enum LocalPortPattern {
    Ephemeral,
    Range(PortRange),
}



type PortRange = RangeInclusive<u16>; // Can also represent single ports by storing an identical `start` and `end` port.

struct IpRange(ipnet::IpNet); // Can also represent a single address

struct Domain(String);

enum DomainPattern {
    Single(Domain),
    Wildcard(Domain), // Allows the domain itself, along with every subdomain.
}

Domain-based policy strategy

THe IP and Port-based patterns above should speak for themselves. The domain name policies might need to explanation: the idea is to hook into ip-name-lookup::resolve-addresses to keep track of which IP address belongs to which domain names at runtime:

• In ip-name-lookup::resolve-addresses:
  • Before making the syscall: validate that any TcpOutbound or UdpOutbound grant exists with a Any or matching Domain host pattern.
  • After making the syscall: if the previous step matched any Domain-based grants: register the resolved addresses in DynamicPolicy::resolved_names (see below)
• In tcp-socket::bind: validate that any TcpOutbound or TcpInbound grant exists with a matching local_interface and local_port
• In tcp-socket::connect, validate that any TcpOutbound grant matches the local_interface & local_port. Also match the remote_host & remote_port:
  • first by the IP address passed to the connect call. If none found:
  • then by all resolved_names for that IP.

pub struct DynamicPolicyConfig { // Shared across many component instances.
    grants: Vec<Grant>,
}

pub struct DynamicPolicy { // Instantiated once per component
    // Reference to the "static" rules:
    config: Arc<DynamicPolicyConfig>,

    // Mapping between resolved IP addresses and the queried domain names.
    resolved_names: LruCache<IpAddr, Vec<Domain>>,

    // (Recently) active UDP flows
    udp_flows: LruCache<(/*local*/SocketAddr, /*remote*/SocketAddr), ()>
}

UDP directionality

UDP has no traditional notion of "client" and "server". However, in practice many UDP applications do fit that model. I've modeled the grant types above based on what stateful firewall do. In order to know the directionality (inbound vs outbound) we need to keep track of "who talked first".

CLI syntax

Inbound syntax:

--expose 80                      // Grant::TcpInbound(...) & Grant::UdpInbound(...) // Shorthand inspired by Docker
--expose 127.0.0.1:80            // Grant::TcpInbound(..., local_interface: "127.0.0.1") & Grant::UdpInbound(..., local_interface: "127.0.0.1")
--expose udp://127.0.0.1:80      // Grant::UdpInbound(..., local_interface: "127.0.0.1")

Outbound syntax:

--connect tcp://example.com:80    // Grant::TcpOutbound(...) & Grant::HttpOutbound(...)
--connect udp://192.168.0.1:80    // Grant::TcpOutbound(...)
--connect http://*.example.com/   // Grant::HttpOutbound(...)
--connect https://example.com/    // Grant::HttpOutbound(...)

Let me know what you think.

[alexcrichton]

This all sounds like a great idea to me, thanks for writing this up @badeend!

One comment I would have is that I think it would be best to implement this in a way that's not baked-in to wasmtime-wasi itself, e.g. baking it into the WasiCtx. You've seen already (but for others reading this too) discussion at https://github.com/bytecodealliance/wasmtime/issues/7694 about ways we might achieve that, and I think it would be good if we could fit this model into that extension. Put another way this could be one implementation of socket_addr_check (more-or-less, I realize that single callback isn't enough) that embedders could opt-in to.

I think this is pretty reasonable syntax to add to the CLI though and I definitely agree it'd be best to have more than just an "everything on" switch!

[rylev]

This all looks really great! Seems like we're headed in the right direction. A few small thoughts and questions:

• I see that tcp://example.com:80 implies allowing wasi:http usage. I'm not sure this is a good idea. For example, this does not make sense in an HTTP/3 world.
• For the CLI syntax, we might want to consider always requiring a port. So if the user does not care about which port, they must opt into that through like so tcp://example.com:*
• What do port ranges look like on the CLI look like? I would imagine we would support a syntax that mirrors Rust exclusive range syntax like 5000..6000 and 1023...
• Will we allow lists of domains? Something like https:{example.com, google.com}:* or do we expect those always to be listed out separately? I think I would lean towards requiring they be listed separately.
• This isn't really related to this change since it's a result of the way they wasi interfaces line up, but I'm slightly worried about user confusion due to HTTP being treated differently. If users do HTTP but that traffic happens to go over a normal wasi:sockets instead of wasi:http, they may be confused by the difference in behavior. I don't think there's much we can do about that, but it's something that's on my mind.

[badeend]

@alexcrichton

One comment I would have is that I think it would be best to implement this in a way that's not baked-in to wasmtime-wasi itself

I agree. Continuing on #7694 , I have the following in mind:

• Extract the actual syscalls out of preview2/host/*.rs and into distinct types.
• Delegate the actual invocation of those new types to newly introduced "intercept" methods.

This example is for TCP Bind only, but you can imagine the same for other operations.

pub trait WasiTcpView: WasiView {
    /// Custom state maintained per socket instance.
    type Socket;

    /// Create new custom socket state. Guaranteed to be called at most once per WASI socket resource.
    fn new(&mut self) -> Self::Socket;

    /// Called at the moment the actual syscall would have been called. So _after_ the WASI-specific state & input validations.
    /// The `bind` parameter represents the actual sycall implementation that can be executed (or ignored) by the interceptor.
    /// With this general design, the interceptor:
    /// - has full control of what to execute before & after the syscall,
    /// - may conditionally execute or reject the syscall depending on the parameters,
    /// - can keep track of additional state per socket, that would otherwise not be maintained by wasmtime-wasi itself 
    /// - (not in this example, but:) maybe manipulate parameters and return values. For example:
    ///   rewrite the address parameter from port 80 ("inside wasm") to 8080 ("outside wasm")
    fn intercept_bind(&mut self, socket: &mut Self::Socket, bind: TcpBind);
}



#[must_use="For clarity, explicitly drop it using one of the consuming methods."]
pub struct TcpBind<'a> {
    // ...
}
impl TcpBind<'_> {
    /// The address passed in by the user.
    pub fn requested_address(&self) -> &SocketAddr {}

    /// Consume self and perform the bind on the requested address. Returns the actually bound address.
    pub fn execute(mut self) -> std::io::Result<SocketAddr> {
        // Call rustix::bind etc.
    }

    /// Consume self and abort the bind with an error code.
    pub fn fail(self, error: TcpBindError) {
    }

    // The typical consuming methods will be `execute` and `fail`.
    // But `UdpSend` could have an `skip` method as well, that pretends to have sent the message, but actually it was dropped.
}

/// Subset of all wasi-sockets errors that are appropriate for `bind` to return.
#[non_exhaustive]
pub enum TcpBindError {
    AccessDenied,
    AddressInUse,
    AddressNotBindable,
}






// Example implementations
impl WasiTcpView for WasiCtx {
    type Socket = ();

    fn new(&mut self) {}

    // Example #1: allow everything
    fn intercept_bind(&mut self, socket: &mut Self::Socket, bind: TcpBind) {
        bind.execute()
    }

    // Example #2: deny everything
    fn intercept_bind(&mut self, socket: &mut Self::Socket, bind: TcpBind) {
        bind.fail(TcpBindError::AccessDenied)
    }

    // Example #3: arbitrary logic
    fn intercept_bind(&mut self, socket: &mut Self::Socket, bind: TcpBind) {
        if bind.requested_address().ip().is_loopback() {
            bind.execute()
        } else {
            bind.fail(TcpBindError::AccessDenied)
        }
    }
}

---

@rylev

• I see that tcp://example.com:80 implies allowing wasi:http usage. I'm not sure this is a good idea. For example, this does not make sense in an HTTP/3 world.
• This isn't really related to this change since it's a result of the way they wasi interfaces line up, but I'm slightly worried about user confusion due to HTTP being treated differently. If users do HTTP but that traffic happens to go over a normal wasi:sockets instead of wasi:http, they may be confused by the difference in behavior. I don't think there's much we can do about that, but it's something that's on my mind.

Ah, yes. The classical "user experience" vs. "security" tradeoff. From an end-user's perspective I don't care how a specific Wasm module decided to implement their network requests. I.e. I want to say "you're allowed to fetch https://example.com/hi.txt" without knowing having to care whether that will be done using TCP/UDP directly or using a wasi-http client. To properly guard this off would effectively be a man-in-the-middle attack. However, I think the other way around is perfectly reasonable, and more importantly: feasible 😛 . Allowing TCP traffic to a specific endpoint should also allow HTTP(1&2) traffic to that endpoint. Same for UDP and HTTP3.

That being said, none of this is really essential for to this issue, so I'm happy to postpone all "implying" to a future iteration. I we allow wildcards in the protocol (e.g. *://example.com:443) that should be good enough for now.

---

• For the CLI syntax, we might want to consider always requiring a port. So if the user does not care about which port, they must opt into that through like so tcp://example.com:*

I agree. At least for TCP and UDP. For HTTP(S), the port should be inferred IMO

---

• What do port ranges look like on the CLI look like? I would imagine we would support a syntax that mirrors Rust exclusive range syntax like 5000..6000 and 1023...
• Will we allow lists of domains? Something like https:{example.com, google.com}:* or do we expect those always to be listed out separately? I think I would lean towards requiring they be listed separately.

A mix of "TBD" and "I don't care" 😛

[badeend]

Coming back on my previous comment:

Even though the initial refactor may be more work, I think it's more straightforward to forget about "interceptors" completely and put the entire socket implementation behind traits whose implementations can be swapped out. My current train of thought is to create a "vanilla Rust" trait that loosely follows the wasi-sockets interface. With "vanilla" I mean:

• async instead of start_/finish_
• std::net::SocketAddr instead of bindings::sockets::network::IpSocketAddress
• std::io::Result instead of SocketResult
• tokio::io::AsyncRead instead of preview2::stream::HostInputStream
• etc..

#[async_trait]
pub trait TcpSocket {
	type InputStream: AsyncRead;
	type OutputStream: AsyncWrite;
	type AcceptStream: Stream<Item = io::Result<Self>>;

	fn new(addr: SocketAddr) -> io::Result<Self>;

	async fn bind(&mut self, addr: &SocketAddr) -> io::Result<()>;
	async fn connect(&mut self, addr: &SocketAddr) -> io::Result<(Self::InputStream, Self::OutputStream)>;
	async fn listen(&mut self) -> io::Result<Self::AcceptStream>;

	fn local_address(&self) -> io::Result<SocketAddr>;
	fn remote_address(&self) -> io::Result<SocketAddr>;

	fn keep_alive_enabled(&self) -> io::Result<bool>;
	fn set_keep_alive_enabled(&self, value: bool) -> io::Result<()>;

	// ...
}

Besides defining that trait, wasmtime_wasi should also provide a default implementation for it. Containing much of our current implementation. Custom implementations can then reuse that default implementation:

pub struct RestrictedTcpSocket {
    inner: SystemTcpSocket, // The default, native implementation
    // ...
}
#[async_trait]
impl TcpSocket for RestrictedTcpSocket {
    async fn bind(&mut self, addr: &SocketAddr) -> io::Result<()> {
        if addr.ip().is_loopback() {
            inner.bind(addr)
        } else {
            return Err(Error::new(ErrorKind::PermissionDenied, "Nope."));
        }
    }

    // ...
}

---

With that in place, impl<T: WasiView> crate::preview2::host::tcp::tcp::HostTcpSocket for T can focus fully on WASI stuff:

• Enforcing WASI-specific state invariants and parameter requirements
• Looking up resources from tables
• Mapping std errors into WASI error codes.
• Converting Rust's async & IO primitives into their WASI counterparts

WasiTcpView becomes a bit simpler too:

pub trait WasiTcpView {
	type Socket: TcpSocket;
}

[Greensue]

@badeend hi, excuse me , do you have plan to achieve this fine-grained network plolices? Is there a specific timestone?

[rylev]

This is being worked on in https://github.com/bytecodealliance/wasmtime/pull/7705. Hoping that it shouldn't be too much longer before this is ready to merge.

[Greensue]

@rylev thanks, but I did not found a "grant" struct as badeend described use pseudo code。 would this be achieve in 7705 in the future? if there is a plan？for now, define a "socket_addr_check" func it's complexing。

[badeend]

#7705 contains the preparational work discussed further in this issue. We haven't started on the design in the initial comment