Streaming support

[noamr]

There are new proposals coming for allow streaming HTML into an existing document. See https://github.com/whatwg/html/issues/11542 and https://github.com/whatwg/html/issues/2142#issuecomment-3163420231.

For this to work, it would need to come along with sanitation support.

To connect the dots, trusted-types would need some form of sanitizing an HTML stream on the go, probably using a TransformStream.

This can have the following shape:policy.createHTMLTransformStream(), where the source of the untrusted HTML would be piped to the result's writable and the readable would be piped to the element receiving the markup.

[otherdaniel]

Thought a bit about this. Creating a stream is quite unlike any other Trusted Types API, but writing to a stream seems to fit the existing model just fine: That's the boundary where a string gets accepted for inclusion into the DOM. So if the write operation for any stream created by any of the patch methods (patchSelf, patchBefore, etc.) would do a TT check, then this would seem to fit quite neatly with the existing design:s

• Keeps guarding the string-to-DOM boundary.
• Keeps operating on a single string value.
• Near 100% re-use of spec and (likely) implementation.
• The write method already supports any as parameter type.
• The only difference to existing APIs would likely be that a check failure would reject the promise, rather than throwing.

In more spec-y terms, the writers created by any WritableStream created by any of the patch methods would call get Trusted Type compliant string in their write method, and would return a rejected promise if the TT check has thrown.

Maybe one useful extension could be that the stream creator could set a TrustedTypesPolicy on the stream. That is, instead of reyling on the default policy as a catch-all for the entire page, a page developer could take an existing policy which accepts/rejects (or even re-writes) incoming strings, and apply that to any non-trusted values being written. That'd be a little different from what we do so far, but seems to fit well within the existing framework.

---

It'd be really nice if there was a non-TT version. You already mention sanitization. If there were "safe" and "unsafe" versions (with Sanitizer's definition of safe, like setHTML and setHTMLUnsafe), then the safe one wouldn't even need any TT checks at all. Even better, a page could set up a stream with whatever sanitization rules it wants, and then pass it around to libraries or semi-untrusted code and could still be sure the resulting DOM matches its expectations.

I see Trusted Types and Sanitizer a bit like "carrot and stick", in that TT blocks DOM operations and thus potentially gets in your way (the stick), while Sanitizer makes it really easy to modify the DOM (within certain security constraints, the carrot). For this new streaming API, it'd be really nice if it had a "carrot"-type API, too.

---

An alternative I've considered is that, in TT world, the stream in Sreaming HTML isn't really like a string value, because it can accept arbitrary values and arbitrarily many values. It's arguably more like a TT policy, which in TT world is the thing that can bless arbitrary input values. So one could try to treat streams as policies and restrict stream creation to a set of named streams; and/or have a callback for any created stream that a policy could then reject (or modify, by setting a sanitizer). But… while I think that train of thought has some merit, I haven't really managed to cast this into an API that I'd actually want to use; my attempts were all a little awkward. I guess sticking to the string-to-DOM boundary has more promise.

[noamr]

The sanitizer version is being worked on separately. We are indeed considering something equivalent to setHTML and setHTMLUnsafe. See https://github.com/WICG/declarative-partial-updates/issues/42.

What I am struggling with regarding streaming and the string-to-DOM boundary, is that to perform effective sanitation of a string, the string would need to be buffered somewhere, and that buffering is not always controlled directly by whoever writes to the stream.

I am not sure how that exactly works for document.write(), as the following strings might be safe separately but not together? <scr and ipt>do_something</ script>.

Please forgive me if these are noob questions in the sanitation world.

[annevk]

It doesn't really work for document.write(), though ideally we don't propagate that to new APIs.

[foolip]

I'm trying to make the API surface concrete enough to start writing spec text. The starting point in the explainer is a patchUnsafe() method that returns a WritableStream. Fitting TT into that, all the alternatives I can think of are:

• Let patchUnsafe() return a WritableStream with replaced internals that always passes chunks through TT. (May need something new in Streams to prevent tampering.)
• Treating the TT step as a TransformStream and let patchUnsafe() return a TransformStream already piped to a WritableStream. (May need something new in Stream to prevent them from being unlocked.)
• A new createHTMLStream() method on TrustedTypePolicy and let patchUnsafe() take a stream as an optional argument, which it would also return. A stream created from the default policy would be used by default if no argument is given.

For all options there's also the question of whether a plain WritableStream or TransformStream is OK here, or if a new TrustedHTMLStream is needed. Opinions on precise API shape welcome :)

Other than API shape my main question is the same as @noamr had, how to handle dangerous markup being written in small chunks.

[noamr]

I'm trying to make the API surface concrete enough to start writing spec text. The starting point in the explainer is a patchUnsafe() method that returns a WritableStream. Fitting TT into that, all the alternatives I can think of are:

• Let patchUnsafe() return a WritableStream with replaced internals that always passes chunks through TT. (May need something new in Streams to prevent tampering.)
• Treating the TT step as a TransformStream and let patchUnsafe() return a TransformStream already piped to a WritableStream. (May need something new in Stream to prevent them from being unlocked.)

• A new createHTMLStream() method on TrustedTypePolicy and let patchUnsafe() take a stream as an optional argument, which it would also return. A stream created from the default policy would be used by default if no argument is given.

For all options there's also the question of whether a plain WritableStream or TransformStream is OK here, or if a new TrustedHTMLStream is needed. Opinions on precise API shape welcome :)

Other than API shape my main question is the same as @noamr had, how to handle dangerous markup being written in small chunks.

The way I see it, the TT step should be a TransformStream (with a new createHTMLStream method or so if exists), and its writable part would be what's returned to the caller of patchUnsafe by the platform.

The implementer of the stream-enabled method should take care of buffering.

If there is a policy without that method, we can either:

• buffer everything and fall back to createHTML
• Pass each string separately to createHTML, but I'm not sure that's safe enough.

[lukewarlow]

If there is a policy without that method, we can either:

• buffer everything and fall back to CreateHTML

• Pass each string separately to createHTML, but l'm not sure that's safe enough.

Alternatively we could also just throw a type error like we would in most other cases.

[foolip]

The way I see it, the TT step should be a TransformStream (with a new createHTMLStream method or so if exists), and its writable part would be what's returned to the caller of patchUnsafe by the platform.

@noamr do you mean that patchUnsafe() would internally create this TransformStream and return it? If so, what is the createHTMLStream() method for? Or does one have to call createHTMLStream() and pass it to patchUnsafe()? Both API shapes feel a bit unusual, but maybe that's OK.

[noamr]

The way I see it, the TT step should be a TransformStream (with a new createHTMLStream method or so if exists), and its writable part would be what's returned to the caller of patchUnsafe by the platform.

@noamr do you mean that patchUnsafe() would internally create this TransformStream and return it? If so, what is the createHTMLStream() method for? Or does one have to call createHTMLStream() and pass it to patchUnsafe()? Both API shapes feel a bit unusual, but maybe that's OK.

It's how trusted types work. They protect against injecting sensitive HTML via script. The functions are called by the platform when the developer calls any HTML injection script (like setting innerHTML).

createHTMLStream would be called by the platform when patchUnsafe() is called by any script, and would inject the transform stream in the middle between the "unsafe" stream and the stream created by the patching internal as a security/custom-sanitation layer that does its own buffering.

[foolip]

I've never used Trusted Types, so let me double check my understanding. The most straightforward and typical way to use TT is a header like Content-Security-Policy: require-trusted-types-for 'script'; trusted-types my-policy and JS like this:

function sanitize(input) { /* good stuff */ }

// Early on
const policy = trustedTypes.createPolicy("my-policy", {
  createHTML(input) { return sanitize(input); }
});

// Later
element.innerHTML = policy.createHTML(input);

So the script both calls createHTML() and passes the returned TrustedHTML object to another API, the innerHTML setter.

If there's a default policy then createHTML() is automatically called, but the spec describes this as a last resort so I guess it's not how TT is typically used. The way it's described on MDN seems to confirm this.

So, what would be the equivalent to the typical "my-policy" case above? I think it would have to be:

const policy = trustedTypes.createPolicy("my-policy", {
  createHTMLStream() {
    return new TransformStream({
      transform(chunk, controller) {
        // TODO: some buffering so that chunks 
        controller.enqueue(sanitize(chunk));
      }
    });
  }
});

// Later
const trustedStream = policy.createHTMLStream(input);
const writable = element.patchUnsafe(trustedStream);

If relying on the default policy, the argument could be omitted, I suppose.

@noamr is this the shape of the integration you were thinking?

[foolip]

In https://github.com/w3c/trusted-types/pull/597 I have sketched out some idea of what the TT spec changes might look like.

[noamr]

I've never used Trusted Types, so let me double check my understanding. The most straightforward and typical way to use TT is a header like Content-Security-Policy: require-trusted-types-for 'script'; trusted-types my-policy and JS like this:

function sanitize(input) { /* good stuff */ }

// Early on
const policy = trustedTypes.createPolicy("my-policy", {
  createHTML(input) { return sanitize(input); }
});

// Later
element.innerHTML = policy.createHTML(input);

So the script both calls createHTML() and passes the returned TrustedHTML object to another API, the innerHTML setter.

If there's a default policy then createHTML() is automatically called, but the spec describes this as a last resort so I guess it's not how TT is typically used. The way it's described on MDN seems to confirm this.

So, what would be the equivalent to the typical "my-policy" case above? I think it would have to be:

const policy = trustedTypes.createPolicy("my-policy", {
  createHTMLStream() {
    return new TransformStream({
      transform(chunk, controller) {
        // TODO: some buffering so that chunks 
        controller.enqueue(sanitize(chunk));
      }
    });
  }
});

// Later
const trustedStream = policy.createHTMLStream(input);
const writable = element.patchUnsafe(trustedStream);

This is probably not needed. The underlying patching system can handle TrustedHTML chunks differently.

If relying on the default policy, the argument could be omitted, I suppose.

@noamr is this the shape of the integration you were thinking?

Other than the comment above, yes! But would love to hear other voices.

[foolip]

The model in https://github.com/w3c/trusted-types/pull/597 was that Trusted Types would ensure that a policy-controlled and unforgeable TransformStream sits in front of the parser. But that's a bit rigid and I'm not sure what it would take to make it unforgeable.

Here's an alternative approach that I think is more in line with the discussion in this issue. The model is that the individual chunks arriving at the parser need to be trusted. Reusing TrustedHTML would be technically straightforward, but I think a new wrapped type like TrustedHTMLChunk is needed to avoid existing policies from being circumvented by clever chunking.

Here's what it would look like:

// Setup
const policy = trustedTypes.createPolicy("my-policy", {
  createHTMLChunk(input) { return sanitize(input); }
});

// Later
const response = await fetch("./patch");

class SanitizerStream extends TransformStream {
  constructor() {
    super({
      transform(chunk, controller) {
        controller.enqueue(policy.createHTMLChunk(chunk));
      }
    });
  }
}

const writable = element.patchUnsafe(/* no arguments */);
response.body
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(new SanitizerStream())
  .pipeTo(writable);

Does that look like what people are expecting?

[annevk]

But that's a bit rigid and I'm not sure what it would take to make it unforgeable.

We can't have a subclass or annotate it with some internal slot that indicates it was created by a policy? It seems much better than having to bless individual chunks that can be re-ordered and such.

[foolip]

But that's a bit rigid and I'm not sure what it would take to make it unforgeable.

We can't have a subclass or annotate it with some internal slot that indicates it was created by a policy? It seems much better than having to bless individual chunks that can be re-ordered and such.

The TrustedTransformStream I added in https://github.com/w3c/trusted-types/pull/597 could only be created by a policy, similar to TrustedHTML instances. The "make it unforgeable" aspect I was unsure about was if there could be any way of inserting attacker-controlled code between a TrustedTransformStream and the parser. But after discussing with @noamr we don't think there is, Streams doesn't really have APIs for introspection.

Something that I didn't like about https://github.com/w3c/trusted-types/issues/594#issuecomment-3258296179 is that when using a non-default policy, the TrustedTransformStream has to be passed as an argument to patchUnsafe().

A tweak to the API shape would be to let patchUnsafe() always take a ReadableStream argument, instead of returning a WritableStream. Then, a non-TT use would be element.patchUnsafe(response.body) and a TT use would be element.patchUnsafe(response.body.pipeThrough(policy.createSomething(...))).

But taking a step back, I think the options fall in one these categories:

• A TrustedTransformStream or TrustedReadableStream is the unforgeable type. The browser just needs to do a type check once. (This assumes that there's no way to tamper with the pipeline.)
• TrustedHTML or TrustedHTMLChunk as the unforgeable type. There's a simplicity to this (https://github.com/w3c/trusted-types/issues/594#issuecomment-3269953463) but chunks could be filtered or reordered.
• Both. With TrustedHTMLChunk as the starting point, the purpose of the policy-created stream would be to allow for buffering and backpressure.

Right now I'm leaning towards the first, but probably with a modified API shape where element.patchUnsafe() takes a ReadableStream.

@annevk WDYT?

[lukewarlow]

Fwiw I had a TrustedTransformStream in mind when thinking about this, rather than doing stuff per chunk. Making patchUnsafe() an argument seems fine? But I guess it could be odd having different API shape for trusted vs not trusted usages.

[noamr]

Another option is to override the pipe operation of the trusted readable stream.

So patch and patchUnsafe would still return a ReadableStream (or a subclass of it or some such), and myTrustedHTMLStream.pipeTo(element.patchUnsafe()) would operate as if that the chunks are trusted, but inserting anything in between would not.

[noamr]

• TrustedHTML or TrustedHTMLChunk as the unforgeable type. There's a simplicity to this (Streaming support #594 (comment)) but chunks could be filtered or reordered.

btw we could make a TrustedHTMLChunk have an internal ID and the internal ID of the previous one, which would prevent reordering or omitting chunks.

This might make it nicer in terms of fitting with the streaming idioms and can use the same API for trusted/normal.

[foolip]

But I guess it could be odd having different API shape for trusted vs not trusted usages.

@lukewarlow yes, the extra argument when using a TT policy made it feel "bolted on" to me. An argument which can have different types is more like the innerHTML setter which can be either a string or a TrustedHTML object.

[foolip]

@noamr good ideas!

Another option is to override the pipe operation of the trusted readable stream

I suspect that we'd have to add steps to https://streams.spec.whatwg.org/#readable-stream-pipe-to which would be nice to avoid for cleaner layering. Just overriding pipeTo in Web IDL unfortunately wouldn't be enough, since you could still use ReadableStream.prototype.pipeTo.call(...) to use the pipeTo from the base class.

btw we could make a TrustedHTMLChunk have an internal ID and the internal ID of the previous one, which would prevent reordering or omitting chunks.

This might make it nicer in terms of fitting with the streaming idioms and can use the same API for trusted/normal.

Yeah, tagging chunks with something identifying the patch operation and a counter would work. (Just a counter wouldn't since you could then use chunks from another stream.)

[annevk]

But if you do chunk-in-chunk-out you can't do anything like buffering or waiting for more information before you apply the filtering? Or we end up reinventing streams with a different API shape?

The whole reason we wanted streams here is because the Sanitizer API requires waiting in certain cases, such as with replaceWithChildren.

[noamr]

But if you do chunk-in-chunk-out you can't do anything like buffering or waiting for more information before you apply the filtering? Or we end up reinventing streams with a different API shape?

The whole reason we wanted streams here is because the Sanitizer API requires waiting in certain cases, such as with replaceWithChildren.

I am not sure what you are referring to exactly; We are definitely going to use streams, and the question is what is the unforgeable type (equivalent to TrustedHTML in setHTMLUnsafe).

We can either:

• use an unforgeable ReadableStream and pass it as a parameter, and the platform will use that info to circumvent the default policy's stream
• use a stream that passes unforgeable chunks, and as long as they are in order they don't need to go through the default policy's stream
• use a special ReadableStream that bypasses the default policy's stream when piped

All of these are in the streams world and the author of the policy would have to deal with buffering in their TransformStream.

[foolip]

use a stream that passes unforgeable chunks, and as long as they are in order they don't need to go through the default policy's stream

This option seems the most promising to me, and here's what I'm thinking.

The code when using an explicitly created policy would look like this (taken from the example in https://github.com/whatwg/html/pull/11631 combined with https://github.com/whatwg/fetch/pull/1862):

const policy = trustedTypes.createPolicy("my-policy", {
  createTransformStream() {
    return new TransformStream({
      transform(chunk, controller) {
        // TODO: some buffering
        controller.enqueue(sanitize(chunk));
      }
    });
  }
});

const response = await fetch('/fragments/something');
const readable = response.body.textStream();
const transform = policy.createTransformStream();
const writable = element.streamHTMLUnsafe();
await readable.pipeThrough(transform).pipeTo(writable);

The trusted types spec machinery would be on the readable side of the transform stream. When the author-controlled policy code calls controller.enqueue(sanitize(chunk)) or similar, the string chunk would be wrapped in an TrustedHTMLChunk object that is also tagged with a sequence number and the ReadableStream it is for, and then passed along.

streamHTMLUnsafe() would then check that all of the chunks are trusted, and come from a single ReadableStream, in the original order.

When the default policy is used, the code would instead look like this:

trustedTypes.createPolicy("default", {
  createTransformStream() {
    return new TransformStream({
      transform(chunk, controller) {
        // TODO: some buffering
        controller.enqueue(sanitize(chunk));
      }
    });
  }
});

const response = await fetch('/fragments/something');
const readable = response.body.textStream();
const writable = element.streamHTMLUnsafe();
await readable.pipeTo(writable);

Here, streamHTMLUnsafe() would make a decision when getting the first chunk to create a transform stream using the equivalent of trustedTypes.defaultPolicy.createTransformStream(). Chunks would be passed through that transform stream. The same rules would apply, but an implementation could optimize away checks that cannot fail.

If that seems like a sensible API shape, I will try to spec it.

[annevk]

I think fundamentally this has the same kind of flaw as the existing Trusted Types mechanism, which is that we don't really get actual types, we just have wrappers. Which means that whoever implements the policy will have to re-implement the parser essentially. And then some kind of sanitizer on top. I.e., the bit called "TODO: some buffering" is hiding a lot of complexity.

We kinda had to do that for the existing set of APIs because of the demand that it was compatible with legacy APIs, but carrying that forward to new APIs does not seem necessary.

cc @mozfreddyb @otherdaniel

[mozfreddyb]

I don't think there's any way to vet or sanitize a string based on arbitrary chunk sizes. While the suggestions above seem to work for a desired API shape, I don't think this will actually work for the use cases.

[annevk]

I think the simplest thing that gives web developers (some) control and doesn't require a SAX-like interface would be to create a trusted variant of the second argument to setHTMLUnsafe() and only argument of streamHTMLUnsafe() as currently proposed. I initially thought TrustedSanitizer (to be created by the Trusted Types policy) would be sufficient, but you really need to trust all the parser options. And if you can trust all parser options including sanitizer you don't have to trust the input.

[noamr]

I think the simplest thing that gives web developers (some) control and doesn't require a SAX-like interface would be to create a trusted variant of the second argument to setHTMLUnsafe() and only argument of streamHTMLUnsafe() as currently proposed. I initially thought TrustedSanitizer (to be created by the Trusted Types policy) would be sufficient, but you really need to trust all the parser options. And if you can trust all parser options including sanitizer you don't have to trust the input.

Can you clarify "all the parser options"? You mean either with or without a sanitizer?

[annevk]

I think currently the only option there is the sanitizer, but it seems likely we'll add runScripts, and in the past there was a suggestion of being able to toggle declarative shadow root support (probably not needed). There might be more in the future.

I don't think it really matters whether the Trusted Types policy vends trusted options that include a sanitizer or not. What matters is that they realize that the trusted options can be used with any input so if they don't include a sanitizer the chance for XSS is high.

I think that's a reasonable option for Trusted Types policy creators as we know that with the correct options arbitrary input can be safe (or at least not unsafe).

It would be good to get some input from those using Trusted Types in the wild though. @koto perhaps?

[noamr]

I think currently the only option there is the sanitizer, but it seems likely we'll add runScripts, and in the past there was a suggestion of being able to toggle declarative shadow root support (probably not needed). There might be more in the future.

This option is already implicitly available because of ’createContextualFragment’... So trusted type policy creators today have to assume scripts might be running as a result of the setter. But perhaps allowing them to vet that can change the behavior of createContextualFragment if given.

Perhaps whatever the policy creator decides can go into the TrustedHTML or TrustedHTMLStream object rather than having to pass it again in the last argument?

[annevk]

I'm not sure I see how createContextualFragment is related to any of this.

[noamr]

I'm not sure I see how createContextualFragment is related to any of this.

It's equivalent to setHTMLUnsafe(html, {runScripts: true}).

So if we make it so that TrustedTypes policies can vet whether an HTML setter can run scripts, it should affect that behavior as well, no? Otherwise policies would have something like createHTMLSetterOptions: () => ({sanitizer, runScripts: false}) or some such and it would be a false premise.