414 URI Too Long

[forabi]

I'm currently experimenting with an auth proxy that integrates with Supabase and RLS policies. In order to check that the authorization is correct, I am doing two requests on the server:

1. create a Supabase client impersonating the end user and use that to fetch all rows from the table to be synced.
2. Use the IDs of the rows as a parameter for the where clause in the URL to be forwarded to Electric server.

Example code:

  const originUrl = new URL(`http://localhost:3000/v1/shape`);

  // Copy over the relevant query params that the Electric client adds
  // so that we return the right part of the Shape log.
  new URL(req.url).searchParams.forEach((value, key) => {
    if ([`live`, `table`, `handle`, `offset`, `cursor`].includes(key)) {
      originUrl.searchParams.set(key, value);
    }
  });

  const { data, error } = await supabase
    .from("patient_profiles")
    .select("id");

  originUrl.searchParams.set(
    `where`,
    ` id IN (${data?.map((d) => `'${d.id}'`).join(", ")})`
  );


  let resp = await fetch(originUrl.toString());
  if (resp.headers.get("content-encoding")) {
    const headers = new Headers(resp.headers);
    headers.delete("content-encoding");
    headers.delete("content-length");
    resp = new Response(resp.body, {
      status: resp.status,
      statusText: resp.statusText,
      headers,
    });
  }

  return resp;

This is the only way I can confidently say that my RLS policies are being enforced all the way down to the end user's browser.

When the user has lots of rows to be returned, a 414 URL Too Long error is returned.

I understand that this is NOT the recommended way to implement security rules, but this issue could actually come up even if I were to rewrite the logic, e.g.

  originUrl.searchParams.set(
    `where`,
      organization_id IN (${data?.map((d) => `'${d.organization_id}'`).join(", ")}) // What if the user belong to too many organizations?
    `
  );

How should such cases be handled? Can we instead use a POST request?

[KyleAMathews]

Oh interesting. Is that 414 coming directly from electric? Perhaps it's some built-in limit we can extend as we don't really care about the url limit.

[forabi]

@KyleAMathews yes, it's directly from the electric server. I confirmed that by logging the status code of the response before the proxy forwards it. TBH I'd prefer if POST was supported (despite being non-ideomatic)

[KyleAMathews]

Curious why you'd prefer POSTs? We've considered it but decided against it as it'd add a lot more round trips to the server plus you'd need to manage mapping between your shape definition and the shape handle. Another complexity is that we don't guarantee shapes will last as they might need deleted for a variety of reasons. With GETs you just fetch in a loop which is relatively simple.

[KyleAMathews]

We'll look into the URL limit. How many characters are you sending before hitting the limit?

[forabi]

URL looks like http://localhost:3001/v1/shape?table=patient_profiles&where="id in (1000 uuids separated by a comma)" so for this specific table and origin the exact length is 39151 chars. Obviously it could be larger for larger sets of rows or different domains and table names.

Curious why you'd prefer POSTs? We've considered it but decided against it as it'd add a lot more round trips to the server plus you'd need to manage mapping between your shape definition and the shape handle. Another complexity is that we don't guarantee shapes will last as they might need deleted for a variety of reasons. With GETs you just fetch in a loop which is relatively simple.

In my experience, many proxies are not happy with long URLs (at least nginx and Cloudflare, the two I have run into similar issues with). Maybe I don't understand how the sync engine works (I literally just started prototyping with Electric a couple hours ago). My current solution is to chunk request at the proxy level into sets of ~100 UUIDs and send x number of requests to Electric, then stitching them together, but I honestly have no idea what electric-offset electric-schema, or electric-handle are supposed to be in this case. I'm just experimenting.

[KyleAMathews]

Yeah there's limits at different points for sure.

What you could do though is the client just requests /api/shape/organizations or whatever and that's what gets through cloudflare etc and then at your API is where you translate it to the actual shape request to pass onto electric.

The headers are how the server and client coordinate to help the client follow the shape log

https://electric-sql.com/docs/api/http#shape-log