compiler doesn't check if fields and functions exist (probably by design?)

[jmagaram]

I started writing some rules using the non-existent Google tools and found it quite frustrating. So I was excited to find your tool and ran some little experiments with it. The compiler doesn't validate any properties of any objects or any function names, which seems to defeat what I thought would be a major advantage of the tool - compile-time checks. For example, all the below compiles just fine even though request does not have a doesnotexist property, there is no nonexistentfunc, Google strings don't have a flippy method, and my ShoppingList does not have a goofy property. I do think all the auto-generated functions to check the structural validity of the objects is really cool - like the function is____ShoppingList(data, prev) - and this would be super tedious to write by hand.

type Email = { address : string, verified: bool }

type ShoppingList =
    {
        version: '1',
        createdOn: timestamp,
        modifiedOn: timestamp,
        id: string,
        owner: { uid: string, email? : Email },
        members: map
    }

match /lists/{listId} is ShoppingList {
    allow create: 
        request.auth != null 
        && request.auth.uid.flippy() == 1
        && request.doesnotexist == "xyz"
        && request.auth.uid == listId
        && data.id != "gorilla"
        && data.goofy == 1
        && nonexistentfunc('abc') == 3
        && members.size() == 6
        && members.sizespellingerror() == 7;
}

[bijoutrouvaille]

@jmagaram you ask an excellent question. First, to confirm, Fireward does not have any type validations, as you note. The main purpose of the tool is to concisely enforce structural integrity, and, if your project is written in TypeScript, linking this validated structure with your typescript codebase. I've used the Firebase suite for many projects, and in my experience, this is by far the bigger problem than type-checking the rules code.

That said, I have run into situations where this type checking would have been useful, so I'm with you there. Currently a couple of smaller, topical validations are in the works, but if this project grows enough, maybe we can introduce a full-blown type system, which is something that the monadic architecture of Fireward generally supports very well.

[jmagaram]

When you say structural integrity, are you talking about all that automatic checking you do to make sure only the correct keys/fields are provided? I can see how writing that manually and keeping it in sync with the type definitions would be a big hassle. Is that the primary benefit of the whole thing? Maybe another hassle is making sure to only run validation code on non-optional types? I've been using your tool for several hours now and appreciate all that. But it is still extremely painful to write rules for a pretty simple situation I've got since the expressions end up being really long, too many parenthesis with confusing && and || branches, can't have let statements below the "allow create:" blocks, no editor support that formats expressions so I can read them.

I assume the let statements ARE NOT lazy, which makes it so I can't do something like...

    let isEmailProvided = ["email"] in data;
    let isEmailValid = data.email.matches("some regular expression");
    return !isEmailProvided || isEmailValid;

Would be a nice addition to have unlimited (maybe lazy) "let" statements. And put them inside the "allow create/update" blocks.

I think it is useful to have the validations inside the types to modularize this. I've got an Email type, for example. I put validation in there. But it turns out the validation is different for different contexts, like the owner context vs. the member context. So I created two types, OwnerEmail and MemberEmail, to deal with this, even though the data is exactly the same.

Overall looks like a high quality project.

[jmagaram]

I'm tempted to create a bunch of TypeScript types for the Map MapDiff RulesString etc. and code it up in VS Code and take advantage of all the type checking, intellisense, Prettier code formatting, etc. And then copy the transpired code and my data types into your tool.

[bijoutrouvaille]

When you say structural integrity, are you talking about all that automatic checking you do to make sure only the correct keys/fields are provided?

Sounds like it.

Is that the primary benefit of the whole thing?

That in tandem with the synchronized TypeScript definitions were the original motivation.

I'm tempted to create a bunch of TypeScript types for the Map MapDiff RulesString etc. and code it up in VS Code and take advantage of all the type checking, intellisense, Prettier code formatting, etc. And then copy the transpired code and my data types into your tool.

Sounds interesting, though there are some differences between TypeScript and the rules language. Please post your findings when you're done!

can't have let statements below the "allow create:" blocks, no editor support that formats expressions so I can read them

All great suggestions.

Overall looks like a high quality project.

Thanks!

[jmagaram]

Well I gave it a go. Check this out! It was a pleasure to type the rules in TypeScript in VS Code with the compile-time type-checking, auto-complete, Prettier, refactoring, etc. I pasted the transpiled code into the rules file and the emulator didn't complain about any syntax errors. I made a few small changes - semicolons into commas - and was able to paste it into Fireward.

https://tinyurl.com/mwx9k0zo

===

I'm continuing to work on this but from the perspective of defining the types in the rules language with special Map String Timestamp types. So the rules become the source of truth, like Fireward. Then use a mapping to generate the types that can be used in client code. I think this approach has a lot of promise. Will let you know soon.

[bijoutrouvaille]

thanks for this research @jmagaram. Please keep posting updates here so that other users could benefit from it.

[jmagaram]

Quick update. I created a bunch of types used in the rules engine like Request, Resource, Auth, Map, List, Token, etc. and was able to safely author a bunch of security rules in TypeScript. I really liked the VS Code experience - intellisense made it go pretty quick and without errors. Then I compiled them and copied them into my rules file. All worked fine. Here are the types if you want to play around with them...

https://github.com/jmagaram/grocery-list-ng/blob/main/src/app/firestore/security-rule-types.ts

Hassles - The manual copying. Trying to enforce read-only and other structural requirements like the model can only have certain keys. Your tool is great for that. I'm still struggling with how to deal with optional fields. Should they be null or missing entirely? Without optional chaining, ?. working with my object model might be a hassle.

Other cool stuff - I figured out how to use TypeScript's mapped and conditional types to automatically convert a deep data model you create to types the rules engine understands like Map and String. Also, it is possible to provide type safety whenever property names need to be typed in, like in the get(["person","email","name"],-1). This type below generates all valid property names in an object.

type PropertyNames<T extends object, M extends 'deep' | 'shallow'> = {
  [TKey in keyof T & (string | number)]: T[TKey] extends Function
    ? never
    : T[TKey] extends object
    ? TKey | (M extends 'deep' ? PropertyNames<T[TKey], M> : never)
    : TKey;
}[keyof T & (string | number)];

Here is how I defined the Map type to make use of it...

export interface MapFire<T extends object> {
  _kind: 'Map';
  keys: () => ListFire<StringFire>;
  size: () => number;
  values: () => ListFire<T>;
  diff: (other: MapFire<T>) => MapDiff;
  get: <U, MODE extends 'deep' | 'shallow' = 'shallow'>(
    key: MODE extends 'shallow'
      ? StringFire | PropertyNames<T, 'shallow'>
      : (StringFire | PropertyNames<T, 'deep'>)[],
    defaultValue: U
  ) => MODE extends 'shallow'
    ? PropertyTypes<T, 'shallow'> | U
    : PropertyTypes<T, 'deep'> | U;
}

I haven't incorporated this kind of safety into MapDiff or other places but should be straightforward.

I'm working on my first Firebase project and am doing stuff breadth-first. Now trying to get Functions working. So I haven't gotten too deep into my whole data model and security rules. Once I do that I'll see whether what I built here is truly useful.

=== UPDATE ===

I'm not sure my MapFire type is working right. See comments in my original source code. Can this type be defined in any way other than recursively? Don't have time to get into it right now.

[jmagaram]

More progress on this. See https://github.com/jmagaram/grocery-list-ng/tree/main/firebase/firestore

After compiling the security rules from TypeScript to Javascript I've got a simple script that does some regular expressions to convert it so it can be used in the rules engine directly. The original TypeScript is annotated in a few special ways like this comment is used to generate the match expression.

\\ MATCH \documents\users\{userId}

Probably the coolest thing is the get function on maps - to get a value from a path - is expressed in TypeScript using a type-safe lambda like this...

let ownerEmailVerifiedMatches =
  // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
  doc.get(-1, (i) => i.owner.email!.verified) ===
  auth.get(-1, (i) => i.token.email_verified);

...and the script turns it into...

    let ownerEmailVerifiedMatches = 
    doc.get(["owner", "email", "verified"], -1) ==
        auth.get(["token", "email_verified"], -1);

Overall I think this approach works really well. I get type-safe access to all objects used in the rules engine.

I think it might be possible to add in some of your special features like read-only fields and checking for required and optional parameters by introducing special functions in the TypeScript that get transpiled into the rules file, kind of like how I did with the get function.

[bijoutrouvaille]

@jmagaram first, apologies for the late reply; I'm traveling. This is very cool, and I'm glad to see this experiment is working out for you. You are clearly very effective with TypeScript. Before I made Fireward, I also considered making use of TypeScript resources in a similar way, but I decided that the full control of parsing that one gets by writing a proper parser might be more maintainable than the approach with regular expressions. And so it is interesting to me to see how far the alternative path can take.