Custom validators versus built-in types/validators

[dumblob]

I wonder what are the plans for solving the current status quo regarding input validation?

Grist has some very basic data "types" which it validates. The number of supported data "types" and their variants is very small and thus far from enough for real-world use.

There'll be increasing pressure to draw a line between user-facing types (date, picture, video, plaintext, time, Markdown, UUID, etc.) and "backend types" (usually some serialization formats like JSON, 7bit ASCII, HTML, BASE64, etc.). It is though impossible to draw such a line simply due to the nature of such formats - namely arbitrary embedding of one into each other arbitrarily recursively.

So my suggestion would be to generalize the support for types in a way there will be no types whatsoever. And everything will be done simply using input validators. There can (should) be some predefined validators covering at least the current "types". User-added validators should be indistinguishable from those provided by Grist by default.

I'd recommend introducing a "hidden" namespace for all such validators (both the default ones as well as the user-added ones) to allow versioning. E.g. namespace matching the POSIX extended regular expression ^v[1-9][0-9]*$ (e.g. v1) will be reserved for Grist (i.e. your company internal). Namespace ^future$ will be reserved for a publicly editable centralized repository of "proposals" of validators which will be considered for inclusion into v... namespace (and thus removal from future) when releasing new Grist versions. Any other namespace will be arbitrary. If no namespace will be given, only the v... namespace will be searched through.

It'd then be easy to select Date for a column or just switch to future in the selector and suddenly thousands of existing validators would pop up.

I guess first custom validators which would appear in future and downstream client installations would be slightly modified versions of those default validators - e.g. because of conventions used in the specific country (Asia, Arabic countries, Germany, ...) the Grist installation is meant to serve.

One more thing - validators should be ideally usable 1:1 on client devices (from both the Grist interface as well as from custom downstream widgets) without any server<->client communication to save bandwidth and to allow for <1ms latency for superior user experience. So maybe restricting validators to pure self-contained JavaScript sounds like a good start.

Thoughts?

[dsagal]

Could you describe a couple of different specific examples of what you'd like to become possible with this?

Data types in Grist involve more than validation, but also formatting options (e.g. date and number formats), type-specific ways to display them (e.g. booleans can be displayed as toggles, and attachments as thumbnails), and type-specific widgets for editing the values (date picker, choice dropdown, etc.)

I agree that it would be cool to allow users to extend what data types are available, and to be able to share what they build so that others can use those too.

[dumblob]

Could you describe a couple of different specific examples of what you'd like to become possible with this?

I'd like to "throw away" (i.e. ignore) almost all data types Grist supports currently and use custom ones. As mentioned above the reason is the inability of Grist to work with specific formats, conventions in a given cultural context etc.

Data types in Grist involve more than validation, but also formatting options (e.g. date and number formats),

Not needed for general use cases. It'll just show as "raw" data in "Grist views". The only thing which matters is that Grist will guarantee validity of the accepted/stored data.

type-specific ways to display them (e.g. booleans can be displayed as toggles, and attachments as thumbnails),

Again, not needed for general cases. In the worst case there can be optional converter function paired with a given validator and this converter will convert the data to some of the constrained types Grist natively supports just to offer a quick look - but with a big red triangle with an exclamation mark that the shown data are not 1:1 to what is in the DB.

and type-specific widgets for editing the values (date picker, choice dropdown, etc.)

Yes, validation is part of the whole editing process. Therefore I mentioned validators shall run (also) on client. And of course each validator can optionally provide also a default editing UI. But it shouldn't be necessary as I expect all the editing to happen in custom widgets - see also https://github.com/gristlabs/grist-core/issues/37#issuecomment-1048625090 .

I agree that it would be cool to allow users to extend what data types are available, and to be able to share what they build so that others can use those too.

:+1:

[dumblob]

Potentially related: fully automated generation of typescript declarations from validators: https://github.com/colinhacks/zod .

[dumblob]

Just realized that this'll be inevitable to support migrating basically any existing DBs and workflows to Grist. In other words if Grist wants to become "enterprise-level" then this is an absolute must-have or Grist will be usable only for new workflows which are generally being discouraged in larger enterprises (in which case only migration is the way to go).

Speaking of workflows, any plans to integrate with the BPMN and DMN and CMMN widgets/services?

[bamboowonder]

It would be nice to have simple masking for text input validation. As an example for phone numbers: (999) 999-9999 or postal codes: A9A 9A9. Where the owner/designer that add validation if needed (yes, I know every country has different masking/validation requirements for phone/postalcode etc, which is why letting it be specified simply by alpha or number or a few others would be great.