[feat] when parsing lists - need alternatives to silently dropping invalid values

[Elijas]

Problem: https://gloochat.notion.site/Array-parsing-logic-1b6bb2d262168014b022e1604860853d

Solution: https://discord.com/channels/1119368998161752075/1365282045755916360/1365369529697435802

I feel that in the spirit of sharp prompts (#12fa) baml should never leak parsing-related workarounds and flags into the prompts when they don't help inform the LLM in any way

Case in point:

name Type[] @parse.strict this will leak or null into the prompt for no reason (it's a parsing related thing)

How about a solution 4

Consider (Lead | _.Error)[] instead of Lead[] this will return Error classes with the raw data and maybe an explanation of why it failed

Or even better, solution 5

name Type[] @parse.lenient (too vague 👀) or name Type[] @parse.loose (too vague 👀) orr name Type[] @errors.as_nulls (readable without using docs 👍) which later allows syntax flexibility for: name Type[] @errors.as_objects which would be the solution 4 i.e. (Lead | Error)[] but with a better syntax

• why it's a better syntax:
  • @ explicitly hints that this information is for the parser and not the LLM
  • (Lead | _.Error)[] might give the vibes that llm is the origin of the error object)

---

One more idea:

• we already have the mechanism for returning "warnings" from baml parser, and that's the @check
• why not name Type[] @errors.warn which would fill the check-warnings list with not only @check errors but also list object parse-error-drop warnings which would include all the transparency data

---

name Type[] @errors.ignore (which is the same as) name Type[] which would be the default (implied)

• Why still have optional @errors.ignore even when it's default? So that it would be possible to set @errors.as_nulls application-wide and add exceptions to that as the default through @errors. for specific fields. Application-wide defaults would be perfect e.g. next to default_client_mode "sync" ,e.g. default_error_handling with possible values any of:
• "return_as_nulls" | "ignore" | "return_as_err_objects" | "raise_errors" "return_as_check_warnings" (the @check-like implementation)

TL;DR my proposed spec:

• sensible default for most users and most use cases
• which can be modified with params for anyone willing to exchange simplicity for flexibility. Those params should be:
  • clearly directed at baml parser and not the LLM
    • @... seems the most readable, since it's about baml behavior
    • editing Type[] in any way makes the function definition more confusing for users, IMO,
      • but I could imagine (Type!)[] but that's way less readable than @ and seem too related to (Type?)[] when it's not
  • shouldn't change LLM prompt in any way (unless the LLM has to get involved)
    • (Type?)[] is for when you actually want the LLM to output nulls
• (Optionally) ability global default, but this comes with cons too, as the .baml code is no longer self-sustained in describing how it functions (important for portability of code examples accidentally without introducing bugs etc). Maybe at least baml class-scope defaults? like @@errors.ignore

[linear[bot]]

BAML-379 [feat] when parsing lists - need alternatives to silently dropping invalid values

[aaronvg]

Bumping this -- we just ran into another issue with this

[aaronvg]

another instance (at this timestamp) https://youtu.be/qPPqNV5J2N8?t=490

[aaronvg]

another instance https://discord.com/channels/1119368998161752075/1375236182342107206/1375532178275832028

[runnedrun]

Hey all, any updates on this? Its an infrequent, but quite frustrating problem when I have items disappearing from arrays with no hints as to why.

[sxlijin]

Hey, sorry for the radio silence on this. No updates yet, we've been heads down on other language features, but we haven't forgotten about this and will try to get to this soon!