`filter` does not accept "special" native functions - improve error message

[hugoclrd]

Bug

filter throws an error when the test function is a native one, such as is-err, is-ok, is-some, is-none. The error is: use of illegal / unresolved function 'is-some. The following code snippets are considered invalid because of the error. ⬇️

(filter is-none (list (some u1) none))
(filter is-some (list (some u1) none))
(filter is-err (list (ok u1) (err u1)))
(filter is-ok (list (ok u1) (err u1)))

Edit

It looks it the case for map and fold. The following snippets throw the same error

(map is-some (list (some u1))) ;; expected output: `(list true)`
(fold is-eq (list true false false) true) ;; expected output: true 

Partial workaround

Custom functions can be defined like so but it's not ideal

(define-private (is-err-custom (res (response uint uint))) (is-err res))
(define-private (is-some-number (n (optional uint))) (is-some n))

[obycode]

This seems to be intentional in the type_checker code. There is a related comment here:

    // we will only lookup native or defined functions here.
    //   you _cannot_ map a special function.
    let function_type = get_simple_native_or_user_define(function_name, checker)?;

The functions mentioned in this issue are all "special" functions. @kantai, can you explain the reason for this restriction?

[obycode]

@reedrosenbluth pointed me to SIP002, quoted below, but I am not clear yet on why.

filter map and fold functions may only be called with user-defined functions (i.e., functions defined with (define-private ...), (define-read-only ...), or (define-public ...)) or simple native functions (e.g., +, -, not).

[kantai]

Yes, it's a simplifying constraint for the type analysis. By only allowing functions with fully-specified type signatures for those higher-order functions, the type analysis doesn't need to do any additional analysis to type check the higher-order functions. In many cases, this is a much too aggressive constraints. is-some, etc. would be very easy to type check, and even the arithmetic operations would be fairly easy to typecheck, but it would require a more sophisticated system for typechecking those "special" functions.

Some special function would definitely not be allowed without a new language feature (like trying to fold a map-get?).

[obycode]

That makes sense. Thanks @kantai. Since the workaround is very easy (defining a wrapper function), adding that slightly more sophisticated type-checking can be low priority.

Let's use this issue then to track adding a better error message from clarinet for this case. Sound good @hugocaillard?

[obycode]

We also should open an issue for the documentation to explain this as well.

[hugoclrd]

Some special function would definitely not be allowed without a new language feature (like trying to fold a map-get?).

I guess that would require partial application, which would be awesome (things like (map (map-get? datamap) (list u0 u1 u2)) or (filter (is-eq u0) (list u0 u1 u0 u2)). But it's a bit off-topic.

Let's use this issue then to track adding a better error message from clarinet for this case.

Sounds good 👍

[njordhov]

Just having filter accept these four simple predicates (is-some, is-none, is-err and is-ok) would be an improvement, with is-some being a particularly useful filter function.