Make set and all build-in objects cross-phase persistent

[gus-massa]

From a recent article, one side observation was that it is confusing that sets are not cross-phase persistent. Let's make all build-in object cross-phase persistent.

(This could be implemented in racket1.)

[rocketnia]

This seems related to #42.

[rocketnia]

I see value in having a consistent experience between the language kernel and the library ecosystem, so that someday the kernel can easily be relegated to library status if a better kernel design comes along. With that in mind, I prefer one of two specific directions here:

• Most built-in types are not cross-phase persistent, to match most types in library-defined languages.

• Most types in library-defined languages are cross-phase persistent, to match most built-in types.

I think having most types be cross-phase persistent is more useful, so I'd be happiest to see something like this:

• Most struct type definitions have the result of declaring a cross-phase persistent submodule, defining the struct type in there, and requiring it.

• When necessary, the user can explicitly use a different struct type definition that makes non-cross-phase-persistent types.

(As a digression: I think there's also an interesting middle ground between these two kinds of struct definition. There could be a struct type definition that created a non-cross-phase persistent submodule, and that submodule would define not only the struct type but also some operations to help convert struct instances back and forth to the different phases' versions of the struct type. However, I'm not really confident I can say the specific interaction points where values would be converted as they passed through, and in any case, I can't think of any place I would actually need this.)

[gus-massa]

I think it is for historical reasons. Most of the libraries were written before the existence of the cross phase persistent modules (I have to lookup the name every time) and nobody care to write them in a cross phase persistent way because the concept didn't exist.

After the addition nobody take the time to rewrite all of them. Most changes are not so big, but it is a lot of work.

Anyway, I think it would be nice to make this change. (It's a tiny back compatibility break.) For example why hash are cross phase persistent but set are not? (It's clear in the internal organization of the implementation, but there are not good reasons for the users.) It's more natural to have all the build-in types cross phase persistent are recommend libraries writers to make the same effort.

[rocketnia]

Most changes are not so big, but it is a lot of work.

Yeah, I think it takes a significant amount of work to migrate libraries to be cross-phase persistent.

Come to think of it, it's not just the struct types. Modules themselves should be cross-phase persistent by default so that the generic methods and property implementations of cross-phase persistent struct types can make use of them.

As much as the migration effort is a factor here, there are also a few things that have made it hard for me, as a library writer, to justify doing this migration for my libraries:

• I think the only module that's documented to be cross-phase persistent is racket/kernel, but the documentation says "Currently, the set of bindings [in racket/kernel] is not especially small, nor is it particularly well-defined, since the set of built-in functions can change frequently. Use racket/kernel with care, and beware that its use can create compatibility problems." If a library author does go to the trouble to define things in cross-phase persistent ways, then they have to use racket/kernel, so their work is resting on undocumented behavior that's explicitly subject to change.

• As a convenience layer over programming in '#%kernel, I've attempted to fully expand some submodules and then replace their language and imports after the fact to make them cross-phase persistent. As a result, I've found (struct ... #:methods ...) expands to code that doesn't conform to the cross-phase persistence requirements. So right now, it seems like cross-phase persistent struct types can't implement generic interfaces. And since the documentation declares prop:custom-write and prop:equal+hash to be deprecated in favor of gen:custom-write and gen:equal+hash, a lot of struct types can only be cross-phase persistent if they rely (not only on undocumented racket/kernel bindings that are explicitly subject to change, but also) on explicitly deprecated features.

[gus-massa]

Recently @samth made stx.rkt cross phase persistent in https://github.com/racket/racket/commit/c6dd371ed6ee1c583f5e8241bf6425009f729192

You are right, he used only the '#%kernel language, so it's more painful than what I expected.

[jackfirth]

What are the main use cases for cross-phase persistent modules? Can we put together a short list summarizing when they're appropriate?

[gus-massa]

One useful part is that they can be loaded once and shared in all phases, so they save some memory and load time.

Another useful part is that they can define objects that are shared between phases. For example in

#lang racket
(define s (with-handlers ([exn:fail:syntax?
                           (lambda (e) (car (exn:fail:syntax-exprs e)))])
            (eval #'(begin
                      (define-syntax (x stx)
                        (define phase (variable-reference->phase (#%variable-reference)))
                        (raise-syntax-error #f "" #`(-> #,phase <-)))
                      (x)))))
(displayln (syntax? s))
(displayln s)
(displayln (variable-reference->phase (#%variable-reference)))

Output:

#t
#<syntax:unsaved editor:7:52 (-> 1 <-)>
0

Note that the there is a syntax that is created inside a macro inside the eval. The phase at that point is 1 so it creates #'(-> 1 <-). After that the error is raised and then the exprs part is extracted. So now the main part of the code that is running at phase 0 has a syntax that was created at phase 1. So it is important that phase 0 and phase 1 recognizes the object as a syntax.

It has very specific uses, and it's even difficult to create an example. To teleport an arbitrary object you must create your own subclass of error. So I guess the cross-pass definition is only important in a few objects that are commonly raised in errors.

It's probably too difficult to make everything cross-pass persistent until there are more friendly tools, so I'm 99% for delaying this until racket3.

[rmculpepper]

I think that making a library fully cross-phase persistent would be hard, and you'd have to sacrifice many useful features, like contracts. But I think that making a few interfaces (in the form of struct-type properties, possibly wrapped by generics) effectively cross-phase persistent should be relatively easy.

On the other hand, I think that cross-phase persistence is only rarely the right solution to a problem, since even cross-phase persistent structures aren't bytecode-serializable. I think it would be useful to have a generic, extensible datum->expression procedure. There's a basic, limited version in syntax/parse/private/datum-to-expr.rkt; I plan to generalize it and make a PR soon.