Add EEP for native records

[bjorng]

[bjorng]

there are no paths between "maps -> native records". Perhaps this is intentional

We didn't really think about migrations from maps. Your suggestion seems reasonable. We will discuss this in the OTP team.

[bjorng]

Functions that expect classic records, tagged tuples, and maps could have new function clauses added to handle native records in a backwards compatible way, unless I am mistaken. It seems that due to not being compatible with maps or tuples there would be very little ability to update existing functions to return native records.

Yes. If a function returns a tuple record, all we can do is to create a new function that returns a native record.

Or is the expectation that Erlang/OTP code will use different data structures depending on how old it is?

To some extent, yes. I think that is already the case.

If they all were to largely use native records then this friction go away, making interop between languages would be a much better experience.

Agreed.

[lpil]

Thank you @bjorng.

To confirm: all fields must have names? There are no positional fields?

Can one define a record which does not have any fields?

-record #none{}.
-record #some{value = term()}.
-type option() :: #none{} | #some{}

[bjorng]

To confirm: all fields must have names?

Yes.

Can one define a record which does not have any fields?

Yes.

[dgud]

Great work!

Would it make sense to have finer grained control on who can do what? For example restrict creation to the defining module while still providing access to fields; or read-only fields outside the defining module. Probably doesn't matter for 95% of use cases I reckon.

We are trying to limit the scope here to try to get it in to 29.

Hmm, this would require some additional syntax, 'private' | 'protected' | 'public' (borrowing from ets access types), personally I don't think this is necessary nor wanted, Opaqueness on fields have been up for discussion but dropped for now at, implementation and reflection reasons.

[RaimoNiskanen]

@lpil wrote:

Is the only difference between the native and classic record syntaxes the # character in the name of the definition? This seems like it will be very error prone, ...

It is a different format, not only the # character, it is like record creation:

-record(foo, {a, b, c}).
%% vs.
-record #foo{a, b, c}.

So there is also a comma and parentheses that are different.

[tsloughter]

I don't see the concern raised that we'll now have record, native records and maps. I know they all serve different purposes, but the differences are slight and new users definitely get confused about which to use. I don't think it should be a blocker to adding a new data structure that can help improve devex or performance when choosing the right one but it does worry me that this can't replace records.

Related to not replacing records, I think -record being overloaded is confusing. To be clear, I don't like the idea of -native_record either.

[essen]

Hmm, this would require some additional syntax, 'private' | 'protected' | 'public' (borrowing from ets access types), personally I don't think this is necessary nor wanted, Opaqueness on fields have been up for discussion but dropped for now at, implementation and reflection reasons.

Right it's something that likely doesn't have to be done at runtime, but it is important to consider at least in the documentation part, as internal fields definitely shouldn't be documented. Read-only fields can be marked as such easily in the text. But this depends on what the documentation will look like I suppose.

[bjorng]

Update: there can now be two distinct errors when Rec#rec.field fails: {badrecord,Term} or {badfield,field}.

[RaimoNiskanen]

@tsloughter wrote:

Related to not replacing records, I think -record being overloaded is confusing. ...

"Not replacing records" is really phrased: not "Replacing all tuple record usage scenarios.", and "Replace most tuple-record usages without having to update anything but the declaration.".

With that in mind I think overloading -record may be more good than bad.

[tsloughter]

@RaimoNiskanen unless it outright replaces, 100%, named tuple records I think using -record will cause additional confusion to the confusion that will already exist from there being 2 types of records and maps.

I take it there are a million reasons -type shouldn't and can't also define a record:

-type #pair(A, B) :: #pair{
                           first :: A,
                           second :: B
                          }.

At first I wanted #{} but that is already used by maps!

I can concede there is no good alternative to -record... except maybe -frame (I kid, I kid). But making the times that tuple records are needed be as small as possible may be important though, to not have Erlang grow its reputation of confusing. Probably ets usage is the biggest one there.

[lpil]

I do share the concern that the similar syntax is confusing, and the difference being 2 characters of punctuation makes it challenging to differentiate between the two when reviewing code.

I can concede there is no good alternative to -record... except maybe -frame

struct was the one option that came to mind.

-struct #state{
  values = [] :: list(number()),
  avg = 0.0 :: float()
}.

Elixir does already have "defstruct", though that construct does seem very similar to this proposal in design and purpose.

[potatosalad]

Question: Behavior when adding a field across distributed nodes

Consider this scenario:

Node A (old code):

-record #state{count = 0, name = "default"}.
State = #state{count = 5, name = "server1"}

Node B (new code):

-record #state{count = 0, name = "default", version = 1}.

When Node A sends a State record to Node B:

1. Reading works fine: Node B can read State#state.count and State#state.name since those fields exist in the record value.
2. Reading the new field fails: State#state.version will raise {badfield, version} because the field doesn't exist in the record value (it was created with the old definition).
3. Pattern matching is unclear: Can Node B do #state{version = V} = State? Based on the spec: "Pattern matching fails if the pattern references a FieldK and the native-record value does not contain this field." This would fail.
4. Updating appears problematic: Can Node B do State#state{version = 1} to add the missing field? The EEP states: "A native-record value is updated according to its native-record definition" and "An update operation fails with a {badfield,FN} error if the native-record update expression references the field FN which is not defined (in the structure definition)."

Issue: This seems to check against the definition on Node B, not the fields in the value from Node A. It's unclear whether the update would:

• Succeed and add the version field to the record value
• Fail because version doesn't exist in the value
• Create a new record with all three fields (losing the old value's identity)

[potatosalad]

Question: Field renaming across nodes

Consider this scenario where a field is renamed:

Node A (old code):

-record #user{id, username, city}.
User = #user{id = 1, username = "alice", city = "Stockholm"}

Node B (new code - username renamed to name):

-record #user{id, name, city}.

When Node A sends User to Node B:

1. The record value still contains username: The field names are captured when the record is created, so the value has fields [id, username, city].
2. Node B cannot read the new field: User#user.name raises {badfield, name} because the value doesn't have a name field.
3. Node B CAN still read the old field: User#user.username should work because field access doesn't consult the definition—only the value. But is this problematic because username isn't in Node B's definition?
4. Pattern matching breaks: case User of #user{name = N} -> N end fails because name doesn't exist in the value.

Issue: Field renames appear to be breaking changes in distributed systems.

The EEP states:

to perform read operations on native-record values — accessing native-record fields and pattern matching over native-record values — the runtime does not consult the current native-record definition.

This means reading works purely based on the value's fields, not the definition. So:

• Old nodes can't read renamed fields (they use the old name)
• New nodes can't read renamed fields (the value has the old name, pattern matching fails)
• Updating is impossible (the definition has the new name, the value has the old name)

[potatosalad]

Question: Removing a field

Consider this scenario:

Node A (old code):

-record #config{host, port, legacy_timeout}.
Config = #config{host = "localhost", port = 8080, legacy_timeout = 5000}

Node B (new code - legacy_timeout removed):

-record #config{host, port}.

When Node A sends Config to Node B:

1. The removed field still exists in the value: The record value contains [host, port, legacy_timeout] because that's what Node A created.
2. Reading removed fields: Based on the spec, Config#config.legacy_timeout should still work on Node B because field access doesn't consult the definition—it only checks if the field exists in the value. This means:

• Code on Node B can accidentally read fields that "don't exist" in its definition
• Linters/dialyzer on Node B would flag legacy_timeout as an error, but it works at runtime
• Might this create a confusing situation for developers?

3. Pattern matching works: case Config of #config{host = H, port = P} -> ... end works fine (only matching fields that exist in both value and definition).

Issue: The interaction between field access (which ignores definition) and pattern matching (which may or may not check definition) is unclear.

[bjorng]

It turns out that I had failed to fully update the EEP regarding when a record definition is consulted (or not). In our internal meetings we had decided that the definition for a native record is only used when creating a record. All update and read operations will only refer to the value of record (which includes the names, whether it was exported at value-creation time, the name of the record, and of course all values).

I've now pushed another commit to hopefully make that clearer.

[michalmuskala]

If the definition is never consulted on update, how do you upgrade values to the new definition? Do you have to explicitly deconstruct & reconstruct and keep version of the code doing that for all possible values that are live in the system? With the previous design, upgrading a value (given all new fields have a default) was potentially as simple as OldValue#record{}, and could handle all sorts of versions. AFAIK the intention of the previous design was to define update in terms of creation + read from the existing value for fields that weren't explicitly provided

[bjorng]

If the definition is never consulted on update, how do you upgrade values to the new definition? Do you have to explicitly deconstruct & reconstruct and keep version of the code doing that for all possible values that are live in the system?

Yes. Upgrades must be done explicitly.

With the previous design, upgrading a value (given all new fields have a default) was potentially as simple as OldValue#record{}, and could handle all sorts of versions.

While the design was simple, we discovered that the actual implementation was far from simple, especially when more than one node is involved (or even when loading from the disk native records written by a previous instance of the runtime system). Also, we are not sure that is easy for the user to understand and handle all the implications of automatic upgrades of native records.