Transitional Field Statuses: Allow fields to be marked 'introduced' and 'removing' in addition to 'required' and 'optional'

[clord]

Problem

Modifying required fields in our API can lead to compatibility issues between clients and servers, especially when fields are added, removed, or have their required/optional status changed. The key challenges are:

1. When adding a new required field, clients using older versions of the API may not include the new required field in their requests to the updated server, resulting in validation errors and failed requests.
2. When removing a required field and a client is updated before the server, it may stop sending a field that the server still expects as required, causing processing errors on the server side.
3. Promoting optional fields to required, or demoting required fields to optional, can create inconsistencies and unexpected behaviours if not managed carefully.

Making all fields optional is not a feasible solution because:

• It increases the complexity of runtime validation.
• It allows for invalid or meaningless combinations of data, potentially leading to logical errors in the application.

Proposal: Transitional Field Statuses

Implement transitional statuses for fields to manage their lifecycle without breaking existing client/servers:

becoming-required — introducing a new field that will eventually become required

• Writers (data producers) must include this field in their output.
• Readers (data consumers) should handle cases where this field is absent.

becoming-optional — planning to deprecate and eventually remove an existing required field

• Writers must continue to include this field in their output.
• Readers should prepare to handle cases where this field might be missing in the future.

This approach allows for a gradual transition.

• When adding new fields, mark the new field as becoming-required to ensure writers start including it while readers remain compatible with data that may not contain it. Once all writers are updated, we can then mark the field as required, so that clients can begin assuming it's always present.
• When removing fields, mark the field as becoming-optional to signal that it will eventually be removed, allowing readers to adjust without immediate impact on writers. Once all readers are updated to no longer expect the field, we can remove the field, and writers can stop emitting the field.

Reader/Writer vs Client/Server

Please note the distinction between client/server and reader/writer. Clients write to structs, and read from structs, as do servers. This distinction is only meaningful to readers/writers.