Metadata violates standard and misses expectations

[jaraco]

In https://github.com/pypa/setuptools/issues/1390#issuecomment-762410744, I learned a few things:

• PEP 345 only indicates Description as supporting multiline values.
• PEP 345 indicates a pretty arbitrary prefix for continuation lines in multiline values ( |) and alludes to that being RFC 822 compatible, but RFC 822 only expects a single space or tab for a continuation.
• Distutils violates the PEP spec with another arbitrary continuation ( ), additionally violating RFC 822.
• Users would like to include multi-line values in more than just the Description field.
• The RFC 822 long header fields spec isn't particularly well-designed for formats where the newlines are actually meaningful.
• PEP 566 describes a JSON-compatible format, but that format suggests to pass through the values still with continuation characters.

Here's what I propose:

• Make explicit the limits and divergence from RFC 822.
• Drop the | value and update the spec to honor the 8-space format.
• Allow any field to have newlines (from a metadata format perspective).
• Explicitly indicate which fields may or may not have newlines (similar to how each field indicates multi- or single-value).
• Libraries (like importlib.metadata) should provide the metadata with these encoding details (continuation characters) removed, either naturally or through the JSON form.

@pfmoore What do you think of the proposal? Would you expect a PEP for this proposal?

[KOLANICH]

Drop the | value and update the spec to honor the 8-space format.

distlib will be required to be fixed.

[pfmoore]

See here for a discussion on the matter. Your proposal sounds reasonable, but you should co-ordinate with the people in that discussion.

As this is a change that would affect software interoperability, it should technically require a PEP according to the documented process. I would not expect such a PEP to be large, or controversial, assuming there were no substantial objections during the discussion.

Personally, I think that having a PEP for every change like this might be too much, but that's a PyPA governance discussion, so rather than making a unilateral decision, I'll raise that on the Packaging discourse (Edit: Opened a discussion here).

[merwok]

Latest metadata is 2.1, with description supported in the message-body, which avoids the delimiter/indent issues noted here: https://peps.python.org/pep-0566/#description

This does not solve the issue. But distutils is gone, and setuptools can be changed (if still needed) to follow the specs (the living docs that replace PEPs as reference). If the specs still say incorrect things about RFC *822 format, there is now a process for minor and major revisions.

So I think this could be closed @jaraco

[jaraco]

While I agree the newer landscape makes this problem easier to address, I believe as you say the issue is still unsolved. I'd like to see generalized support for multiline fields and not just a suitable workaround for the Description. I think this issue tracks that need suitably.