apache
[Format][Docs] Clarify (remove?) usage of the term "logical types"
In several places in the Arrow specification and documentation we use the term "logical types", although we don't use it consistently and we don't actually have physical types (only physical layouts) to contrast it with.
Current usage
The Columnar Format doc page has a section called "Logical Types" (https://github.com/apache/arrow/pull/41685) to contrast those types from the physical layouts:
The Schema.fbs defines built-in logical types supported by the Arrow columnar format. Each logical type uses one of the above physical layouts. Nested logical types may have different physical layouts depending on the particular realization of the type.
It explains an Array as having a logical data type, where "Each logical data type has a well-defined physical layout."
The authoritative Schema.fbs also uses the term:
https://github.com/apache/arrow/blob/07a30d9a5784852187d100660325b8c12b4ff6c8/format/Schema.fbs#L18
although it uses the term also in a "correct" way (but incorrect in the way we define the term currently):
https://github.com/apache/arrow/blob/07a30d9a5784852187d100660325b8c12b4ff6c8/format/Schema.fbs#L101-L105
The Python docs (https://arrow.apache.org/docs/15.0/python/data.html#type-metadata):
We use the name logical type because the physical storage may be the same for one or more types. For example,
int64,float64, andtimestamp[ms]all occupy 64 bits per value.
Further, in various implementations the term is obviously used as well.
In the Terminology section of the Columnar Format docs (https://arrow.apache.org/docs/15.0/format/Columnar.html#terminology), we define it as:
Logical type: An application-facing semantic value type that is implemented using some physical layout. For example, Decimal values are stored as 16 bytes in a fixed-size binary layout. Similarly, strings can be stored as
List<1-byte>. A timestamp may be stored as 64-bit fixed-size layout.
which is mostly correct with our current usage ("using some physical layout"), but it is also confusing that it explains strings as List<1-byte> as we have a different physical layout used for strings
Previous discussion
Generally we use the term relatively consistently to contrast "logical types" from the "physical layouts", but confusion around the terminology has come up regularly (what are "physical types" then? And extension types are essentially "logical types", but annotating our own logical types). This was specifically discussed in https://github.com/apache/arrow/issues/14752.
@amoeba proposed (https://github.com/apache/arrow/issues/14752#issuecomment-1550157549):
Still some discussion to be had about avoiding "logical" vs. "physical" in favor of "types" and "layouts" and possibly updating the format docs comprehensively
I was definitely confused by the term "logical type" when I became involved with Arrow several years ago and came slowly to the understanding that every system seems to use the terms logical and physical in a slightly different way and that it is more of a spectrum than a dichotomy.
We also have the term "encoding" used in two different ways: "run end encoded" is a "type" but "dictionary" is basically a special case in Schema.fbs and in the Arrow C Data interface. Implementations typically present these both as "types" (e.g., pa.dictionary()).
I agree that it is confusing but I am not sure what it should be replaced with. I suppose they could just be called "types" and "layouts", perhaps with dictionary encoding being a layout rather than a type?
I would maybe generally talk about "data types" to not just have "types". For the layouts I think the adjective "physical" is still useful, or otherwise more consistently use "memory" as adjective. So that would give "physical layouts" or "memory layouts" and "data types"
Issue resolved by pull request 41958 https://github.com/apache/arrow/pull/41958
