Add a `$private` property for tokens

[c1rrus]

Should our spec add an optional $private property, along similar lines to the one proposed for StyleDictionary by @silversonicaxel? It would instruct tools not to display or export a token by default (though tools may provide an option for users to override that and still see / export private tokens if they wish). Its value is a boolean. true makes the token private, false makes it (explicitly) public. If there is no $private property, the token is public.

As per the thread on the StyleDictionary issue, I think this could be convenient way for teams to exclude (or flag) tokens that they don't consider part of their public API from code, design tools, styelguides, etc.

For example, the following DTCG token file:

{
  "red": {
    "$type": "color",
    "$value": "#ff0000"
  },
  "green": {
    "$type": "color",
    "$value": "#00ff00",
    "$private": true
  },
  "blue": {
    "$type": "color",
    "$value": "#0000ff",
    "$private": false
  }
}

...might cause a SASS export tool to output code like this:

$red: #ff0000;
$blue: #0000ff;

(Note how the "green" token is omitted because it is private)

Furthermore, if a token is an alias to a private token, then the dereferenced value must be output. E.g.:

{
  "red": {
    "$type": "color",
    "$value": "#ff0000"
  },
  "blue": {
    "$type": "color",
    "$value": "#0000ff",
    "$private": true
  },

  "danger-color": {
    "$value": "{red}"
  },
  "link-color": {
    "$value": "{blue}"
  }
}

...might be exported to SASS like so:

$red: #ff0000;
$danger-color: $red;
$link-color: #0000ff;

(Note how $link-color has the dereferenced value, because the "blue" token is private)

Finally, I'd suggest that this property should be inheritable from groups, just like $type is. This would make it easier to make all tokens within a group private. For example:

{
  "color": {
    "$type": "color",
    "$private": true,

    "black": {
      "$value": "#000000"
    },
    "white": {
      "$value": "#ffffff"
    }
  }
}

is equivalent to:

{
  "color": {
    "$type": "color",

    "black": {
      "$value": "#000000",
      "$private": true
    },
    "white": {
      "$value": "#ffffff",
      "$private": true
    }
  }
}

What do you think?

[jeromefarnum]

I really like the idea of of the $private property. We have, and use, a similar property in our token system. In addition to the use cases listed above, it’s also helpful to exclude color and dimensional operation tokens (handy shortcuts for replicating modifications throughout our token themes) that we don’t want to expose through the published tokens.

[silversonicaxel]

Just a note regarding this issue

Here I already setup a PR to achieve pretty much https://github.com/amzn/style-dictionary/pull/704 But the idea was abandoned, since it is a breaking change, and Style Dictionary people were trying to avoid this scenario.

That's why it was suggested and implemented the idea of built-in filters.

You can raise the discussion there.

I'm fine with the way I implemented, since I had scenarios where few design tokens needed to be filtered out. If you need to remove a group, it sounds logical wrong to have an entire set of design tokens that needed to be removed, but maybe I'm not foreseeing a situation like this.

In that case, a custom filter could be a solution?

[TravisSpomer]

I like this idea, but I wonder if true/false isn't sufficient. I can see two different "private" token possibilities:

• This token is a "base unit" of the design system but we don't need it exported to code (what we've been talking about)
• This token should never be used by designers, but we do want it exported to code

What would the latter look like? The Salesforce design system specifies a token brand.background.dark which is a pretty normal color token, but also brand.background.dark.transparent which is always defined to be the same color with an alpha value of 0. No one should ever be changing that token, but engineers would still want that value to be available in code. There might be examples of tokens that are perfectly valid but for whatever reason the design system maintainers don't want them showing up in design tools like Figma.

"$private": true isn't clear enough to say which of those two cases we're talking about. I like the words $hidden and $export to specify tokens to hide from design tools and tokens to be filtered out of exported code respectively, but I don't like how one is a "positive" word and one's a "negative" word; maybe the former could be $visible instead. So for the previous cases in this thread, instead of marking "$private": true you'd add "$export": false, and for something like the Salesforce transparent brand color you'd use "$visible": false.

Or maybe that idea of "for engineers only" tokens is too small of an edge case to matter?

[chazzmoney]

@TravisSpomer Your point that the term "private" is ambiguous is spot on, irregardless of the use case. We have a goal here (to avoid export) and a term that we are considering using to define that ("private"), and that could make things confusing in the future as it isn't explicit enough. If someone else has a different mental model of what "private" means... it could be problematic.

So, as you said, the real question is then what term is best? Something for the community to figure out.

[MatthiasDunker]

Some other wording ideas: expose: false, hide: true, helper: true.

[CITguy]

What if you just prefix the JSON property with an underscore?

• This is a pretty common convention among many programming languages to denote "private" members in a class/module.
• It's shorter to type than an explicit $private property.
• It won't affect alias lookup ({_foo.bar} and {fizz._buzz} would still work as expected).
• Whatever label you give it, the intent is to exclude tokens from direct consumption in translated output.
  • For languages that have module encapsulation, "private" tokens are those that are not exported from the module.
  • For languages that don't have module encapsulation, translation tools may choose to omit "private" tokens and resolve private aliases to their final value. Alternatively, translation tools may do nothing different and let things fall where they may.

{
  "_green": {
    "$type": "color",
    "$value": "#00ff00"
  },
  "red": {
    "$type": "color",
    "$value": "#ff0000"
  },
  "blue": {
    "$type": "color",
    "$value": "#0000ff"
  },
  "grass": {
    "$value": "{_green}"
  }
}

• _green is "private" (internal) due to naming convention.

Examples

JavaScript

ESM Format

const _green = '#00ff00';

export const red = '#ff0000';
export const blue = '#0000ff';
export const grass = _green;

CJS Format

const _green = '#00ff00';

const red = '#ff0000';
const blue = '#0000ff';
const grass = _green;

module.exports = {
  red,
  blue,
  grass,
}

SCSS Module

$_green: #00ff00;

$red: #ff0000;
$blue: #0000ff;
$grass: $_green;

CSS

exclude "private"

:root {
  --red: #ff0000;
  --blue: #0000ff;
  --grass: #00ff00;
}

include "private"

:root {
  --_green: #00ff00;

  --red: #ff0000;
  --blue: #0000ff;
  --grass: var(--_green);
}

Flutter

import 'package:flutter/material.dart';

const Color _green = Color(0xff00ff00);

class MyColors {
  MyColors._();

  static const red = Color(0xffff0000);
  static const blue = Color(0xff0000ff);
  static const grass = _green;
}

[romainmenke]

This is a pretty common convention among many programming languages to denote "private" members in a class/module.

Important to note that this is only a convention when a language doesn't have a native way to express private.

Once there is a native mechanic, the _ convention is immediately abandoned :)

[CITguy]

To piggyback off of the "private" vs "internal" discussion...

• private: exist after parsing
• internal: do not exist after parsing

Private

A private token is one that is still included in code after transformation has taken place (see my previous comment). The responsibility lies with the translation tool to identify "private" members, regardless of how private content is defined by the spec (whether it be _ convention or explicit configuration).

Internal

An internal token is one that is not explicitly defined in code after transformation has taken place.

If the spec allowed it, I'd be inclined to define internals using $defs (similar to how SVG <defs> behaves) and apply them using $ref (similar to SVG <use> or the same property in JSON Schema).

{
  // $defs will NOT be used for token _discovery_, but
  // will be used for token property _resolution_
  "$defs": {
    "colors": {
      "$type": "color",
      "green": {
        "$value": "#00ff00"
      }
    }
  },
  "red": {
    "$type": "color",
    "$value": "#ff0000"
  },
  "blue": {
    "$type": "color",
    "$value": "#0000ff"
  },
  "grass": {
    // alias of an "internal" token
    "$ref": "#/$defs/colors/green"
  }
}

This would resolve to the following flattened hierarchy...

{
  "red": {
    "$type": "color",
    "$value": "#ff0000"
  },
  "blue": {
    "$type": "color",
    "$value": "#0000ff"
  },
  "grass": {
    "$type": "color",
    "$value": "#00ff00"
  }
}

With the above configuration, you'd never see any reference to "green" in generated code, you'd only have the resolved value.

example (JavaScript: ESM format)

export const red = '#ff0000';
export const blue = '#0000ff';
export const grass = '#00ff00';

example (CSS)

:root {
  --red: #ff0000;
  --blue: #0000ff;
  --grass: #00ff00;
}

[Sidnioulz]

My org is also in a situation where we want more granularity than private or public. We have:

• Primary tokens that are exposed first in our documentation, autocompletion, etc and exported to code platforms
• Secondary tokens for edge cases (esp. state management) that are hidden by default in the doc to help newcomers focus on the core tokens
• Occasionally, tokens we create only as fixes to rare edge cases within the DS code library (and intended to be temporary)
• Internal tokens that are used as value refs (e.g. raw color tokens) and we don't want to expose to consumers or even define in exported files

I'm not a fan of wording like private because it's got very specific semantics in OOP, that differ from the lay understanding of designers. A designer may expect a private token is used inside the DS component library but not exposed to consumers. Another designer may expect it to not be exported. If we map private tokens to a private property in Swift color classes (the OOP meaning), neither mental model is correct.

Besides, whether or not a (raw) token is exported is very easy to manage at the moment. One only has to filter their tokens prior to exporting them in Style Dictionary or similar tools. So building a rigid semantic into the standard just to spare ourselves a one-liner filter feels like over-engineering.

So I'm not sure we would benefit from a semantic enforced directly in the standard unless it is extensible or flexible. One way this could be achieved is, instead of using words with a fixed meaning, allowing a visibility property with a numerical value, so we can define orders of visibility. All tool makers can easily handle order relations defined by numbers, without having to attribute meaning. And tools like SD or Cobalt can define default semantics that can easily be overridden.

An order relationship would support many cases and allow extending them, e.g.:

• 1: public tokens
• 2: private or internal tokens

Or:

• 1: primary/core public tokens
• 2: secondary/extra public tokens
• 3: DS-only exported tokens
• 4: tokens exported to code but not usable directly (their name's only visible in debugging tools as they are used as value refs)
• 5: tokens that aren't exported at all (their value is always resolved when they're referenced)
• 6: forbidden tokens whose value must not be referenced at all, which might help with error reporting when deprecating tokens in a large distributed system

And anyone would be free to add however many extra layers of visibility they need and to configure their export tools to map the order relation onto concepts supported by the tools.

[nesquarx]

I believe a simple $meta array variable which can classify tokens for categorizing, processing and extraction purposes would suffice any needs.

For example $meta: ["private", "internal", "debug", "lightThemeOnly"];

Or whatever combination, and you can call upon tokens using another combination of meta keywords, lets the processing define its own paradigms than sticking to specific cases.

On Fri, Jan 19, 2024, 6:25 AM Steve Dodier-Lazaro @.***> wrote:

My org is also in a situation where we want more granularity than private or public. We have:

• Primary tokens that are exposed first in our documentation, autocompletion, etc and exported to code platforms
• Secondary tokens for edge cases (esp. state management) that are hidden by default in the doc to help newcomers focus on the core tokens
• Occasionally, tokens we create only as fixes to rare edge cases within the DS code library (and intended to be temporary)
• Internal tokens that are used as value refs (e.g. raw color tokens) and we don't want to expose to consumers or even define in exported files

I'm not a fan of wording like private because it's got very specific semantics in OOP, that differ from the lay understanding of designers. A designer may expect a private token is used inside the DS component library but not exposed to consumers. Another designer may expect it to not be exported. If we map private tokens to a private property in Swift color classes (the OOP meaning), neither mental model is correct.

Besides, whether or not a (raw) token is exported is very easy to manage at the moment. One only has to filter their tokens prior to exporting them in Style Dictionary or similar tools. So building a rigid semantic into the standard just to spare ourselves a one-liner filter feels like over-engineering.

So I'm not sure we would benefit from a semantic enforced directly in the standard unless it is extensible or flexible. One way this could be achieved is, instead of using words with a fixed meaning, allowing a visibility property with a numerical value, so we can define orders of visibility. All tool makers can easily handle order relations defined by numbers, without having to attribute meaning. And tools like SD or Cobalt can define default semantics that can easily be overridden.

An order relationship would support many cases and allow extending them, e.g.:

• 1: public tokens
• 2: private or internal tokens

Or:

• 1: primary/core public tokens
• 2: secondary/extra public tokens
• 3: DS-only exported tokens
• 4: tokens exported to code but not usable directly (their name's only visible in debugging tools as they are used as value refs)
• 5: tokens that aren't exported at all (their value is always resolved when they're referenced)
• 6: forbidden tokens whose value must not be referenced at all, which might help with error reporting when deprecating tokens in a large distributed system

And anyone would be free to add however many extra layers of visibility they need and to configure their export tools to map the order relation onto concepts supported by the tools.

— Reply to this email directly, view it on GitHub https://github.com/design-tokens/community-group/issues/110#issuecomment-1900324107, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEKS36F2Y3TUACOQA2FUWELYPJQ2BAVCNFSM5NQD7GS2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJQGAZTENBRGA3Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Sidnioulz]

@nesquarx at the end of the day, yes we'll have access to arbitrary attributes with no meaning attached. I'm ok with going down that path and handling the meaning-making of $private myself, but I believe this ticket is about attributing meaning in the spec :)

[nesquarx]

Agreed, if the most common cases call for $private, having it makes sense, but trying to increase the scope there instead of an additional generic solution might become exhausting.

On Mon, Jan 22, 2024, 1:29 AM Steve Dodier-Lazaro @.***> wrote:

@nesquarx https://github.com/nesquarx at the end of the day, yes we'll have access to arbitrary attributes with no meaning attached. I'm ok with going down that path and handling the meaning-making of $private myself, but I believe this ticket is about attributing meaning in the spec :)

— Reply to this email directly, view it on GitHub https://github.com/design-tokens/community-group/issues/110#issuecomment-1903405091, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEKS36BGNH5BUNO54HAK4SDYPYIOBAVCNFSM5NQD7GS2U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJQGM2DANJQHEYQ . You are receiving this because you were mentioned.Message ID: @.***>