[5.0.0-rc.1] Seems new implementation of expand make tokens clone in Dictionary structure

[Delagen]

updating any token in tokenMap don't reflect the same token in allTokens because of structureclone

[jorenbroekema]

Could you add a bit more information about how you're trying to update tokens in the tokenMap?

A workaround for now would be:

import StyleDictionary from 'style-dictionary';
import { convertTokenData } from 'style-dictionary/utils';

const sd = new StyleDictionary();
await sd.hasInitialized;

// mutate the Map
sd.tokenMap.set('{foo.bar}', tokenDefinition);

// Sync to the other 2 data structures
sd.allTokens = convertTokenData(tokenMap, { output: 'array' });
sd.tokens = convertTokenData(tokenMap, { output: 'object' });

Usually we don't really recommend changing tokens outside of the supported Style Dictionary hooks, e.g. preprocessors or transforms. But you certainly can, just have to sync to the other 2 data structures because v5 still uses the object/array data structure in all of the built-in formats. For transforms, the Map is used. So in order for everything to work, when you mutate one structure, you have to update the other 2. When using the Style Dictionary hooks, any kind of syncing should be already handled for you, if not, then this would be a bug and we should fix that, but I need more info then.

In a future v6 we're likely going to only be using the Map data structure internally, and users will have to convert it on their end to an array or object on demand, but this is a bigger breaking change so we're not including this in v5.

[Delagen]

@jorenbroekema

In v4 modifying any token in Dictionary was synced in other accessors. convertTokenData clones tokens so it works as it was in v4.

But initial Dictionary returned by library now contains copies of tokens in different structures, in v4 it was links to the same objects.

So in v4

dictionary.tokens[0].value = 1;
// get the same token from tokenMap
console.log(dictionary.tokenMap.get('....').value) // outputs 1

In v5 console.log outputs undefined or original value

[jorenbroekema]

I'm open to making it so that the tokenMap is considered the "source of truth", and that calling tokenMap.set manually will update the other 2 structures for you for that item (token). Similarly, I could also make it so that setting sd.tokenMap also goes through a getter/setter pair to update the tokens / allTokens as a whole.

I think it's a bit much to make it sync bi-directional for all 3 structures, even though that may have been the behavior in v4, that behavior wasn't necessarily spec'd as being public API so to speak. That said, I think we should add a note to the v5 changelog that this is a pseudo breaking change (not spec'd, but you may rely on it nevertheless, and then it's breaking for you).

I'm considering deprecating sd.tokens / sd.allTokens right now already to prep for removal in v6, where you'd only get the tokenMap and have to convert it yourself if you need in the other data structures. This would help performance for large tokensets a lot. Theforefore, I'm not sure if fixing this right now has a high priority, especially given that it's pretty easy to work around it and that mutating tokens manually like that isn't a documented or recommended practice.