Great post! I agree with most of what's said here, as controversial as some of it may be, because it is a paradigm shift for sure.
I've actually been planning to address this partially, in Style Dictionary (I'm the current most active maintainer of it). Using a Map() / flat structure makes a lot more sense for this library internally, it's cheaper to iterate and to do lookup for resolving references and the majority of output formats are flat, so in v4 we will make a lot of (non-breaking) improvements there, and potentially in v5 the Map() flat structure will be standard, with converting to a nested taxonomical object being an opt-in via a utility (this is a breaking change), e.g. if your output format is a nested one like DTCG JSON format.
That by itself does not address token breaking changes, so I am also very open to seeing if we can start supporting UIDs for tokens and resolving references that way, as well as allowing our "name transforms" (the functions that convert the taxonomy to a token name) to start using perhaps a $taxonomy property instead of the token's nested object taxonomy path (aka its ancestor properties chain), as an opt-in. For me to start work on that in Style Dictionary, as a prerequisite I think we should also start the discussion on the DTCG GitHub repository about potentially using system-generated $id props and also use it for reference resolution, and get some feedback/consensus there as well.
In extension of that, I honestly don't really see a reason why the DTCG format couldn't be a flat one as well and introducing something like a $taxonomy prop to store taxonomy metadata about a token for grouping/categorization purposes. The only con I see is the authoring experience in raw JSON in a code editor, where nested objects make it easy to group, collapse/expand. However, the future imo is token management tools which will address the UX of authoring design tokens, so I think this will be an increasingly moot argument as time goes on.
If you want to collaborate / spar about this topic, my DMs are open, either on X @jorenbroekema or on the Tokens Studio Slack (link on our website).
