titletoc: `\contentsmargin` with font-dependent units causes misalisngment

[sim642]

The documentation for \contentsmargin does say

Although standard classes use font dependent units (mainly em), it is recommended using absolute ones (pc, pt, etc.) to ensure they entries are aligned correctly.

But that's a bit of an understatement because it really doesn't work at all when font size is changed for some \titlecontents, e.g. when trying to replicate standard \part look.

What's particularly confusing is that, when \contentsmargin isn't used, but the standard \@pnumwidth and \@tocrmarg are used to tune the same things, em-s are just fine, even with titletoc. From what I understand, this is because titletoc uses those by default, but redefines them to always be in pt-s (by going through tempskipa) during initialization: https://github.com/jbezos/titlesec/blob/e5b48d9f273c5d4db720804a5f706998a2b642db/titletoc.sty#L581-L584

It would be nice if font-dependent units still worked for \contentsmargin even when \titlecontents changes font size for a particular kind. It looks like currently \pnumwidth is used a few times, but there may be a font size command between, causing misalignment.

As a workaround, instead of changing font size in the <above-code> part of \titlecontents, duplicating it in <numbered-entry-format>, <numberless-entry-format> and <filler-page-format> (in the optional argument for \contentspage) avoids the problem. Although I didn't check if that maybe has some other unintended effects.