localmathalphabets handling has an issue with \mathchoice

[FrankMittelbach]

Brief outline of the bug

If we hit the limit of allocated math alphabets inside \mathchoice the code issues and \aftergroup, but that ends up between the arguments of \mathchoice and then TeX complains. One could really argue that this is a design bug in TeX, because it means that you can't use \aftergroup in a choice situation. But it is as it is and I doubt that Don will acknowledge this as a bug (or even if, will declare it as a feature :-))

Minimal example showing the bug

\RequirePackage{latexbug}       % <--should be always the first line (see CONTRIBUTING)!
\documentclass{article}

\setcounter{localmathalphabets}{5}

\DeclareSymbolFont{fooi}{OT1}{cmr}{m}{n}
\DeclareSymbolFont{fooii}{OT1}{cmr}{m}{n}
\DeclareSymbolFont{fooiii}{OT1}{cmr}{m}{n}
\DeclareSymbolFont{fooiv}{OT1}{cmr}{m}{n}
\DeclareSymbolFont{foov}{OT1}{cmr}{m}{n}
\DeclareSymbolFont{foovi}{OT1}{cmr}{m}{n}
\DeclareSymbolFont{foovii}{OT1}{cmr}{m}{n}
\DeclareSymbolFont{fooviii}{OT1}{cmr}{m}{n}
\DeclareSymbolFont{fooix}{OT1}{cmr}{m}{n}
\DeclareSymbolFont{foox}{OT1}{cmr}{m}{n}  % only one slot open   

\begin{document}

\section{Test}

$\mathcal{A}$ 

$\mathchoice{1}{\mathtt{B}}{3}{4}$   % this dies now ...

$\mathbf{C}$  

\end{document}

Plan for attack ...

Something along these lines maybe ...

\makeatletter

\newif\if@in@mathchoice

\let\@@mathchoice\mathchoice
\protected\def\mathchoice#1#2#3#4{\@in@mathchoicetrue
  \@@mathchoice{#1}{#2}{#3}{#4}\maybe@freeze\@in@mathchoicefalse}


\let\maybe@freeze\@empty

\def\document@select@group#1#2#3#4{%
 \ifx\math@bgroup\bgroup\else\relax\expandafter\@firstofone\fi
 {%
 \ifmmode
   \ifnum\csname c@mv@\math@version\endcsname<\e@mathgroup@top
     \ifnum \numexpr\e@mathgroup@top-\c@localmathalphabets
              >\csname c@mv@\math@version\endcsname
     \else
       \ifcsname mv@\math@version @frozen\endcsname \else
           \expandafter\freeze@math@version\expandafter{\math@version}%
       \fi
     \fi
     \begingroup
       \escapechar\m@ne
       \getanddefine@fonts{\csname c@mv@\math@version\endcsname}#3%
       \globaldefs\@ne  \math@fonts
     \endgroup
     \expandafter\extract@alph@from@version
         \csname mv@\math@version\expandafter\endcsname
         \expandafter{\number\csname
                       c@mv@\math@version\endcsname}%
          #1%
     \global\advance\csname c@mv@\math@version\endcsname\@ne
   \else
     \let#1\relax
     \@latex@error{Too many math alphabets used in
                   version \math@version}%
        \@eha
  \fi
 \else \expandafter\expandafter\expandafter\non@alpherr\fi
 \expandafter#1\ifx\math@bgroup\bgroup{#4}\else#4\fi
 }%
}

\ExplSyntaxOn
\cs_set_protected:Npn\freeze@math@version #1 {
  \@font@info{Freeze~ math~ alphabet~ allocation~ in~ version~
              #1.\MessageBreak
              Allocated~math~groups:~\int_use:c{ c@mv@ #1 }~
              (local:~  \int_use:N\c@localmathalphabets)      }
  \cs_gset_eq:cc { mv@#1@frozen }{ mv@#1 }
  \tl_gset:cx { g__nfss_frozen_mv_ #1 _tl }{ \int_use:c { c@mv@#1 } }
\if@in@mathchoice
%  
  \gdef\maybe@freeze{
       \group_insert_after:N \__nfss_init_mv_freeze:N
       \exp_after:wN \group_insert_after:N \cs:w mv@#1@reset \cs_end:
       \gdef\maybe@freeze{}
  }
\else
    \group_insert_after:N \__nfss_init_mv_freeze:N
    \exp_after:wN \group_insert_after:N \cs:w mv@#1@reset \cs_end:
\fi
  \tl_gput_right:No \check@mathfonts
       {
         \exp_after:wN \group_insert_after:N \cs:w mv@#1@reset \cs_end:
       }
  \cs_gset:cpn{mv@#1@reset}
     {
         \int_compare:nNnTF { \int_use:c{c@mv@#1} } >
                            { \tl_use:c{g__nfss_frozen_mv_ #1 _tl} }
             {
               \@font@info{Undo~ math~ alphabet~ allocation~ in~ version~ #1}
               \cs_gset_eq:cc { mv@#1 }{ mv@#1@frozen }
               \int_gset:cn { c@mv@#1 }{ \tl_use:c {g__nfss_frozen_mv_ #1 _tl} }
               \group_begin:
                 \cs_set_eq:NN \getanddefine@fonts \use_none:nn
                 \use:c {mv@#1}
               \group_end:
             }
             {
               \@font@info{No~ math~ alphabet~ change~ to~ frozen~ version~ #1}
             }
         \if@ignore \ignorespaces \fi
    }
}
\ExplSyntaxOff

\makeatother

but that needs some further checking and obviously some cleanup

[davidcarlisle]

won't

\protected\def\mathchoice#1#2#3#4{\@in@mathchoicetrue
  \@@mathchoice{#1}{#2}{#3}{#4}\maybe@freeze\@in@mathchoicefalse}

mean @in@mathchoice is false at \mathtt ?

\mathchoice{\mathchoice{a}{b}{c}{d}... \mathtt{1}}
                    {2}{3}{4}

I guess we should at least document this doesn't work

$\mathchoice \bgroup a\egroup\bgroup b\egroup\bgroup c\egroup\bgroup d\egroup$

[FrankMittelbach]

yeah, bad definition, better would be

\protected\def\mathchoice#1#2#3#4{\begingroup\@in@mathchoicetrue
  \@@mathchoice{#1}{#2}{#3}{#4}\maybe@freeze\endgroup}

I guess. The version with implicit args would be hard to catch unless you really go for slow parsing. However, that is nowhere ever been used so I guess just documenting it would be enough.