CLN: Simplify map_infer_mask

[lithomas1]

• [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
• [ ] Tests added and passed if fixing a bug or adding a new feature
• [ ] All code checks passed.
• [ ] Added type annotations to new arguments/methods/functions.
• [ ] Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

This is needed for upcoming numpy string dtype support (getting rid of the ndarray[object] annotation).

We might also get a decent speedup from using numpy C API functions (but probably not).

EDIT: Looks like benchmarks aren't significantly changed.