MSVC: fix nan/inf issues on msvc and gcc

[echoix]

Replaces #355

I cherry-picked and manually adjusted the remaining of #355 to be updated, mergeable, and reviewable. I don't necessarily endorse or think everything is needed, but allows to close #355 for the time being. Please look if these still make sense, considering the changes done in #2681. Do they apply correctly to MSVC?

[echoix]

I've reverted in a separate commit to use the initialized value that is now used. When the original commit was made, it was initialized with nan (0.0/0.0). I wanted to show what the commit made when cherry-picking, so there would be a trace,