Various fixes, with a view toward the vscode extension

[pzinn]

This branch has two purposes:

• allow any user to run Macaulay2 in Macaulay2Web in local mode (npm start local) or more importantly to use Macaulay2 in the newly developed vscode extension, with the same quality of rendering as with the forked branch (vanilla) used on the official Macaulay2Web server.
• in the process, I've found quite a few commits that fell through the cracks, so I'm putting them here in the hope to have the vanilla branch and the main M2 branch converge in the near future.

This is very much work in progress.

[pzinn]

this gtest stuff is driving me insane. it seems each time I have a new commit, it silently changes gtest and it's not clear why / how to know about it.

[pzinn]

OK this might be enough for a PR.

• updated webapp.m2: lots of changes under the hood to improve M2Web/vscode extension.
• small changes to some output routines.
• hypertext FilePosition now produces a link, as discussed on zulip.
• changes to tex in webapp mode: M2 will now try harder to use direct html output rather than katex, so texMath is hacked to do so.
• an annoying issue is that M2 outputs with both stdout and stderr. That's not great when M2 is spawned as a sub-process and the output is forwarded, as is the case for M2Web and vscode extension, because the outputs of stdout and stderr are typically no longer in sync and error messages appear in the wrong place. There are two solutions: -- use a shell and redirect (2&>1 or whatnot). this is fine but creates additional technical issues that I was unable to resolve for the vscode extension (SIGINT is no longer forwarded to M2). -- not use stderr! so in the current PR, as a possibly temporary measure, stderr is redefined as stdout in webapp mode.
• ~a small unrelated commit fixing a problem with promotion of fraction fields, cf the discussion in #3184 and the added test.~

[mikestillman]

A few notes about this.

• It compiled fine for me
• Using it in the vscode extension:
  • After e.g. help matrix, the example code is made into a series of buttons. That would be great, except it contains the prompt. the input, and also the output. So one gets errors when pressing such a button. (Maybe this was the case earlier too?)
  • On the plus side, the errors one gets do come out in the correct location!
• The promote changes seem good (and desirable) to me. I like the idea mentioned in the issue, of deprecating liftable, changing to isLiftable too.

@pzinn Do you want to make any other changes for the moment, before this gets accepted?

[mikestillman]

@d-torrance Do you see any issues with the changes to use of stdin and stderr? It seems that since this is done only for webapp mode, it should not be a problem?

[pzinn]

thanks for testing @mikestillman. Yeah it's not perfect yet (in particular when it comes to help and error messages), but it's an improvement over the current situation. In particular one thing which should work fine now is the links to code in say

code methods ideal

You're right that code in examples is not properly parsed -- unfortunately this is the way examples are currently produced, as a block rather than with separate input/output. On my vanilla branch, help examples are produced in webapp mode, which automatically inserts all the character codes that allow to distinguish say input from output.

I would say this PR is enough as it is. Unless there's a problem with it, it's ready to be merged. I'll definitely work on a second one, but it will take some time because any further improvements require significantly more work.

[pzinn]

I've moved the promote stuff to another PR. this is ready to be merged.

[pzinn]

bump!

[pzinn]

Did you mean to update the googletest submodule?

no.

[pzinn]

I somehow made it worse. now one of my commits has updated editors and various submodules.

[d-torrance]

It looks like most of the issues cropped up in the "update hypertext, FilePosition" commit. You could do an interactive rebase and edit that commit. Then when you're at that commit, something like (assuming your development branch is up to date):

git checkout development -- M2/Macaulay2/editors
git checkout development -- M2/submodules
git commit --no-edit

[pzinn]

neither of the first two lines do anything. I don't understand this submodule mess.

[pzinn]

is there something wrong with my configuration? here's my .git/config

[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = [email protected]:pzinn/M2.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[submodule "M2/submodules/memtailor"]
	active = true
	url = https://github.com/Macaulay2/memtailor
[submodule "M2/submodules/mathic"]
	active = true
	url = https://github.com/Macaulay2/mathic
[submodule "M2/submodules/mathicgb"]
	active = true
	url = https://github.com/Macaulay2/mathicgb
[submodule "M2/submodules/fflas-ffpack"]
	active = true
	url = https://github.com/linbox-team/fflas-ffpack.git
[remote "upstream"]
	url = https://github.com/Macaulay2/M2.git
	fetch = +refs/heads/*:refs/remotes/upstream/*
[submodule "M2/submodules/bdwgc"]
	active = true
	url = https://github.com/Macaulay2/bdwgc.git
[submodule "M2/submodules/flint2"]
	active = true
	url = https://github.com/Macaulay2/flint2.git
[submodule "google-googletest"]
	active = true
	url = https://github.com/google/googletest.git
[submodule "M2/submodules/libatomic_ops"]
	active = true
	url = https://github.com/Macaulay2/libatomic_ops.git
[submodule "M2/submodules/mpir"]
	active = true
	url = https://github.com/Macaulay2/mpir.git
[submodule "M2/submodules/givaro"]
	active = true
	url = https://github.com/Macaulay2/givaro.git
[submodule "M2/Macaulay2/editors/emacs"]
	active = true
	url = https://github.com/Macaulay2/M2-emacs.git
[submodule "M2/submodules/frobby"]
	active = true
	url = https://github.com/Macaulay2/frobby.git
[branch "main"]
	remote = origin
	merge = refs/heads/main
[branch "experimental"]
	remote = origin
	merge = refs/heads/experimental
[branch "numeric"]
	remote = origin
	merge = refs/heads/numeric
[branch "vanilla"]
	remote = origin
	merge = refs/heads/vanilla
[branch "locate"]
	remote = origin
	merge = refs/heads/locate
[branch "virtualtally"]
	remote = origin
	merge = refs/heads/virtualtally
[pull]
	rebase = false
[branch "zeroonzero"]
	remote = origin
	merge = refs/heads/zeroonzero
[branch "multibetti"]
	remote = origin
	merge = refs/heads/multibetti
[submodule "M2/submodules/flint"]
	active = true
	url = https://github.com/Macaulay2/flint2.git
[remote "tor"]
	url = https://github.com/d-torrance/M2.git
	fetch = +refs/heads/*:refs/remotes/tor/*
[remote "mahrud"]
	url = https://github.com/mahrud/M2.git
	fetch = +refs/heads/*:refs/remotes/mahrud/*
[branch "promote"]
	remote = origin
	merge = refs/heads/promote
[branch "keyword"]
	remote = origin
	merge = refs/heads/keyword
[branch "factorWA"]
	remote = origin
	merge = refs/heads/factorWA
[branch "vscode"]
	remote = origin
	merge = refs/heads/vscode

and my .gitmodules

[submodule "M2/submodules/memtailor"]
	path = M2/submodules/memtailor
	url = https://github.com/Macaulay2/memtailor
	ignore = all
[submodule "M2/submodules/mathic"]
	path = M2/submodules/mathic
	url = https://github.com/Macaulay2/mathic
	ignore = all
[submodule "M2/submodules/mathicgb"]
	path = M2/submodules/mathicgb
	url = https://github.com/Macaulay2/mathicgb
	ignore = all
[submodule "M2/submodules/bdwgc"]
	path = M2/submodules/bdwgc
	url = https://github.com/Macaulay2/bdwgc.git
	ignore = all
[submodule "M2/submodules/flint"]
	path = M2/submodules/flint
	url = https://github.com/flintlib/flint
	ignore = all
[submodule "M2/submodules/frobby"]
	path = M2/submodules/frobby
	url = https://github.com/Macaulay2/frobby.git
	ignore = all
[submodule "M2/submodules/fflas-ffpack"]
	path = M2/submodules/fflas_ffpack
	url = https://github.com/Macaulay2/fflas-ffpack.git
	ignore = all
[submodule "M2/submodules/givaro"]
	path = M2/submodules/givaro
	url = https://github.com/linbox-team/givaro
	ignore = all
[submodule "M2/submodules/googletest"]
	path = M2/submodules/googletest
	url = https://github.com/google/googletest.git
	ignore = all
[submodule "M2/Macaulay2/editors/emacs"]
	path = M2/Macaulay2/editors/emacs
	url = https://github.com/Macaulay2/M2-emacs.git
	ignore = all

[pzinn]

what I ended up doing, for reference: instead of resetting the submodules to the current development branches (which has changes compared to when I started this PR) I ended up resetting them to the latest commit pre-PR. Even that didn't quite work (it somehow didn't delete the Readme.md file in the editors/, which I had to delete manually). submodules are !@$%!

[d-torrance]

I think the ignore = all in .gitmodules was likely at least partially to blame. We tried that for a while in development, but it was removed after a while because it was causing more trouble than it was worth. Maybe your branch branched off when it was in there