Add sage.misc.latex.pdf to save the image of objects to pdf

[kwankyu]

trafficstars

Hence

sage: from sage.misc.latex import pdf
sage: pdf(pi, 'test.pdf')

works. This is a companion to the existing sage.misc.latex.png, which saves png image of objects. Of course, pdf image is scalable and hence more suitable for inclusion into latex documents.

Also we fix it that view() sporadically fails on mac because the temporary file is removed before the viewer opens the file. For example,

sage: view(pi)  # show pi

Thus after this PR, the temporary file is removed when the viewer (Preview app on mac, not the window showing the image) closes. This behavior is consistent with that on linux.

:memo: Checklist

• [x] The title is concise and informative.
• [x] The description explains in detail what this PR is about.
• [x] I have linked a relevant issue or discussion.
• [ ] I have created tests covering the changes.
• [ ] I have updated the documentation and checked the documentation preview.

:hourglass: Dependencies

[github-actions[bot]]

Documentation preview for this PR (built with commit 841298f9ae41c01c5d012c7ac48464e2b47654de; changes) is ready! :tada: This preview will update shortly after each push to this PR.

[orlitzky]

sleep(5) will fail randomly, why replace the explicit synchronization?

[orlitzky]

The report at https://groups.google.com/g/sage-support/c/YfyaZxc8Z6k has nothing to do with the viewer opening too fast. LaTeX failed to build the PDF in the first place.

[kwankyu]

The report at https://groups.google.com/g/sage-support/c/YfyaZxc8Z6k has nothing to do with the viewer opening too fast. LaTeX failed to build the PDF in the first place.

No. The PDF was built. This code

    # Return immediately but only clean up the temporary file after
    # the viewer has closed. This function is synchronous and waits
    # for the process to complete...
    def run_viewer():
        run([viewer, output_file], capture_output=True)
        tmp.cleanup()    

does not work as the comment promised. This is what happens: The viewer "opens" (that is, the operating system successfully launched the viewer). Then tmp.cleanup() cleans up the temporary PDF file, before the viewer opens up the PDF file!. The viewer says "no pdf file".

We have to give the viewer enough time to really open the PDF file after it launched. There is no way to detect when the viewer has opened the file. 5 seconds seems enough.

[orlitzky]

The error ! ==> Fatal error occurred, no output PDF file produced! comes from LaTeX, not sage. You can run pdflatex -halt-on-error on a file containing junk and that's what it will display.

[kwankyu]

I don't know the reporter's system. He says "Latex does work though". So I guess that the message is from the viewer.

[kwankyu]

By the way, I also experience this "no pdf file" symptom on my system (mac). Why I prepared this PR :-)

[orlitzky]

He tells you his system. Like I said, you can run pdflatex yourself and see exactly where the error comes from. You're trying to fix something that isn't broken.

There may be a problem on mac, but I can't test. You should probably figure out why the current approach doesn't work. Or just add a special case for mac if you don't feel like it. But don't make the situation worse for everyone else because nothing is broken on linux.

[kwankyu]

Like I said, you can run pdflatex yourself and see exactly where the error comes from. You're trying to fix something that isn't broken.

I see what you mean. I cannot be sure on the reporter's case.

There may be a problem on mac, but I can't test. You should probably figure out why the current approach doesn't work.

For me, the symptom is sporadic. If the viewer opens up the file quickly enough, no problem. Sometimes not. Maybe dependent on the pdf file size...

Or just add a special case for mac if you don't feel like it.

From the reporter's case, I guessed the symptom can also happen on linux, perhaps depending on the latex system or pdf viewer.

But don't make the situation worse for everyone else because nothing is broken on linux.

OK. No problem. I can wait for input from other people.

[dimpase]

how can this happen at all? view is one of many commands using a temporary file, but the typical issue is that these files are left uncleaned

[dimpase]

For me, the symptom is sporadic. If the viewer opens up the file quickly enough, no problem. Sometimes not. Maybe dependent on the pdf file size...

Do you by any chance have an antivirus or some other spyware^H^H^^ thing like this installed on your mac? Often that's the case with employer-issued machines...

[kwankyu]

No.

I explained how it happens above: https://github.com/sagemath/sage/pull/38339#issuecomment-2267958420

There is nothing strange why it happens.

[dimpase]

No.

I explained how it happens above: #38339 (comment)

There is nothing strange why it happens.

Do you mean it's a bug in CPython on macOS? Could be, of course, in how threads are handled, but this is not something that should be fixed by adding dodgy extra waiting times, unconditionally on the platform.

Or, perhaps how run_viewer is launched:

    from threading import Thread
    t = Thread(target=run_viewer)
    t.daemon = True
    t.start()

does not work correctly on macOS?

[dimpase]

reading https://docs.python.org/3/library/threading.html does not convince me about the validity of the syncronisation via t.daemon used in Sage. It seems to be for something else.

[orlitzky]

The daemon flag wasn't for synchronization per se. Without it, sage would refuse to exit while the viewer was still open; sage was waiting for the view/cleanup thread to finish. That page says,

A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. The initial value is inherited from the creating thread. The flag can be set through the daemon property or the daemon constructor argument.

Adding it does allow sage to quit while the viewer is open.

[dimpase]

it seems to be the default behaviour of open on macOS is not to wait until the process ends. However, one can do open -W to let it wait until it completes.

    -W  Causes open to wait until the applications it opens (or that were already open) have exited.  Use with the -n flag to allow open to function as an appropriate app for the $EDITOR
         environment variable.

This way, run() would end only when the viewer completes, and so the files can be cleaned up safely.

[orlitzky]

it seems to be the default behaviour of open on macOS is not to wait until the process ends. However, one can do open -W to let it wait until it completes.

This is the way e.g. exo-open works on linux, too. But if adding -W on macos makes the problem go away that's still a great solution.

[kwankyu]

Adding -W works. That is, tmp.cleanup() is executed after Preview (the default pdf viewer on macos run by open) closes (this is closing the application, not just closing a window showing the pdf file).

On the other hand, If I end sage session without closing Preview, the thread seems gone and tmp.cleanup() is not executed. I am not sure if the temporary file was cleaned up in this case...

[kwankyu]

On the other hand, If I end sage session without closing Preview, the thread seems gone and tmp.cleanup() is not executed. I am not sure if the temporary file was cleaned up in this case...

What is the behavior on linux in this case? Is the temporary file cleaned up?

[dimpase]

the proper way to have temporary files cleaned up is to get them registered with sage-cleaner using atexit, see sage.cpython.atexit.

@orlitzky knows all about it :-)

[orlitzky]

the proper way to have temporary files cleaned up is to get them registered with sage-cleaner using atexit, see sage.cpython.atexit.

@orlitzky knows all about it :-)

By itself atexit has two downsides,

1. It doesn't clean up if sage crashes
2. It can leave unused temporary files there for a long time, filling up an in-memory /tmp over the course of a few days

The first may be rare but people ocassionally complain about the second. The current approach was supposed to avoid those two issues without introducing any new ones (relative to using atexit). If the temporary file isn't removed after closing sage and the viewer, that's a new one to me.

Are there any PDF viewers that will "update" the preview when the file is removed?

If not, and if the removal issue is also happening on linux, maybe belt-and-suspenders is called for: remove the temp dir when the viewer closes, and also atexit.

I'm getting ready for a conference and don't have time to mess around with this but it sounds like adding -W on macos is uncontroversial and then we can iterate to improve whatever issues remain.

[kwankyu]

By itself atexit has two downsides,

1. It doesn't clean up if sage crashes
2. It can leave unused temporary files there for a long time, filling up an in-memory /tmp over the course of a few days

My original solution (removing the temporary file after some seconds) avoids these problems, at least on mac :-) Preview does not complain that the file is gone (as it is on memory?)

[dimpase]

Do all these viewers really load the whole pdf file into RAM - or we are just lucky to have examples fit into an i/o buffer?

The only safe solution it to wait until the viewing is done.

[orlitzky]

By itself atexit has two downsides,

1. It doesn't clean up if sage crashes
2. It can leave unused temporary files there for a long time, filling up an in-memory /tmp over the course of a few days

My original solution (removing the temporary file after some seconds) avoids these problems, at least on mac :-) Preview does not complain that the file is gone (as it is on memory?)

It doesn't solve any problems relative to using -W, but it adds several new ones. Without the daemon flag, it will prevent sage from quitting. With the daemon flag, if sage quits, the temporary file (probably) still will not be removed. On top of that, it will fail randomly on slow/busy machines.

[orlitzky]

Do all these viewers really load the whole pdf file into RAM - or we are just lucky to have examples fit into an i/o buffer?

So long as the file is open, deleting the path from the filesystem won't really remove it, so in that sense it shouldn't matter. But many PDF readers monitor the file for changes, and for one example, the current version of Zathura doesn't handle it well if you rename the open file and then move it back. Deleting the file may confuse some other reader. It seems like a bad idea to guess how a PDF viewer will be implemented.

The only safe solution it to wait until the viewing is done.

If at all possible, I agree. This goal may ultimately conflict with having sage delete the file when it quits, but we're talking corner cases at that point.

[kwankyu]

My original solution (removing the temporary file after some seconds) avoids these problems, at least on mac :-) Preview does not complain that the file is gone (as it is on memory?)

It doesn't solve any problems relative to using -W, but it adds several new ones. Without the daemon flag, it will prevent sage from quitting. With the daemon flag, if sage quits, the temporary file (probably) still will not be removed. On top of that, it will fail randomly on slow/busy machines.

The behavior is not such on mac. As I said above, the temporary file is removed after it is loaded into memory by Preview. Sage can quit normally.

But I agree that removing the temporary file while viewing the file is not good.

[kwankyu]

Now I am using Dima's solution open -W. It is working well on my side.

[kwankyu]

Anything else?

The problem of not cleaning up the temporary file if sage session ends while the viewer is open persists, before and after the PR. This PR fixes only the problem on mac that the viewer sporadically crashes because the temporary file is cleaned up too soon.

[kwankyu]

and of course, we get a new pdf(obj) command.

[dimpase]

Anything else?

The problem of not cleaning up the temporary file if sage session ends while the viewer is open persists, before and after the PR. This PR fixes only the problem on mac that the viewer sporadically crashes because the temporary file is cleaned up too soon.

see https://github.com/sagemath/sage/pull/38339#issuecomment-2269250771