Error on PYTHON 3.5 / Windows when testing example

[sruizr]

There is no output because it has problems with the temporary directory

subprocess.CalledProcessError: Command '['latexmk', '-pdf', '-pdflatex=pdflatex -interaction=batchmode -halt-on-error -no-shell-escape -file-line-error %O %S', 'C:\\Users\\sruiz\\AppData\\Local\\Temp\\tmpfroqoypt\\tmpt171wtiv.latex']' returned non-zero exit status 12

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\data\__init__.py", line 252, in temp_saved
    yield tmp
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\latex\build.py", line 98, in build_pdf
    raise_from(LatexBuildError(base_fn + '.log'), e)
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\future\utils\__init__.py", line 399, in raise_from
    exec(execstr, myglobals, mylocals)
  File "<string>", line 1, in <module>
latex.exc.LatexBuildError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\data\__init__.py", line 255, in temp_saved
    os.unlink(tmp.name)
PermissionError: [WinError 32] El proceso no tiene acceso al archivo porque est▒ siendo utilizado por otro proceso: 'C:\\Users\\sruiz\\AppData\\Local\\Temp\\tmpfroqoypt\\tmpt171wtiv.latex'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\latex\build.py", line 100, in build_pdf
    return I(open(output_fn, 'rb'), encoding=None)
  File "c:\python35\Lib\contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\data\__init__.py", line 258, in temp_saved
    reraise(e)
TypeError: reraise() missing 1 required positional argument: 'value'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tmp.py", line 10, in <module>
    pdf = build_pdf(min_latex)
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\latex\build.py", line 203, in build_pdf
    return builder.build_pdf(source, texinputs)
  File "<decorator-gen-1>", line 2, in build_pdf
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\data\decorators.py", line 82, in _
    return f(*bvals.args, **bvals.kwargs)
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\latex\build.py", line 100, in build_pdf
    return I(open(output_fn, 'rb'), encoding=None)
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\tempdir.py", line 25, in __exit__
    return self.dissolve()
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\tempdir.py", line 30, in dissolve
    shutil.rmtree(self.name)
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\shutil.py", line 488, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\shutil.py", line 383, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\shutil.py", line 381, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] El proceso no tiene acceso al archivo porque est▒ siendo utilizado por otro proceso: 'C:\\Users\\sruiz\\AppData\\Local\\Temp\\tmpfroqoypt\\tmpt171wtiv.latex'
Exception ignored in: <bound method TempDir.__del__ of <tempdir.TempDir object at 0x0000000000B4C9E8>>
Traceback (most recent call last):
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\tempdir.py", line 19, in __del__
    self.__exit__(None, None, None)
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\tempdir.py", line 25, in __exit__
    return self.dissolve()
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\site-packages\tempdir.py", line 30, in dissolve
    shutil.rmtree(self.name)
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\shutil.py", line 488, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\shutil.py", line 383, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\sruiz\Projects\ManageQualityBetter\env\lib\shutil.py", line 381, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] El proceso no tiene acceso al archivo porque est▒ siendo utilizado por otro proceso: 'C:\\Users\\sruiz\\AppData\\Local\\Temp\\tmpfroqoypt\\tmpt171wtiv.latex'

[mbr]

I'll admit, Windows support is probably poor, because I don't use it at all at the moment. Also, my spanish isn't that good either =).

Could you reduce this to a minimal test-case maybe?

[sruizr]

I'm working on both. But I like more Linux :-)

[sruizr]

Let me test changing the latex builder. Now I use MkTex. I think something blocks the archive .tex. Maybe subprocess or the own MkTex. I'm trying with TexLive on Windows

[sruizr]

Tested with TexLive and I get the same error Translating to english: PermissionError: [WinError 32] The process can not access to the file because is used by other process: 'C:\Users\sruiz\AppData\Local\Temp\tmpm9t8hy00\tmpahz6_eei.latex'.

It's clearly a Windows issue and subprocess or data package.

[sruizr]

Confirmed to get error in data package. debuging till error and it's on line 105:

return I(open(output_fn, 'rb'), encoding=None)

[mbr]

I think it's because Windows does not allow file to be reopened in the same way that *nix does. At least that would be in line with the error message.

As I said, I do not have a Windows box ready to go at this time. A guess would be that a temporary file is supposed to be read by mklatex/pdflatex, but is still open by the latex package. If that's the case, maybe reordering things slightly and adding a close(), as well as using the volatile package (https://github.com/mbr/volatile) might fix things on the data side and that may be enough.

However, this is all speculation for now, sorry =(

[Grimmick]

There are actually two problems here, he just didn't reach the second one, yet:

1. On Windows, you cannot delete a folder, when there are still open handles pointing into it. In the line

   return I(open(output_fn, 'rb'), encoding=None)
   

   you open a file and with the return you exit the context of the TempDir object causing it to delete the directory. This will result in this error message on an English Windows system:

   WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'c:\users\\appdata\local\temp\tmp23gzgh\tmpmx9xnk.pdf'

   This can be fixed by using

   return I(open(output_fn, 'rb').read(), encoding=None)
   

   which is less memory efficient, of course.

2. On Windows, access to the file of tempfile.NamedTemporaryFile is exclusive: "Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later)" (https://docs.python.org/2/library/tempfile.html). But with

   source.temp_saved(suffix='.latex', dir=tmpdir) as tmp
   

   you keep the temp file opened in temp_saved, thus pdflatex / latexmk cannot open the latex file. This can be fixed in data.init.py:Data.temp_saved by explicitly closing the file before yielding it:

       tmp.flush()
       tmp.close()
       yield tmp
   

With these two changes, I can build PDF files on Windows. (OK, I also added this, because I don't have perl installed:

    import latex.build

    latex.build.PREFERRED_BUILDERS = [
        latex.build.PdfLatexBuilder,
    ]

but that's another story)

[mbr]

Thanks for the investigation. I don't have a Windows machine at the moment and it has been years since I did anything but play games on one.

The fixes look okay and I don't think reading the full file into memory is a big issue. However, I cannot test things on Windows. Would you be willingt to make a pull request?

[Grimmick]

OK, done and works for me. I first wanted to address the issue with temp_saved in the data package, but it didn't work out well with the tests. So changing it directly in the latex package also avoids the interface semantic change. Maybe you want to leave a comment about this potential problem on Windows in the data package, though.

[mbr]

I've merged this for now, before the next release, I'll have to check if it still works on *nix.

[UnoYakshi]

The very same problem appeared on Arch Linux. It looks like a temporary directory is not created. If I remove subprocess.check_call() I meet FileNotFoundError for /tmp/tmp*/tmp*.aux.

I ran it from PyCharm for both pdflatex and latexmk, the results are the same (despite of different result-codes, 12 against 1).

pdflatex:

Traceback (most recent call last):
  File "/home/whs/Projects/ownDeploy/ownenv/lib/python3.6/site-packages/latex/build.py", line 174, in build_pdf
    stdout=open(os.devnull, 'w'), )
  File "/usr/lib64/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pdflatex', '-interaction=batchmode', '-halt-on-error', '-no-shell-escape', '-file-line-error', '/tmp/tmp7fa9i0d4/tmptdfavgkq.latex']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/whs/Projects/ownDeploy/own-agent/agents/science/report.py", line 35, in <module>
    pdf = builder.build_pdf(main_tex_template)
  File "<decorator-gen-2>", line 2, in build_pdf
  File "/home/whs/Projects/ownDeploy/ownenv/lib/python3.6/site-packages/data/decorators.py", line 82, in _
    return f(*bvals.args, **bvals.kwargs)
  File "/home/whs/Projects/ownDeploy/ownenv/lib/python3.6/site-packages/latex/build.py", line 176, in build_pdf
    raise_from(LatexBuildError(base_fn + '.log'), e)
  File "/home/whs/Projects/ownDeploy/ownenv/lib/python3.6/site-packages/future/utils/__init__.py", line 398, in raise_from
    exec(execstr, myglobals, mylocals)
  File "<string>", line 1, in <module>
latex.exc.LatexBuildError: None

latexmk:

Traceback (most recent call last):
  File "/home/whs/Projects/ownDeploy/ownenv/lib/python3.6/site-packages/latex/build.py", line 111, in build_pdf
    stderr=open(os.devnull, 'w'), )
  File "/usr/lib64/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['latexmk', '-pdf', '-pdflatex=pdflatex -interaction=batchmode -halt-on-error -no-shell-escape -file-line-error %O %S', '/tmp/tmpyhuwhiz9/tmpq7qje5m7.latex']' returned non-zero exit status 12.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/whs/Projects/ownDeploy/own-agent/agents/science/report.py", line 35, in <module>
    pdf = builder.build_pdf(main_tex_template)
  File "<decorator-gen-1>", line 2, in build_pdf
  File "/home/whs/Projects/ownDeploy/ownenv/lib/python3.6/site-packages/data/decorators.py", line 82, in _
    return f(*bvals.args, **bvals.kwargs)
  File "/home/whs/Projects/ownDeploy/ownenv/lib/python3.6/site-packages/latex/build.py", line 113, in build_pdf
    raise_from(LatexBuildError(base_fn + '.log'), e)
  File "/home/whs/Projects/ownDeploy/ownenv/lib/python3.6/site-packages/future/utils/__init__.py", line 398, in raise_from
    exec(execstr, myglobals, mylocals)
  File "<string>", line 1, in <module>
latex.exc.LatexBuildError: None

[aadibajpai]

Same error, I'm using MikTeX and have pdfLaTeX installed. I guess it could directly use pdflatex to build it instead of latexmk

Traceback (most recent call last):
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python37\lib\site-packages\latex\build.py", line 111, in build_pdf
    stderr=open(os.devnull, 'w'), )
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 328, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['latexmk', '-pdf', '-pdflatex=pdflatex -interaction=batchmode -halt-on-error -no-shell-escape -file-line-error %O %S', 'C:\\Users\\AADIBA~1\\AppData\\Local\\Temp\\tmpllaqidgp\\tmpwj0oq0wi.latex']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\aadibajpai\Desktop\ltx.py", line 263, in <module>
    latex.build_pdf(tex)
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python37\lib\site-packages\latex\build.py", line 232, in build_pdf
    return builder.build_pdf(source, texinputs)
  File "<decorator-gen-1>", line 2, in build_pdf
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python37\lib\site-packages\data\decorators.py", line 82, in _
    return f(*bvals.args, **bvals.kwargs)
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python37\lib\site-packages\latex\build.py", line 113, in build_pdf
    raise_from(LatexBuildError(base_fn + '.log'), e)
  File "C:\Users\aadibajpai\AppData\Local\Programs\Python\Python37\lib\site-packages\future\utils\__init__.py", line 398, in raise_from
    exec(execstr, myglobals, mylocals)
  File "<string>", line 1, in <module>
latex.exc.LatexBuildError: None