Multiple adopath additions in profile.do breaks the kernel

[gaksaray]

Problem description

Kernel doesn't start when my profile.do contains more than one ado-path additions.

For example, if my profile.do contains:

set more off

adopath + "C:/Users/gaksaray/Desktop/GitHub/stata-gautil"

everything works fine.

But, if my profile.do contains:

set more off

adopath + "C:/Users/gaksaray/Desktop/GitHub/stata-gautil"
adopath + "C:/Users/gaksaray/Desktop/GitHub/stata-uniqueby"

I get the following:

Traceback (most recent call last):
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\__main__.py", line 4, in <module>
    IPKernelApp.launch_instance(kernel_class=StataKernel)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\traitlets\config\application.py", line 844, in launch_instance
    app.initialize(argv)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\traitlets\config\application.py", line 87, in inner
    return method(app, *args, **kwargs)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\ipykernel\kernelapp.py", line 582, in initialize
    self.init_kernel()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\ipykernel\kernelapp.py", line 444, in init_kernel
    kernel = kernel_factory(parent=self, session=self.session,
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\traitlets\config\configurable.py", line 537, in instance
    inst = cls(*args, **kwargs)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\kernel.py", line 71, in __init__
    self.stata = StataSession(self)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\stata_session.py", line 87, in __init__
    self.init_windows()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\stata_session.py", line 166, in init_windows
    self.start_log_aut()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\stata_session.py", line 230, in start_log_aut
    self.fd = Path(log_path).open()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\pathlib.py", line 1241, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\pathlib.py", line 1109, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\gaksaray\\.stata_kernel_cache\\log0.log'
[JupyterQtConsoleApp] KernelRestarter: restarting kernel (1/5), keep random ports
[JupyterQtConsoleApp] WARNING | kernel restarted
Traceback (most recent call last):
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\__main__.py", line 4, in <module>
    IPKernelApp.launch_instance(kernel_class=StataKernel)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\traitlets\config\application.py", line 844, in launch_instance
    app.initialize(argv)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\traitlets\config\application.py", line 87, in inner
    return method(app, *args, **kwargs)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\ipykernel\kernelapp.py", line 582, in initialize
    self.init_kernel()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\ipykernel\kernelapp.py", line 444, in init_kernel
    kernel = kernel_factory(parent=self, session=self.session,
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\traitlets\config\configurable.py", line 537, in instance
    inst = cls(*args, **kwargs)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\kernel.py", line 71, in __init__
    self.stata = StataSession(self)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\stata_session.py", line 87, in __init__
    self.init_windows()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\stata_session.py", line 166, in init_windows
    self.start_log_aut()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\stata_session.py", line 230, in start_log_aut
    self.fd = Path(log_path).open()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\pathlib.py", line 1241, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\pathlib.py", line 1109, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\gaksaray\\.stata_kernel_cache\\log0.log'
[JupyterQtConsoleApp] KernelRestarter: restarting kernel (2/5), keep random ports
[JupyterQtConsoleApp] WARNING | kernel restarted
Traceback (most recent call last):
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\__main__.py", line 4, in <module>
    IPKernelApp.launch_instance(kernel_class=StataKernel)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\traitlets\config\application.py", line 844, in launch_instance
    app.initialize(argv)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\traitlets\config\application.py", line 87, in inner
    return method(app, *args, **kwargs)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\ipykernel\kernelapp.py", line 582, in initialize
    self.init_kernel()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\ipykernel\kernelapp.py", line 444, in init_kernel
    kernel = kernel_factory(parent=self, session=self.session,
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\traitlets\config\configurable.py", line 537, in instance
    inst = cls(*args, **kwargs)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\kernel.py", line 71, in __init__
    self.stata = StataSession(self)
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\stata_session.py", line 87, in __init__
    self.init_windows()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\stata_session.py", line 166, in init_windows
    self.start_log_aut()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\site-packages\stata_kernel\stata_session.py", line 230, in start_log_aut
    self.fd = Path(log_path).open()
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\pathlib.py", line 1241, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "C:\Users\gaksaray\scoop\apps\miniconda3\4.9.2\envs\stata_nb\lib\pathlib.py", line 1109, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\gaksaray\\.stata_kernel_cache\\log0.log'

and kernel keeps dying repeatedly;

Or, if I don't delete the contents of ~/.stata_kernel_cache before running jupyter, I get .exit, clear after I try to run a command in jupyter):

sysuse auto
. exit, clear

So for some reason multiple ado-path additions causes the kernel to die before start.

Debugging log

console_debug.log

Other information

• OS: Windows 10 64bit
• Stata version: 15.1
• Package version: 1.12.2

[kylebarron]

if I don't delete the contents of ~/.stata_kernel_cache

You should never need to touch or delete anything in that folder. Given that the error is

No such file or directory: 'C:\Users\gaksaray\.stata_kernel_cache\log0.log

It seems like that should be completely orthogonal to your profile.do

[gaksaray]

@kylebarron sorry for the confusion; I only did that to recreate a fresh install state but it doesn't really matter whether the cache files are in place or not; the problem is always there.

To demonstrate the problem again clearly, I did the following:

• Clean installed stata_kernel in conda environment, and ran python -m stata_kernel.install
• I removed profile.do from the C:\ado and ran jupyter qtconsole --kernel=stata. It worked totally fine. No problems here. I quit the kernel and qtconsole.
• Next, I add profile.do back to C:\ado, consistsing only three ado-path additions as follows:

  adopath + "C:/Users/gaksaray/Desktop/GitHub/stata-gautil"
  adopath + "C:/Users/gaksaray/Desktop/GitHub/stata-uniqueby"
  adopath + "C:/Users/gaksaray/Desktop/GitHub/epstopdf"
  

• I run jupyter qtconsole --kernel=stata again. First thing I run adopath command to see if the ado-path addition in my profile.do has been read. I get

In [1]: adopath
  [1]  (BASE)      "C:\Program Files (x86)\Stata15\ado\base/"
  [2]  (SITE)      "C:\Program Files (x86)\Stata15\ado\site/"
  [3]              "."
  [4]  (PERSONAL)  "c:\ado\personal/"
  [5]  (PLUS)      "c:\ado\plus/"
  [6]  (OLDPLACE)  "c:\ado/"
  [7]              "C:\Users\gaksaray\miniconda3\envs\stata_nb\Lib\site-packages\stata_kernel\ado"

in other words, none of the ado paths specified in profile.do is read by the kernel. I repeat the adopath command and get:

  In [2]: adopath
  . exit, clear

after which kernel becomes unresponsive. Here is the log: console_debug.log

I can replicate this every single time: whenever there is an incongruence between the adopaths defined in profile.do and the adopaths initialized by kernel (which to me seems to happen when more than 2 adopaths are defined in profile.do), kernel dies unexpectedly at some point within notebook.

[kylebarron]

I don't know how adopaths work, (I don't even use Stata anymore), but I'm guessing you're overwriting a path that the kernel expects. See here we define a custom adopath: https://github.com/kylebarron/stata_kernel/blob/393feaae8b0978579ddbf4418d488b0612bcfb76/stata_kernel/stata_session.py#L104

[mcaceresb]

@gaksaray There are different things here:

1. If you try to exit via exit, clear from the kernel, it will become unresponsive because you have exited Stata but left an instance of a console running. You need to exit via the kernel.
2. Your profile.do is not getting read. It might be in the wrong place? Does it get read if you start Stata without the kernel? When I start the kernel, I see the messages

running /home/mauricio/ado/personal/sysprofile.do ...

running /home/mauricio/ado/profile.do ...

Since they don't appear in your log, I wonder whether your profile.ado is in the right place. What is

disp "`c(sysdir_personal)'"
disp "`c(sysdir_oldplace)'"

?

3. Your original message contained a crash. Unsure how to reproduce, but might be fixed by point 2 above.

[gaksaray]

@mcaceresb

1. I know but I never write exit, clear on console. As you can see on my post, I just used adopath command twice: first it shows the adopaths and second it exits itself! So that exit, clear is just shown to me by the kernel after I use adopath command second time, and then the kernel tries to reconnect but unable to do so, and it just dies.
2. It's is in the usual C:/ado folder. If I run Stata by itself (no kernel) everything works fine and Stata loads all the adopaths defined in profile.do.

. disp "`c(sysdir_personal)'"
c:\ado\personal/

. disp "`c(sysdir_oldplace)'"
c:\ado/

It is always read by the kernel just fine if it consists only one or two ado paths. Things get weird if the does more than that.

3. They don't appear in my log because the kernel breaks and cannot even write the log. That's why I said I tried deleting the cache files to recreate "what happens on a fresh clean install with no cache files yet, and I run the kernel with a profile.do consisting of 3 or more adopath additions", and it just crashes on the prompt repeatedly as shown on that crash error message; could not even write the log because kernel cannot start. So I would remove the profile.do from C:/ado, run the kernel successfully to allow it to write cache log files, then add profile.do back to C:/ado, run kernel again and we are back again at (1). This I can reproduce every single time.

[mcaceresb]

@gaksaray I am a bit confused, so let me recap:

• You start the kernel with no profile.do. This works fine.
• You close the kernel.
• You add profile.do to C:\ado.
• You start Stata without the kernel. This works fine.
• You start the kernel. This is where I am confused:
  • Does it crash, showing the error message here in your original post?
  • Does it not crash, and give you a prompt, after which you type adopath and get exit, clear somehow, which you did not type?

Does this also happen if you put things in C:\ado\personal\sysprofile.do instead?

[gaksaray]

@gaksaray I am a bit confused, so let me recap:

• You start the kernel with no profile.do. This works fine.
• You close the kernel.
• You add profile.do to C:\ado.
• You start Stata without the kernel. This works fine.

Exactly correct so far.

• You start the kernel. This is where I am confused:
  • Does it crash, showing the error message here in your original post?

No, it doesn't crash. (This is because the previous steps have already produced log files in ~/.stata_kernel_cache folder, so no "No such file or directory" error; but it won't write on log files as I explain below).

• Does it not crash, and give you a prompt, after which you type adopath and get exit, clear somehow, which you did not type?

It gives me a prompt. I type adopath and see the list of adopaths, which does not include the ones specified in profile.do. It doesn't update itself to read any profile.do that consists multiple adopath additions. This means either on the second cell, or at some point in the notebook, I will type a Stata command and the kernel will give me .exit, clear output and become unresponsive. I quit the qtconsole, which supposed to update the log files in ~/.stata_kernel_cache; yet the log files are not updated. They still show the time on the first step! In other words, when this happens, kernel cannot write logs as well.

Does this also happen if you put things in C:\ado\personal\sysprofile.do instead?

Unfortunately it does.

I am attaching a screen record to demonstrate this whole process: https://user-images.githubusercontent.com/72267919/107787291-f5e9e800-6d5f-11eb-9555-dab45d4bbbdb.mp4

[mcaceresb]

@gaksaray I see. My guess is that this has to do with how Stata's automation feature on Windows interacts with profile.do. I would try some things to debug (I seldom have access to Stata on Windows so hard to debug myself, and this error doesn't show up in console mode on my system):

• Try moving profile.do into the folder where your Stata executable is installed.
• Try an empty profile.do and see if the kernel still has this problem.
• Try a profile.do with something unrelated to adopath, like global test hello and see if $test is defined.
• If it is defined, try defining at least 3 globals and see if that still works.
• Try adding exit 0 to the end of profile.do
• Try removing the quotes from the paths in profile.do

For whatever reason Stata isn't registering profile.do properly; it's odd that you've mentioned it happens with many adopath calls but not with just one. Want to see if it happens generally or if it's specific to adopath for some reason.

[gaksaray]

@mcaceresb

• Try moving profile.do into the folder where your Stata executable is installed.

Problem persists: . exit, clear

• Try an empty profile.do and see if the kernel still has this problem.

Kernel works fine. But this is expected. The problem seems to occur only when there are 3 or more adopaths in profile.do. For example, if I include only 1 adopath call in profile.do, kernel works as it should.

• Try a profile.do with something unrelated to adopath, like global test hello and see if $test is defined.

Kernel works fine. I confirm that $test is defined properly within kernel.

• If it is defined, try defining at least 3 globals and see if that still works.

Works fine. All three globals are defined properly within kernel.

• Try adding exit 0 to the end of profile.do

I added exit 0 at the end of the original profile.do with three adopaths. Does not fix it: . exit, clear.

• Try removing the quotes from the paths in profile.do

Again, no effect. Same problem: . exit, clear.

For whatever reason Stata isn't registering profile.do properly; it's odd that you've mentioned it happens with many adopath calls but not with just one. Want to see if it happens generally or if it's specific to adopath for some reason.

It seems to be isolated to multiple adopath calls.