Executing a none python script using "Spacy Projects" generates an error

[dhirajsuvarna]

This issue is related to Spacy Project

How to reproduce the behaviour

Add a non-python command in the script section of the project.yml For eg. In the below snippet, I have included a command of mkdir

  - name: "train"
    help: "Train a named entity recognition model"
    script:
      - "mkdir C:/test_folder"
      - "python -m spacy train ${vars.config} --output ./output --paths.train assets/${vars.train}.spacy --paths.dev assets/${vars.dev}.spacy"
    deps:
      - "assets/${vars.train}.spacy"
      - "assets/${vars.dev}.spacy"
    outputs:
      - "output/model-best"

On including this command, I get following error -

=================================== train ===================================
Running command: mkdir C:/test_folder
Traceback (most recent call last):
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\spacy\__main__.py", line 4, in <module>   
    setup_cli()
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\spacy\cli\_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\click\core.py", line 1137, in __call__    
    return self.main(*args, **kwargs)
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\click\core.py", line 1062, in main        
    rv = self.invoke(ctx)
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\click\core.py", line 1668, in invoke      
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\click\core.py", line 1668, in invoke      
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\click\core.py", line 1404, in invoke      
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\click\core.py", line 763, in invoke       
    return __callback(*args, **kwargs)
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\typer\main.py", line 500, in wrapper      
    return callback(**use_params)  # type: ignore
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\spacy\cli\project\run.py", line 42, in project_run_cli
    project_run(project_dir, subcommand, overrides=overrides, force=force, dry=dry)
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\spacy\cli\project\run.py", line 100, in project_run
    run_commands(cmd["script"], dry=dry, capture=capture)
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\spacy\cli\project\run.py", line 183, in run_commands
    run_command(command, capture=capture)
  File "C:\Users\10347298\.conda\envs\nlpenv\lib\site-packages\spacy\util.py", line 955, in run_command        
    raise FileNotFoundError(
FileNotFoundError: [E970] Can not execute command 'mkdir C:/test_folder'. Do you have 'mkdir' installed?       

Info about spaCy

• spaCy version: 3.2.4
• Platform: Windows-10-10.0.19042-SP0
• Python version: 3.8.11
• Pipelines: en_core_web_sm (3.1.0)

[dhirajsuvarna]

Proposed Solution

On Analyzing the issue more, I found out that, if I add a option of shell=True in the subprocess.run() command in this line here then the error is not thrown.

Can someone please verify this?

[rmitsch]

Thanks for raising this issue and posting a workaround! We will look into it (shell=True might have other effects we need to evaluate first).