Running python modules

[steveoh]

in order to run python -m mymodule using python shell I had to bend the expectations on how it is used.

        var shellOptions = {
            scriptPath: 'press',
            cwd: __dirname + '/scripts',
            pythonOptions: ['-m']
        };

        PythonShell.run('', shellOptions, function (err) {

I had to set the scriptPath to be the module name, I then had to set the cwd to be the path to the module, and setting the pythonOptions for -m was pretty normal. When running this module, I had to pass an empty string to PythonShell.run.

Am I abusing this or is it just not designed to work friendly with modules?

[extrabacon]

I think you are correct, modules are not really user friendly with the current set of options. Maybe we could add a module property, which could set scriptPath and pythonOptions automatically. However I would leave cwd untouched.

For example:

// using the new `module` property

var options = {
  module: 'my_module',
  cwd: __dirname + '/scripts'
};

// would be the same as

var options = {
  scriptPath: 'my_module',
  cwd: __dirname + '/scripts',
  pythonOptions: ['-m']
};

What do you think?

[steveoh]

I think it is a bit confusing to use module and call pythonshell.run with an empty parameter. This solution could work, I just don't think it would be obvious and would need to be a documented use.

Alternatively if you pass a value to script that doesn't have a .py extension, could we assume it was a module and alter the creation of this.script?

[extrabacon]

Sorry, I forgot to mention that the script argument would be optional with this approach.

Here's an example:

// using the new `module` property
// also, `script` argument is omitted

PythonShell.run({
  module: 'my_module',
  cwd: __dirname + '/scripts'
}, callback);

// would be the same as

PythonShell.run('', {
  scriptPath: 'my_module',
  cwd: __dirname + '/scripts',
  pythonOptions: ['-m']
}, callback)

[steveoh]

Yeah that would be great.

[godardt]

Is there any progress on this? Now I get Error: scriptPath cannot be empty! You must give a script for python to run with the latest version with no way to pass in python modules with the -m argument. Thanks.

[Almenon]

I started looking into this. My first thought was that this should be possible by executing something like below:

            PythonShell.run('-m timeit', {
                args: ['-n 1', `'x=5'`]
            }, function (err, results) {
                console.log(results)
            });

But for some reason I got

"C:\Program Files\Python36\python.exe: No module named timeit"

I'll have to look into this further. Python should be able to find timeit, that's a inbuilt module

[godardt]

Hi @Almenon , yeah I tried a lot of hacks like this one too but none of them worked unfortunately :( . The way I start my python project is with python -m src.main instead of python src/main.py because the latter won't resolve the import src.* modules.

[Almenon]

It turns out that for some rediculous reason -m and timeit have to be seperated. Only then will it work.

See example here: https://repl.it/@almenon/FabulousFocusedDiscussion?language=nodejs

Or you can look at my unit test in the commit above:

        it('should be able to run modules', function(done){
            PythonShell.defaultOptions = {};
            
            PythonShell.run('-m', {
                args: ['timeit', '-n 1', `'x=5'`]
            }, function (err, results) {

                PythonShell.defaultOptions = {
                    // reset to match initial value
                    scriptPath: pythonFolder
                };

                if (err) return done(err);
                results.should.be.an.Array();
                results[0].should.be.an.String();
                results[0].slice(0,6).should.eql('1 loop');
                done();
            });
        })

[Almenon]

Maybe it's because argument parsing is different between the python exe and the timeit module. I'm guessing the python exe is too dumb to know that "-m timeit" is actually two arguments, whereas whatever parsing timeit uses can recognize that "-n 1" should be ["-n", "1"]

Edit: wait no that that's not right. The error was "No module named timeit" so python correctly parsed the module. Why would it not be able to find it then??? Bizzare.

[godardt]

I dunno, but apart from running scripts, it seems this project doesn't allow to run entire python projects/modules efficiently. So I'll just fall back to a library which allows me to run shell command as I don't have much choice here :(

[Almenon]

I just gave you a way to use modules :/

[godardt]

@Almenon Oh excuse me I've read only one message on the two in diagonal, thanks a lot!

[godardt]

@Almenon actually that doesn't really work for my case because my module is not global. Basically I have this:

            let pyshell = new PythonShell("-m", {
                mode: 'text',
                pythonOptions: ['-u'], // get print results in real-time
                args: [`src.main`, `eval`, `--input_wsi=${currentWSI.path}`, `--output=${dir}`,
                    `--pred_mode=${pred_mode}`, `--downsample=${downsample}`, `--model=${model}`,
                    `--stdlog`]
            });

But that doesn't work and I end up getting:

/Users/Ekami/Programs/anaconda/bin//python3: No module named src.main.__main__; 'src.main' is a package and cannot be directly executed

So now I'm wodering what is the difference between a python module and a python package and how I can execute the latter =/ . In my console a simple python -m src.main was sufficient.

NB: Even removing the -u doesn't work

[Almenon]

Google has a pretty good explanation for the difference:

A package is a collection of Python modules: while a module is a single Python file, a package is a directory of Python modules containing an additional __init__.py file, to distinguish a package from a directory that just happens to contain a bunch of Python scripts.

[godardt]

Thanks, but even considering this I tried few options which didn't seem to help :( Like:

            let pyshell = new PythonShell("-m", {
                mode: 'text',
                scriptPath: path.join(src_path, "src"),
                args: [`src.main`, `eval`, `--input_wsi=${currentWSI.path}`, `--output=${dir}`,
                    `--pred_mode=${pred_mode}`, `--downsample=${downsample}`, `--model=${model}`,
                    `--stdlog`]
            });

But I get:

python3: can't open file '/Users/Ekami/workspace/GUI-PW/python_modules/Mock/src/-m': [Errno 2] No such file or directory

I don't really know how to properly cd into the Mock directory and execute python -m src.main from there =/

[godardt]

I think I got it:

            let pyshell = new PythonShell(path.join("src", "main.py"), {
                mode: 'text',
                scriptPath: src_path,
                args: [`eval`, `--input_wsi=${currentWSI.path}`, `--output=${dir}`,
                    `--pred_mode=${pred_mode}`, `--downsample=${downsample}`, `--model=${model}`,
                    `--stdlog`]
            });

Setting the scriptPath allowed Python-Shell to find my import src.* imports. Thanks a lot for your time and your patience @Almenon !