imports in scripts

[hottwaj]

This is possible a documentation issue for the most part...

It took me a while to realise that scripts are expected to be "standalone".

As things stand a script that imports local modules within the same folder will fail when "addscript" is used. This is because the script is parsed, doctored to only include ArgumentParser related code, and then saved in /tmp/. The doctored script is then executed, but the other python modules you import are not in the same "/tmp/" folder, so these imports fail and so "addscript" ultimately fails.

To get around this I made sure my scripts were using import paths relative to my django installation e.g. import my_app.my_library. This allows "addscript" to successfully process my script, however the script will then not run when I use it via the Wooey interface, because a copy of the original script (placed in a different folder) is executed "standalone" without reference to the Django/Celery installation.

So my temporary solution so far has been to add the appropriate folder to my PYTHONPATH, but this is also a bit awkward... If you have suggestions for better solutions they'd be appreciated.

I realise that the script parsing/copying elsewhere process is perfectly sensible and is done this way to solve other issues, so perhaps the current setup is just the way things will have to be.

Anyway, thanks for the amazing work you've done on this project!

[mfitzp]

Thanks for the bug report @hottwaj — I've noticed the same problem.

It would be good if we could support relative imports from the scripts ...at the very least add script and running it should work and fail in the same way! It might be possible to store the original path of the script and add to the PYTHONPATH when running it individually — or at least have it configurable within the database UI.

[Chris7]

@hottwaj. This should be possible by disabling the option execute_full_path in the script model within the admin. In this case, it'll look on your system path first, which should find the original script.

[hottwaj]

Thanks both.

@Chris7 I hadn't seen that option before and it could be part of a solution - but out of the box copies of scripts are saved in a special area, e.g. wooey_scripts/my_script_14WYYPr.py, and the admin interface only allows you to overwrite that script file... Of course I can change the file path via the Django shell but that's probably not what you meant.

Using your method though I could also add a symlink in wooey_scripts to the folder where my imports are, which would be a little better than having to change my PYTHONPATH.

[Chris7]

@hottwaj. Because of browser security, there really isn't another way. We have no idea what path a script came from because browser's don't shared that information. I put that option in specifically for this use case though -- where I have a script that is constantly changing but has a stable interface.

An issue though is that we now use xxx, which will break this because it will look in the current directory for the script. Thanks for the report, we are working on getting better methods to handle all situations.

The symlink approach sounds the best until this is resolved. Sorry for the inconvenience, we're racing to get everything stable for the initial release of the django-djangui + wooey merger so your report is appreciated.

[hottwaj]

No need to apologise! As I said, great project you guys have here.

I'm happy to use either workaround for now.

[mfitzp]

@Chris7 Could we support an execution path (or PYTHONPATH) setting to be added either to all scripts or on a script-by-script basis? It would allow custom imports that aren't real Python packages.

[hottwaj]

Just as an FYI, out of the box unticking the "execute full path" option also causes script execution to fail, as python is not being executed from a path where it can see saved scripts, e.g. see below from "console" output on web site.

python my_script_MiFNkuc.py 
python: can't open file 'my_script_MiFNkuc.py': [Errno 2] No such file or directory

Of course I can change the script path in the shell... But instead setting PYTHONPATH is perhaps the easiest way to go at the moment.

Thanks

[hottwaj]

I'm now getting around this problem by putting any modules I'm using in my scripts in a library on my PYTHONPATH. This seems cleaner than messing around with symlinks etc.

So perhaps the only thing to do is to add documentation explaining that a script can only import modules on the PYTHONPATH or in libraries in the Virtualenv Celery is running in?

[Chris7]

I think I have a good solution to this (as I'm currently facing it):

1. We need to add a new model that is the script executable settings. This will also have the side benefit of opening up support for people running pypy instead of python and can support bash/sh/etc. shell scripts. We will have a default for python that can be associated with scripts. Some features it should have: executable path (not a filefield, a charfield),
2. Scripts will have a foreign key to the script executable model (which later we can infer the script type and auto-assign the default interpretter for a given language).
3. Scripts will have the option to execute a path/name instead of a ScriptVersion. This will be a charfield, so scripts installed via pip can just be set to that charfield.

Supporting 3 will solve this problem, but having 1/2 will make this much more agile.

[jchubber]

It's been a few years, so I'd like to ask if adding Django apps to the pythonpath is still the recommended approach for this. If so, does anyone have any advice on how to do that the "right" or "wrong" ways? I've not done much with pythonpath in the past, and will be learning specifically to enable Wooey scripts to import models from other Django apps in the same project for queries.

[Chris7]

Until we have a model that lets users define an execution environment, yes, I think configuring the PYTHONPATH to include what you wish to import from is the best way.