A `nushell` wrapper for `virtualenv`

[amtoine]

Hi everyone :wave: :yum:

i've been using virtualenv to manage python virtual environments for a long time, with projects such as virtualenvwrapper on bash or virtualfish on fish :yum:

i could not find anything on this repo, nor on the discord :thinking: and i miss a nice wrapper for virtualenv in nushell :confused:

i've recently created a new project which is called amtoine/virtnualenv for now and i'll work on that as soon as i have some time :relieved:

no idea if that could be merged into nu_scripts, please let me know your thoughts on that project :wink:

cheers :tada: :wave:

[kubouch]

Hi, we have an official virtualenv integration: https://github.com/pypa/virtualenv/tree/main/src/virtualenv/activation/nushell

[amtoine]

@kubouch

Hi, we have an official virtualenv integration: https://github.com/pypa/virtualenv/tree/main/src/virtualenv/activation/nushell

yes i've seen that :yum: but i'm talking about a layer of abstraction above :smirk:

The current workflow

i'm able to create, activate and deactivate virtual environments :ok_hand: however, this has to be very manual, e.g.

virtualenv /path/to/venv
source /path/to/venv/bin/activate.nu
deactivate

The idea behind my script

i would like to write a script which allows the user to manage virtual environments in a friendly way :+1: for instance, with at least the following help and suite of subcommands:

virtnualenv x.x.x

Usage: vn <command> [<args>]

Available commands:
    activate                 Activate a virtualenv
    deactivate               Deactivate this virtualenv
    help                     Print VirtualFish usage information
    ls                       List all available virtual environments
    new                      Create a new virtualenv
    rm                       Delete one or more virtual environments
    tmp                      Create a virtualenv that will be removed when deactivated

Environment variables:
    VIRTNUALENV_HOME         the location where the virtualenvs will be installed (defaults to `$XDG_DATA_HOME/virtualenvs`).

the typical workflow would then be something like

$ vn new my_venv
my_venv has been created with python 3.10
$ vn ls
my_venv
$ vn activate my_venv 
(my_venv) $ pip install numpy
(my_venv) $ python --version
3.10
(my_venv) $ vn deactivate
$ vn rm my_venv
$ vn ls

and all the environments would be installed at the location stored in VIRTNUALENV_HOME, which could be set by the user :ok_hand:

hope that makes sense :wink: i think that would be a real plus for python developers willing to switch to nushell :yum:

[fdncred]

I meant to say this when you posted your question the other day, but we'd accept a PR for a functional abstraction. There's no harm in it and you're probably no alone wanting such a thing.

[kubouch]

Ah, ok. Sure, that sounds cool, go ahead! One reason I'm using conda and not virtualenv is because it is more user-friendly. I think such an abstraction for virtualenv would be nice.

[kubouch]

One note, though. We're changing how source is going to work: It will be called source-env and would preserve only the environment, not custom commands and aliases. This will impact virtualenv activation: It will be migrated to be an overlay (https://www.nushell.sh/book/overlays.html), so you'll have to call overlay add activate.nu instead of source activate.nu. (At least that's the current plan...)

Just letting you know that there might be some surprises in the future.

[amtoine]

@fdncred

I meant to say this when you posted your question the other day, but we'd accept a PR for a functional abstraction. There's no harm in it and you're probably no alone wanting such a thing.

that is cool :relieved: i've got some time, so i'll begin to work on that :yum:

@kubouch

Ah, ok. Sure, that sounds cool, go ahead! One reason I'm using conda and not virtualenv is because it is more user-friendly. I think such an abstraction for virtualenv would be nice.

then let head towards the abstraction :muscle: i'll keep you up to date when i've got something :ok_hand:

[amtoine]

@kubouch

One note, though. We're changing how source is going to work: It will be called source-env and would preserve only the environment, not custom commands and aliases. This will impact virtualenv activation: It will be migrated to be an overlay (https://www.nushell.sh/book/overlays.html), so you'll have to call overlay add activate.nu instead of source activate.nu. (At least that's the current plan...)

Just letting you know that there might be some surprises in the future.

okey, got it, thanks :+1:

i'll begin with a simple source, read the book more and then switch to source-env and overlay add :wink:

[azzamsa]

@amtoine nice idea! we need it so bad in nu. I am using virtualfish before. In meantime, do you switch back to your old shell to use virtualenvwrapper or virtualfish?

[amtoine]

@azzamsa

@amtoine nice idea! we need it so bad in nu.

not much time this week, but i've made a start on the module/overlay on amtoine/virtnualenv:wip/feature/base-virtnualenv :ok_hand: very bare bone, but i'll improve it greatly with my new knowledge about modules and overlays :+1:

I am using virtualfish before. In meantime, do you switch back to your old shell to use virtualenvwrapper or virtualfish?

i did that at the beginning, before getting tired of switching away from nushell :thinking: now, i source the virtual environment from ~/.local/share/virtualenvs :confused:

[amtoine]

Hello there :yum:

A bit of context

i'm currently writing a first simple draft of the wrapper in amtoine/virtnualenv:wip/feature/base-virtnualenv and i'm having some trouble adding the module as an overlay. according to the book and what @kubouch said, i'm trying to add my ./src/virtnualenv.nu module as an overlay with the command

overlay add --prefix src/virtnualenv.nu as vn

below are the compilation errors i have and do not really understand :+1:

1. Sourcing files

in the activate and deactivate subcommands, i try to source the virtualenv activation files but i have

> overlay add  --prefix src/virtnualenv.nu as vn
Error: nu::parser::sourced_file_not_found (link)

  × File not found
    ╭─[src/virtnualenv.nu:25:1]
 25 │     )
 26 │     source $activation_file
    ·            ────────┬───────
    ·                    ╰── File not found: $activation_file
 27 │ }
    ╰────
  help: sourced files need to be available before your script is run

possible fixes

• using full path: source ~/.local/share/virtualenvs/ocv/bin/activate.nu for instance
• using source-env: source-env $activation-file but the source-env command does not exist in the shell itself
• using overlay add: i get

Error: nu::parser::parse_mismatch (link)

  × Parse mismatch during operation.
    ╭─[src/virtnualenv.nu:25:1]
 25 │     )
 26 │     overlay add $activation-file
    ·                 ────────┬───────
    ·                         ╰── expected valid variable name
 27 │ }
    ╰────

2. Extra arguments

in the ls subcommand, i get

overlay add  --prefix src/virtnualenv.nu as vn
Error: nu::parser::extra_positional (link)

  × Extra positional argument.
    ╭─[src/virtnualenv.nu:62:1]
 62 │     let venvs = (
 63 │         ls (get-venvs-dir) --short-names |
    ·            ───────┬───────
    ·                   ╰── extra positional argument
 64 │         where type == dir |
    ╰────
  help: Usage: ls

this is because the ls subcommand overwrites the built-in ls :confused: is there a way to use ls as a subcommand of the vn overlay? :open_mouth: for now i've renamed this subcommand list, not to conflict with ls and the issue is gone :ok_hand:

3. Unknown flags

in the rm subcommand, i get

> overlay add  --prefix src/virtnualenv.nu as vn
Error: nu::parser::unknown_flag (link)

  × The `rm` command doesn't have flag `-r`.
    ╭─[src/virtnualenv.nu:79:1]
 79 │ ] {
 80 │     rm -rfv (get-venv-name $venv)
    ·         ┬
    ·         ╰── unknown flag
 81 │ }
    ╰────
  help: use rm --help for a list of flags

that's the same as above, i've renamed this remove but what about using vn rm? :wink:

[amtoine]

regarding source-env i get

> vn new foo
created virtual environment CPython3.10.6.final.0-64 in 115ms
  creator CPython3Posix(dest=/home/ants/.local/share/virtualenvs/foo, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/ants/.local/share/virtualenv)
    added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
> vn list
╭───┬────────────╮
│ 0 │ foo        │
╰───┴────────────╯
> vn activate foo
Error: nu::shell::external_command (link)

  × External command failed
    ╭─[src/virtnualenv.nu:25:1]
 25 │     )
 26 │     source-env $activation_file
    ·     ─────┬────
    ·          ╰── did you mean 'source'?
 27 │ }
    ╰────
  help: No such file or directory (os error 2)

probably not implemented for now :yum:

[fdncred]

source-env hasn't landed yet. I wouldn't wait for it. you should probably just use source for now.

[amtoine]

source-env hasn't landed yet. I wouldn't wait for it. you should probably just use source for now.

yep that's what i thought :+1:

but i can't use dynamic paths to source the virtual environment :thinking: i can't hardcode the path as i do not know in advance which venv will be used :thinking:

[kubouch]

1. The source-env should be possible to land relatively soon, we plan it for the next release (3 weeks from now). Note that it will only source the environment variables, not commands or aliases. And the default source is planned to be removed eventually.
2. If you replace ls with a custom ls, there is no way to call the former. I guess you made your own ls inside the module. Then, you can call hide ls before you call ls. This should hide your custom ls and recover the Nushell's ls.
3. Should be the same as 2. Commands inside the module are referenced with their non-prefixed names

However, it seems the solutions for 2. and 3. seem to be bugged. Seems like hiding a command inside itself doesn't work but it should so you could create wrappers around built-ins like you just tried.

I'd suggest stick to the remove and list for now until we get it fixed.

[amtoine]

1. The source-env should be possible to land relatively soon, we plan it for the next release (3 weeks from now). Note that it will only source the environment variables, not commands or aliases. And the default source is planned to be removed eventually.

okey, cool :yum:

the main things done in /path/to/bin/activate.nu are

def-env activate-virtualenv [] {
    ...
    load-env $new_env
}

activate-virtualenv

alias pydoc = python -m pydoc
alias deactivate = source '/home/ants/.local/share/virtualenvs/ocv/bin/deactivate.nu'

with the new source-env coming, pydoc and deactivate won't be accessible right? :thinking: not sure i see how to achieve a full venv activation here :confused:

I'd suggest stick to the remove and list for now until we get it fixed.

yep, that sounds totally fine to me, for now at least :ok_hand:

[kubouch]

The activate.nu will be a module and you'll have to call overlay add activate.nu. Overlays support everything, including aliases so it will be possible to define the deactivate and pydoc.

The only downside is that the overlay would be called activate. You could redefine it with as ... but then, instead of just deactivate, you'd have to call overlay remove ....

I didn't get to do it yet because I'm planning some changes to the module system and I would have to rewrite it again. The planned changes are outlined here -- https://hackmd.io/L0OKeubaRz2yve6K-F5ISw, it is quite relevant to what you're doing.

[amtoine]

The activate.nu will be a module and you'll have to call overlay add activate.nu. Overlays support everything, including aliases so it will be possible to define the deactivate and pydoc.

ooooh i see what the plan is :open_mouth: that will be nice if the virtualenv side follows the changes as well :yum:

The only downside is that the overlay would be called activate. You could redefine it with as ... but then, instead of just deactivate, you'd have to call overlay remove ....

that would be managed in my wrapper itself, so no big deal :+1: at least, the user won't have to mess with that :ok_hand:

I didn't get to do it yet because I'm planning some changes to the module system and I would have to rewrite it again. The planned changes are outlined here -- https://hackmd.io/L0OKeubaRz2yve6K-F5ISw, it is quite relevant to what you're doing.

no worries, there's no hurry :wink: and i'll have a look thanks for the link!

[kubouch]

One thing to take into account is that in overlay remove <name> the <name> must be a static string. So you cannot call overlay remove $name, for example.

[amtoine]

One thing to take into account is that in overlay remove <name> the <name> must be a static string. So you cannot call overlay remove $name, for example.

oooh :thinking: that looks unfortunate :confused:

i'll think about that in the following days :ok_hand:

[amtoine]

hello there :wave:

is there a plan or even a date for the overlay feature to be implemented on the virtualenv side? :yum:

[kubouch]

hello there 👋

is there a plan or even a date for the overlay feature to be implemented on the virtualenv side? 😋

Yes, 27th of September at the latest, as a part of 0.69, unless something breaks horribly. In 0.69, the old stuff will be deprecated and by then, the virtualenv should be ported as well. As of 0.68 we're releasing a set of changes that will make the port possible. I'll try to work on the virtualenv port shortly after so it should be ready well before 0.69.

[amtoine]

@kubouch

Yes, 27th of September at the latest, as a part of 0.69, unless something breaks horribly. In 0.69, the old stuff will be deprecated and by then, the virtualenv should be ported as well. As of 0.68 we're releasing a set of changes that will make the port possible. I'll try to work on the virtualenv port shortly after so it should be ready well before 0.69.

thanks for the information :relieved:

no pressure whatsoever, it's just i feel a bit stuck on this feature as i can not overlay add the virtualenv nushell layer for now :confused:

i am talking about a branch of mine above, would you like me to open a WIP / draft PR? :yum:

[ljnsn]

@amtoine is there any update on this? I'd love to use this feature :slightly_smiling_face:

[amtoine]

@amtoine is there any update on this? I'd love to use this feature 🙂

unfortunately, no... :cry:

i've had no time and stopped using Python at work and in my free time :confused:

[AucaCoyan]

I'm very interested in this feature and would like to apply a PR

[e1turin]

Hi, I tried to use recommended higher in topic virtualenv and have got problem. I've installed packet and created .venv directory with commands:

pip install virtualenv
python -m virtualenv .venv

but I can't activate environment with command nu .venv/Scripts/activate.nu, I receive error message:

~\Desktop> nu .venv\Scripts\activate.nu                                                             03.05.2024 12:57:38
Error: nu::parser::unexpected_keyword

  × Statement used in pipeline.
    ╭─[C:\Users\ivtur\Desktop\.venv\Scripts\activate.nu:96:27]
 95 │ export alias pydoc = python -m pydoc
 96 │ export alias deactivate = overlay hide activate
    ·                           ───┬───
    ·                              ╰── not allowed in pipeline
    ╰────
  help: 'overlay' keyword is not allowed in pipeline. Use 'overlay' by itself, outside of a pipeline.

Did I make any mistake, or it is problem of the given script?

[szevzol]

Hi, I tried to use recommended higher in topic virtualenv and have got problem. I've installed packet and created .venv directory with commands:

pip install virtualenv
python -m virtualenv .venv

but I can't activate environment with command nu .venv/Scripts/activate.nu, I receive error message:

~\Desktop> nu .venv\Scripts\activate.nu                                                             03.05.2024 12:57:38
Error: nu::parser::unexpected_keyword

  × Statement used in pipeline.
    ╭─[C:\Users\ivtur\Desktop\.venv\Scripts\activate.nu:96:27]
 95 │ export alias pydoc = python -m pydoc
 96 │ export alias deactivate = overlay hide activate
    ·                           ───┬───
    ·                              ╰── not allowed in pipeline
    ╰────
  help: 'overlay' keyword is not allowed in pipeline. Use 'overlay' by itself, outside of a pipeline.

Did I make any mistake, or it is problem of the given script?

At the very beginning of activate.nu there is a hint:

# virtualenv activation module
# Activate with `overlay use activate.nu`
# Deactivate with `deactivate`, as usual

[kubouch]

It shows a bug in export alias, it shouldn't fail like that. But the issue can be closed, the correct way to activate the environment is overlay use.