Autoimport

[bagel897]

I've been working on an autoimport plugin for pylsp. I'm wondering what would the feasibility of upstreaming this to mainline Jedi would be. The implementation consists of two parts:

1. The main library, in rope. Its written using python's ast. My new implementation is largely independent of rope's structure. This provides suggestions.
2. The frontend, in pylsp. Its written using parso. This determines when to request for autoimport.

Jedi supporting autoimport would benefit a larger audience and merging should_import with jedi would deduplicate work.

[davidhalter]

That sounds great. If it's using parso already would it be possible to just push it into Jedi?

At the moment the problem is mostly that Jedi doesn't really work with a lot with caches, so it's hard to make it fast. So you would have to deal with that. I feel like if you work with parso anyways, that's a good start I guess.

I'm currently writing a Rust implementation of Jedi, but that will take a few more years until it's good. So any help here is appreciated! The code would probably belong in jedi/api/refactoring/__init__.py.

[bagel897]

1. The only two parts that use parso are for determing what names are already in a file and if a completion should be suggested. The majority of the code is part of the rope project, so either jedi would need to depend on rope for autoimport or copy and adapt the code for jedi. I would prefer if there wasn't two copies of the same code that I'm somewhat responsible for.
2. Caching - it uses its own sqlite3 cache.
3. Rust rewrire- that sounds like a cool project, would it be possible to contribute? Is it focused on LSP support or will it try to support as many editors directly like jedi does? Will it use LSIF?
4. Location - it works by suggesting completions (like List with associated textedit of from typing import List) and should probably put in completion

[davidhalter]

The only two parts that use parso are for determing what names are already in a file and if a completion should be suggested. The majority of the code is part of the rope project, so either jedi would need to depend on rope for autoimport or copy and adapt the code for jedi. I would prefer if there wasn't two copies of the same code that I'm somewhat responsible for.

I kind of understand. I'm also not sure I would want sqlite in there. Usingn Rope as a depencency doesn't make any sense. Since most people actually like Jedi, because they do not have to use rope. It's why I wrote Jedi in the first place.

Rust rewrire- that sounds like a cool project, would it be possible to contribute?

Currently no, there's just too many moving pieces and I have to find a good basic architecture.

Is it focused on LSP support or will it try to support as many editors directly like jedi does?

It will probably have LSP support, but that's like 0.1% of the work (if not less). It's pretty easy once you have all the rest.

Will it use LSIF?

Probably not. LSIF will be extremely slow compared to the Rust binary API and it won't really make sense.

[bagel897]

What alternative caching/indexing layer would you suggest instead of sqlite3? Would it be possible to move autoimport to its own package so both rope and jedi can use it? The current autoimport implementation doesn't require using the rest of rope if that's what you mean.

[davidhalter]

Would it be possible to move autoimport to its own package so both rope and jedi can use it?

I'm not sure. Since Jedi already uses parso and manages its caches, I would strongly prefer this being part of Jedi.

What alternative caching/indexing layer would you suggest instead of sqlite3?

I'm not sure what would fit Jedi to be honest... Maybe sqlite3 is the best there. I generally prefer simpler datastructures for such things (like a simple text file that you could search with grep with all of its definitions).

[bagel897]

I would strongly prefer this being part of Jedi

I want to explore the options available before concluding rewriting around a thousand lines of code is the only option. Furthermore, I'm not sure if I can commit to that option in terms of time. If that is the only option, I'd prefer to close this but can provide pointers to someone who wants to try.

simpler datastructures for such things

Currently, this index sits at around 5.2 Mb on my machine. This is 40590 entries of which 6277 are from the standard library. This could be reduced by only using modules defined in a pyproject.toml, but still will need to be large for large projects. I think that it's possible to use a text file for this but sqlite makes queries on this easier and much faster and simplifies making this persistent.

[davidhalter]

You are probably right about persistence. I'm also not sure if my approach would even be feasible. I have very good approaches for Rust, but there it's easier to structure memory.

[bagel897]

Its probably possible, but it'd be slower and harder for little gain.

[davidhalter]

Yeah, I guess sqlite3 is probably fine to have as an index, as long as invalidation works properly.

[bagel897]

I am working on that. I'll use the modified times of the packages to keep track of modules that need to be updated. I'm also going to release the rope autoimport code as a separate package and update this issue when that is done. Let me know if you have feedback, suggestions or any requirements that I should be aware of.