Support common IDE features (dlang-vscode#53)

[LaurentTreguier]

From https://github.com/dlang-vscode/dlang-vscode/issues/53:

• [ ] Functions snippets
• [x] Find references
• [ ] Function call hierarchy
• [ ] Type hierarchy
• [ ] Semantic highlighting
• [x] Find and replace symbol
• [ ] Auto-implement getters and setters

[testing-in-chrome]

Thank you very much! :) :+1:

[LaurentTreguier]

So it's been a few months since the original issue was posted, and things haven't really moved regarding the features proposed here; so I thought I'd clear up a few things about what I am doing, and planning to do.

Right now, DLS relies on DFMT, DCD and D-Scanner for most things. Although they are mostly working great, there are still some issues: DFMT not being able to properly format struct initializers, or finding declarations being sometimes completely off for example. On top of that there are also lots of things to improve in DLS itself, I've introduced tons of regressions in various releases, and each of these releases also take more and more resources to compile. This is why I'm working more on improving the little things rather than working towards bigger new features: I'm aiming to get a great experience with the current functionality first, and avoid ending up with tons of half-working features.

I've started putting together a test suite. It doesn't test much for now (basic initialization, formatting and symbol searching) but even that alone showed me a few bugs I never realized were there, ready to potentially break things with any client LSP implementation I don't test myself. Heck, if you browse the v0.6.x release, two of them are missing, I had removed them because they would literally crash on startup! So having even the most minimal test suite helps. I plan on adding more tests incrementally with each minor release.

For DFMT, I counted 2 routes I could take:

• continuing to patch it
• creating a brand new formatter

The problem I see with patching DFMT is that the code already feels like a big clump of patches itself in a number of places... I have started some work to make a new formatter on the dls-format branch. It is very far from being ready though, and I am currently trying to produce another (easier) solution: an indenter. As the name implies, it's supposed to keep most of the user's formatting and simply re-indent the code (and trim whitespace, and perhaps later on put spaces around operators and such). This is certainly not a perfect replacement, but at least it will enable some basic formatting without too much risk of butchering some parts of the code.

For DCD, I am thinking about the possibility to replace its usage with custom code (but still using libdparse). Like the others, it was designed to be used via the command line and not as a library, which makes some things awkward (I can't access DCD's inheritance information to show a type hierarchy and would have to re-implement everything myself, even though DCD does it already). I'm also not a fan of the fact that it uses a cache; I would prefer to have a reduced memory footprint. This one is going to be harder than the formatting I think.

For D-Scanner, I have nothing planned, it works fine and I have no problem with it (other than inconsistent errors codes).

TL;DR: re-implementing a formatter and a "symbol handler", so to speak, would help in solving formatting issues, reducing memory usage, maybe getting better definitions/references/... , as well as making further changes easier to implement.

[testing-in-chrome]

So it's been a few months since the original issue was posted, and things haven't really moved regarding the features proposed here; so I thought I'd clear up a few things about what I am doing, and planning to do.

Thank you for the update appreciate it!

Right now, DLS relies on DFMT, DCD and D-Scanner for most things. Although they are mostly working great, there are still some issues: DFMT not being able to properly format struct initializers, or finding declarations being sometimes completely off for example. On top of that there are also lots of things to improve in DLS itself, I've introduced tons of regressions in various releases, and each of these releases also take more and more resources to compile. This is why I'm working more on improving the little things rather than working towards bigger new features: I'm aiming to get a great experience with the current functionality first, and avoid ending up with tons of half-working features.

That's a good idea.

I've started putting together a test suite. It doesn't test much for now (basic initialization, formatting and symbol searching) but even that alone showed me a few bugs I never realized were there, ready to potentially break things with any client LSP implementation I don't test myself. Heck, if you browse the v0.6.x release, two of them are missing, I had removed them because they would literally crash on startup! So having even the most minimal test suite helps. I plan on adding more tests incrementally with each minor release.

For DFMT, I counted 2 routes I could take:

• continuing to patch it
• creating a brand new formatter

The problem I see with patching DFMT is that the code already feels like a big clump of patches itself in a number of places... I have started some work to make a new formatter on the dls-format branch. It is very far from being ready though, and I am currently trying to produce another (easier) solution: an indenter. As the name implies, it's supposed to keep most of the user's formatting and simply re-indent the code (and trim whitespace, and perhaps later on put spaces around operators and such). This is certainly not a perfect replacement, but at least it will enable some basic formatting without too much risk of butchering some parts of the code.

I never relied on DFMT. I always found it to be very buggy unfortunately. Instead, I use clang-format. It works for the most part even though it doesn't support D at all. However I have to add things like:

    // clang-format off
    mList ~= item;
    // clang-format on

Which is obviously not very convenient, but at least it works.

For DCD, I am thinking about the possibility to replace its usage with custom code (but still using libdparse). Like the others, it was designed to be used via the command line and not as a library, which makes some things awkward (I can't access DCD's inheritance information to show a type hierarchy and would have to re-implement everything myself, even though DCD does it already). I'm also not a fan of the fact that it uses a cache; I would prefer to have a reduced memory footprint. This one is going to be harder than the formatting I think.

For D-Scanner, I have nothing planned, it works fine and I have no problem with it (other than inconsistent errors codes).

I vaguely remember there were plans to make DMD itself usable as a library kinda like llvm or libclang, but I'm not sure where that went.

TL;DR: re-implementing a formatter and a "symbol handler", so to speak, would help in solving formatting issues, reducing memory usage, maybe getting better definitions/references/... , as well as making further changes easier to implement.

Thank you for putting in all those efforts!!!

[LaurentTreguier]

Update

Formatting

Since v0.24.0, DLS has a custom formatting backend, that can be enabled by setting d.dls.format.engine to "builtin" in the configuration. The main difference with DFMT is that it doesn't try to break down lines, and only tries to indent code lines as they are. You can put arguments of a long function call each on a different line, and it will simply get indented one level. With this for example:

void foo()
{
S s = {a:3};
}

DFMT will produce this:

void foo()
{
    S s = {a:
    3};
}

while the new DLS formatter will keep it as a one-liner:

void foo()
{
    S s = { a : 3 };
}

There are probably still bugs around, but this is how it's shaping up right now.

Regarding other things: libdparse/dsymbol/dcd/dscanner...

For the foreseeable future, I'm sticking with these libraries. DMD is supposed to be usable as a library; however because it uses versions like 2.085.0 and not 2.85.0 for God knows what reason, it can't be used a dub packages, since v2.085.0 is not semver-compliant. Well, it can, but only with branches and not fixed version, which makes it unusable in practice for stability, build reproducing and such. With DLS 0.24.0 I did some something that could change things a bit. Originally, the update system would use dub to fetch and build a newer DLS version. This was a pain because of the RAM used and the time it took (especially considering that it would compiler in release mode...), so I looked into using binary releases instead.

Now, it's been around 10 months that DLS has had automated binaries for all major platforms, in both 32bit and 64bit (although the 32bit variants have literally 0 downloads). So I removed the old code that tried to build DLS when it failed to download it, since a failure to download anything would mean the failure to fetch the new package with dub as well anyway. Since building from the dub package is not an "officially supported" method of installing/updating DLS anymore, and is only useful for dls:bootstrap, I am thinking that I could try using DMD as a library in a git submodule. The downside is that it would basically mean an almost complete rewrite. The upside is that I would be using the exact same frontend as the compilers, meaning an accurate parsing, and possibly access to information like class inheritance, etc.

The reason I thought about this is that it has been proposed as a potential GSOC project (https://wiki.dlang.org/GSOC_2019_Ideas#Language_Server_Protocol_for_D), so even if I don't do it, maybe someone else will take a stab at it later this year.