Expose AST for use in other extensions

[meisZWFLZ]

I'm asking for a non-user-facing command that would provide the same information as the Show AST command, but rather than displaying the AST to the user, the command would instead return the AST.

This would be useful for extensions that would enjoy a greater understanding of the code than vscode's symbol interface. For example, I am creating a vscode extension that interprets function calls into a graphical representation, which can be modified to edit the underlying function call (these function calls are robot instructions).

[HighCommander4]

This is an interesting idea.

I don't think vscode-clangd currently exposes any API surface meant for consumption by other vscode extensions, but I believe that sort of thing is possible to do.

I think a first step here would be to come up with a proposed API.

[meisZWFLZ]

Alright, I believe there is two different methods to do this:

1. Expose via command - Register a command called "clangd.ast.retrieve" that takes a Range and Uri as parameters and returns an ASTNode (I have already implemented this)
2. Expose via Extension.exports - In the activate() function, return an object containing astRetrieve() with a signature similar to that of the command (There is no documentation/examples on this that I can find, aside from the Extension class docs)

I don't know if either method is any better than the other, but I'd be happy to implement either and submit a pull request.

Also, how could it be made easier to find this api for other extension developers, or is this issue enough?

[HighCommander4]

I had in mind the second approach. I think this is more flexible, as it gives us the ability to grow the extension's public API over time as needed, rather than being limited to commands.

Also, how could it be made easier to find this api for other extension developers, or is this issue enough?

We should document the extension's public API once it has one. Not sure where yet; https://clangd.llvm.org/ is a possibility, but I'm not sure if having client-specific documentation there is appropriate. A wiki page in this repo is another option.

[meisZWFLZ]

I added the ExtensionAPI type and ClangdContext.registerAPIFeature() method, such that any future APIs may be implemented by adding an entry to ExtensionAPI and registering it with the ClangdContext.

Two things I noticed while implementing this with my consuming extension:

Type Library?

I need to use the types in this repository; is there a way to export and import these types in a way that makes sense (and isn't copying and pasting)

onDidModifyAST Event

If a file has been changed, my extension re-retrieves the AST and reprocesses it. This seems to be incredibly expensive, in comparison to subscribing to an "onDIdModifyAST" event which provides the "diff". Is it possible to implement this in the clangd extension or would this require modifying clangd itself in someway?

Documentation

Would api documentation be appropriately placed in the extension-api.ts file (perhaps with a link to it from DEVELOPING.md)

[HighCommander4]

Type Library?

I need to use the types in this repository; is there a way to export and import these types in a way that makes sense (and isn't copying and pasting)

I'm not very well-versed in the Typescript world, but one thing that occurs to me is publishing an additional @types/vscode-clangd package from this repository, containing a .d.ts file containing the type definitions used by the public API?

onDidModifyAST Event

If a file has been changed, my extension re-retrieves the AST and reprocesses it. This seems to be incredibly expensive, in comparison to subscribing to an "onDIdModifyAST" event which provides the "diff". Is it possible to implement this in the clangd extension or would this require modifying clangd itself in someway?

I guess it depends which part of the operation you're concerned about being expensive.

Clangd reparses the file on each modification (but not the contents of #includes at the top; it has an optimization where that top part of the file, called the "preamble", is preserved between parses, assuming the edit doesn't touch the preamble), so that expense is paid regardless.

vscode-clangd could be made to retain a copy of the AST, compute a diff, and return a diff via the public API, which would cut down the amount of data transmitted over the public API, but not the amount of data transmitted over LSP from server to client.

If you want to cut down on the amount of data transmitted from server to client, then clangd itself would have to be modified to compute the diff on the server side.

My suggestion would be to do the simplest thing first, then to optimize the performance where it proves to be an issue in practice.

Documentation

Would api documentation be appropriately placed in the extension-api.ts file (perhaps with a link to it from DEVELOPING.md)

Having some documentation comments in the code sounds good, but it might be nice to consider writing a Wiki page as well (perhaps with an example usage too).

[meisZWFLZ]

Type library

I believe the package should be placed solely in the DefinitelyTyped repo (To add a @types package, one must submit it to that repo, and it wouldn't make sense to have two locations for the library).

Documentation

I found this tool to hopefully produce Markdown from the type library's index.d.ts. This will then be served up by GitHub pages, possibly alongside guides and examples, which would be in another branch dedicated to GitHub pages.

To do

• [ ] Type Library
  • [ ] Produce index.d.ts
  • [ ] Add documentation to index.d.ts
  • [ ] README.md
  • [ ] Publish Package (@types/vscode-clangd-api) to DefinitelyTyped
  • [ ] Remove types from extension source code and instead import them from new package
• [ ] Documentation
  • [ ] Parse index.d.ts into webcontent (Automate?)
  • [ ] Working demo of GitHub Pages containing index.d.ts
  • [ ] Guides
  • [ ] Examples
  • [ ] Styling??

[meisZWFLZ]

Sorry for the super late reply (doing other things and procrastinating this)

Type Library

I asked in the Definitely Typed discord server about whether this would be an appropriate use of Definitely Typed to which they replied that Definitely Typed wouldn't be the place for such a library.

I also found an example of what another extension does to handle this: CMake. CMake handles this with another repository with an api.ts containing a bunch of types and a function to retrieve the api. From api.ts, a .d.ts and .js file are generated.

I believe that we should follow CMake's lead here, but idk how I would go about making a new clangd repo or making a npm package with the @clangd context. It would be possible to make the api package a subdirectory of this repo, but if possible I'd prefer to make a separate repo

Documentation

I don't think its worth it to add documentation outside of within the .d.ts, especially since there will likely be only a few people using this.

Questions

• How do I make / talk to someone to make a new api repo?
• How do I make a @clangd scoped npm package?
• Is no documentation besides being in the type file ok?

[HighCommander4]

It would be possible to make the api package a subdirectory of this repo, but if possible I'd prefer to make a separate repo

Could you elaborate on the reason for this preference?

How do I make a @clangd scoped npm package?

I'm less familiar with the npm side of things. Is there more to this than just a naming convention of having the package name be @clangd/something?

[meisZWFLZ]

Could you elaborate on the reason for this preference?

It will make it simpler to have it as a separate repo, especially when using workflows (publish, and maybe ci). If they're in the same repo, I'd have to make sure the api library is not included in the vscode extension and instead is an npm dependency. It may also confuse developers who try to include the types from the api directory rather than from the npm package.

Is there more to this than just a naming convention of having the package name be @clangd/something?

For me to be able to publish/modify a @clangd-scoped package, I'd have to be given write permissions. (This is why DefinitelyTyped exists, since you can't make your own @types-scoped package)

So what I'd like to know is who should I reach out to, to publish an npm package and perhaps make a new repo under the clangd org.

[HighCommander4]

Apologies, I forgot to reply to this.

So what I'd like to know is who should I reach out to, to publish an npm package and perhaps make a new repo under the clangd org.

For npm, I do not have the relevant permissions to publish a @clangd-scoped package. Looking at https://www.npmjs.com/package/@clangd/install under "Collaborators", it seems the people with this permission are @sam-mccall and @kadircet.

For creating a new repo, technically I have the permission to do that, but since it's kind of a significant operation, I'd prefer to hear Sam or Kadir's opinion on that (vs. just using a subdirectory of the current repo) as well.

[HighCommander4]

I'm going to close this as a duplicate of the recently merged https://github.com/clangd/vscode-clangd/pull/575, since I believe that PR basically satisfies the use case described here.