Allow markdown/myst-markdown code blocks to be executed in interactive window.

[amichuda]

Feature: Notebook Editor, Interactive Window, Python Editor cells

Description

Right now, we can only run the interactive window with a standard .py file with markdown commented out. It would be useful to have a way to associate the interactive window with a standard markdown file with code cells, so that those code cells can be run even though it's not a .py file.

For example:

<<---- Begin myfile.md

My title

Some text I'm writing

import this

print("hello")

------->> End myfile.md

The code cell in this case would have the "Run Cell" prompts appear and you can send that text to the interactive window, even though it's not a .py file.

This would be useful for those that use reproducible workflows with pweave or codebraid to be able to run their code before converting to PDF.

Microsoft Data Science for VS Code Engineering Team: @rchiodo, @IanMatthewHuff, @DavidKutu, @DonJayamanne, @greazer, @joyceerhl

[rchiodo]

This would require something that understands md files to add code lenses (and/or the commands) to send code to the interactive window.

We could add an extensibility point to the python extension to allow other extensions to do this (or we could add the code lenses to md, but that doesn't scale so well).

[amichuda]

Adding an extensibility point would make it possible, but then a separate extension would have to be written to implement it?

I can see what you're saying about the second. In any case, count me in on any input/help you need. This would improve my workflow in academic work CONSIDERABLY.

[amichuda]

Hi! I hope you had a great new year. I wanted to circle back and see if this was anyone's radar at all, but I understand if the team is busy working on other things!

[rchiodo]

@amichuda it's not on our radar at the moment, but we welcome PRs if you want to take a crack at it. I think it would be something like a setting that determines regexes for cells for different languages.

Right now we have these two settings:

"jupyter.codeRegularExpression": "^(#\\s*%%|#\\s*\\<codecell\\>|#\\s*In\\[\\d*?\\]|#\\s*In\\[ \\])"
"jupyter.markdownRegularExpression": "^(#\\s*%%\\s*\\[markdown\\]|#\\s*\\<markdowncell\\>)"

It think if we wanted to make it more generic, we'd need one of these settings per language. Maybe something like we do for the variable queries:

"jupyter.codeLensExpressions" : [
   { 
     "extension": ".py",
     "language": "python",
     "codeExpression": "^(#\\s*%%|#\\s*\\<codecell\\>|#\\s*In\\[\\d*?\\]|#\\s*In\\[ \\])",
     "markdownExpression": "^(#\\s*%%\\s*\\[markdown\\]|#\\s*\\<markdowncell\\>)"
   },
   { 
     "extension": ".md",
     "language": "markdown",
     "codeExpression": "```",
     "markdownExpression": ""
   }
]

[amichuda]

Sorry, I'm new to the typescript/VS code extension game, so I'm trying to make sense of this.

From what I'm thinking, would this involve adding options to the package.json and then changing the regex options in src/client/datascience/cellMatcher.ts. Does that sound right?

What I'm not sure about though is whether opening an md file, even with these new code lenses would allow both markdown intellisense/autocomplete and python intellisense/autocomplete in one file?

I think that might be the real crux of the issue, as wouldn't it be possible to add another OR clause into markdownRegularExpression and codeRegularExpression to allow for the ``` syntax?

[rchiodo]

What I'm not sure about though is whether opening an md file, even with these new code lenses would allow both markdown intellisense/autocomplete and python intellisense/autocomplete in one file?

Intellisense and autocomplete would not be impacted by these code lenses. In fact the file would only support markdown intellisense. Perhaps intellisense isn't clear in this context. Intellisense generally means hovering over text in a document or typing in a document and getting a popup.

The ask you have here is really about this:

Just the 'Run cell' code lenses at the top of each cell.

If you're looking for intellisense support and more in the python blocks, that's a whole different issue. It would probably require the language server to understand markdown files.

However you're correct in that the cellMatcher.ts file would have to be changed for code lenses. It would be there that you'd check the language of a file and then apply the appropriate regex from the user's settings.json:

"jupyter.codeLensExpressions" : [
   { 
     "extension": ".py",
     "language": "python",
     "codeExpression": "^(#\\s*%%|#\\s*\\<codecell\\>|#\\s*In\\[\\d*?\\]|#\\s*In\\[ \\])",
     "markdownExpression": "^(#\\s*%%\\s*\\[markdown\\]|#\\s*\\<markdowncell\\>)"
   },
   { 
     "extension": ".md",
     "language": "markdown",
     "codeExpression": "```",
     "markdownExpression": ""
   }
]

For example, if the user's file is a .py file, you'd apply the python codeExpression regex to search for code cells and the markdownExpression to search for markdown cells.

If the user's file was a .md file, you'd apply the codeExpression regex for the .md section.

[amichuda]

Great, thank you! I think I'm on the right track, and excuse the noob question... but is there an easy way to get a string of the filename to check for the extension?

[rchiodo]

Great, thank you! I think I'm on the right track, and excuse the noob question... but is there an easy way to get a string of the filename to check for the extension?

If you have the TextDocument object, it has a fileName property (and a URI property).

I recommend reading up on the VS code API here: https://code.visualstudio.com/api/references/vscode-api

We wrap a bunch of that in our code but the concepts are pretty much the same.

The property I'm mentioning is here: https://code.visualstudio.com/api/references/vscode-api#TextDocument

[amichuda]

Perfect! Thank you for that. Now that I have a clearer idea of how the software is set-up, I wanted to run this by you. As it stands, to do the language check in https://github.com/microsoft/vscode-jupyter/blob/main/src/client/datascience/cellMatcher.ts, wouldn't that require adding language specific properties in IJupyterSettings, https://github.com/microsoft/vscode-jupyter/blob/2ab1a20fda9d042828bf3f36f291cdf7387c7745/src/client/common/types.ts#L110 , and then all the way through to JupyterSettings. Would it make more sense to do the language check in https://github.com/microsoft/vscode-jupyter/blob/main/src/client/common/configuration/service.ts and apply markdownRegularExpression and codeRegularExpression based on the language? Would that make it easier to extend for the user?

Or is the idea here that the extension should implement language by language and JupyterSettings should have markdown, python (and any other languages later) specific entries?

[rchiodo]

Would it make more sense to do the language check in https://github.com/microsoft/vscode-jupyter/blob/main/src/client/common/configuration/service.ts and apply markdownRegularExpression and codeRegularExpression based on the language? Would that make it easier to extend for the user?

No the configuration service doesn't know what language is current. That changes per file.

Or is the idea here that the extension should implement language by language and JupyterSettings should have markdown, python (and any other languages later) specific entries?

Yes this is what I was suggesting above. We should have a per language setting for these regex patterns. We do this already for variable query setting.

[amichuda]

So I think i have a working prototype of this, but there are two issues (again, apologies for the noob questions)

• Some tests that were meant to test the regular expressions now don't work because they call markdownRegularExpressions and codeRegularExpressions which don't exists because they're now part of the codeLensExpressions array. Should I change the tests or leave them be?
• When switching to python everything works as it should; the "Run Cell" code lenses show up and creating new cells is possible. - When switching to a markdown file, however, the right-click menu to Run the cells works, but the code lenses don't show up and there is no creation of new cells. Could this be because of hardcoded logic to only run things when the file is python? I can think of various places this is the case, from keyboard commands (editorLangId == python) and the PYTHON and PYTHON_ALL_FILES variables in constants.ts. Is that possible?

[rchiodo]

Some tests that were meant to test the regular expressions now don't work because they call markdownRegularExpressions and codeRegularExpressions which don't exists because they're now part of the codeLensExpressions array. Should I change the tests or leave them be?

You should fix the tests. We'd like to have tests covering your new addition as well as our old stuff.

When switching to python everything works as it should; the "Run Cell" code lenses show up and creating new cells is possible. - When switching to a markdown file, however, the right-click menu to Run the cells works, but the code lenses don't show up and there is no creation of new cells. Could this be because of hardcoded logic to only run things when the file is python? I can think of various places this is the case, from keyboard commands (editorLangId == python) and the PYTHON and PYTHON_ALL_FILES variables in constants.ts. Is that possible?

Yeah sorry, I forgot to mention you need to register a code lens provider for the markdown extension too. I think it should also be based on the new settings:

"jupyter.codeLensExpressions" : [
   { 
     "extension": ".py",
     "language": "python",
     "codeExpression": "^(#\\s*%%|#\\s*\\<codecell\\>|#\\s*In\\[\\d*?\\]|#\\s*In\\[ \\])",
     "markdownExpression": "^(#\\s*%%\\s*\\[markdown\\]|#\\s*\\<markdowncell\\>)"
   },
   { 
     "extension": ".md",
     "language": "markdown",
     "codeExpression": "```",
     "markdownExpression": ""
   }
]

Meaning the registration should iterate through the array here and specify a code lens provider for each entry.

[amichuda]

Okay, understood. Can you point me in the right direction on how to go about registering the new provider? I take it that the one being used now is CodeLensProvider from the vscode api?

[rchiodo]

The code lens provider looks like it's registered here: https://github.com/microsoft/vscode-jupyter/blob/b368ea9326c13d6497e0f070d89d810d4ff86416/src/client/datascience/datascience.ts#L44

Looking at that code, I think it should probably be restructured but I think we can do that after you submit. I would just do something like this:

   const codeLensExpressions = this.configuration.getSettings(undefined).codeLensExpresions;
   codeLensExpresions.forEach(c => {
       this.extensionContext.subscriptions.push(
            vscode.languages.registerCodeLensProvider({ language: c.language }, this.dataScienceCodeLensProvider)
        );       
   }

I think that would work. It will register a code lens provider for every language specified in the settings.

[amunger]

closing in favor of https://github.com/microsoft/vscode-jupyter/issues/273