feat request: sync automation for tree-sitter-python updates

[madiedgar]

NOTE: This repository is built on top of tree-sitter/tree-sitter-python.

To stay updated with the tree-sitter official updates for the repositories, we can create a script that will sort through changes in the corpus/*.txt folder attributes. For the most part the txt files follow a similar pattern to what's below:

=====================================
Await expressions
=====================================

testawaitlegesher i(j, 5)
testreturnlegesher testawaitlegesher i(j, 5)

---

(module
  (expression_statement
    (await (call
      (identifier)
      (argument_list (identifier) (integer)))))
  (return_statement
    (expression_list
      (await (call
        (identifier)
        (argument_list (identifier) (integer)))))))

The difficult part is that there will always be differences with the files, as the legesher keywords will make it seem like there are a lot of changes that need to be made. However, the main portion of the file we want to keep an eye out for is the Abstract Syntax Tree description after the ---. If there is a change to this AST, then we can better determine if there are updates to the explicit lines on code above that contain the legesher keywords.

This might be a great issue to revisit after #11 is completed! ⚡️

[Aadithyavarma]

Hi, This one looks interesting!. Can someone give more info about this so that I can prep now and start it as soon as #11 is completed. Also I am going through the repo now.

[madiedgar]

Absolutely @Aadithyavarma, I'll refine the issue description and assign you to this issue 👍 Thank you!

[Aadithyavarma]

Let me just brief what I understood from this. Please correct me if required.

There are different txt files in corpus folder. The task is to check if the AST(Abstract Syntax Tree) is matching the code above ---. For example,

testawaitlegesher i(j, 5) is matching

(module
  (expression_statement
    (await (call
      (identifier)
      (argument_list (identifier) (integer))))))

if it is not matching, then do an action. Is this the requirement?

Also, I am seeing some scripts in examples folder. Do I add this to that?

P.S: This is my first issue so please guide me on how to get started.

[Aadithyavarma]

Quick question, not sure if this is the correct place to post, please do tell me where to post this otherwise. I am trying to install language-legesher-python package in Atom but I am getting the below error:

npm ERR! code E400
npm ERR! 400 Bad Request: https://www.atom.io/api/packages/language-legesher-python/versions/0.56.1/tarball

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\aadit\.atom\.apm\_logs\2019-10-05T16_04_11_069Z-debug.log

Part of the log

7 http fetch GET 400 https://www.atom.io/api/packages/language-legesher-python/versions/0.56.1/tarball 1102ms
8 silly fetchPackageMetaData error for https://www.atom.io/api/packages/language-legesher-python/versions/0.56.1/tarball 400 Bad Request: https://www.atom.io/api/packages/language-legesher-python/versions/0.56.1/tarball
9 timing stage:rollbackFailedOptional Completed in 4ms
10 timing stage:runTopLevelLifecycles Completed in 1294ms
11 verbose stack Error: 400 Bad Request: https://www.atom.io/api/packages/language-legesher-python/versions/0.56.1/tarball

Can someone help me?

[madiedgar]

Looking into this @Aadithyavarma ^^

[madiedgar]

You're doing great @Aadithyavarma! In response to your first comment:

There are different txt files in corpus folder. The task is to check if the AST(Abstract Syntax Tree) is matching the code above ---.

Almost - the task is to sync tree-sitter-legesher-python with any updates from tree-sitter-python. We must check if the tree-sitter-legesher-python's AST is matching the AST in tree-sitter-python.

Using your example above, the line testawaitlegesher i(j, 5) in this repo would look like await i(j, 5) in tree-sitter-python. This different in the keyword would throw a "difference", but it's not necessarily a update to the AST.

The way I would approach it - which if there is a better way, please go for it! - would be to compare the AST below each of the --- from this repository to tree-sitter-python's. If it's not matching, then we know there are updates to the grammar.

The examples/ folder will eventually need to have the same thing done, but I didn't want to overwhelm whoever was doing this! If you want to tackle this as well, be my guest! 😄

[Aadithyavarma]

Hi,

Let me just repeat it back to be clear,

• Compare the ASTs in tree-sitter-legesher-python and tree-sitter-python

• If any one of the AST does not match, sync the changes in tree-sitter-python to tree-sitter-legesher-python

Below is how I plan to tackle this,

• I can read the files in corpus folder in tree-sitter-python, using requests module in Python.

• That I can compare with the files in corpus folder in tree-sitter-legesher-python and check if the AST matches.

• If it does not, then sync the changes in tree-sitter-python to tree-sitter-legesher-python

I hope this is the requirement?

Also I am still not able to install language-legesher-python package in Atom. I am still getting the same error as my above comment. Any help in this would be appreciated.

[Aadithyavarma]

Updates:

• I manually downloaded tree-sitter-legesher-python and tree-sitter-pythonand compared the files in corpus folder.

• The AST is matching for every grammar in all the txt files.

• The end goal would be that tree-sitter-legesher-python would be updated automatically if the AST is not matching with the AST in tree-sitter-python

Is this correct @madiedgar ? Please help me understand the problem statement if otherwise. Also need help in installing the package in Atom.

[madiedgar]

Thanks so much @Aadithyavarma for your work on this 👍 You have the right idea! Let me look into the atom package problem today, I'll get back to you when it's resolved 🤞. 😄

[madiedgar]

Hey @Aadithyavarma, there seems to be an issue with the language-legesher-python atom package syncing with the github releases. Will keep you updated on the progress 👍

[Aadithyavarma]

Hi @madiedgar,

Any updates on this?

[madiedgar]

Yes and no @Aadithyavarma 🙊!

There was a bigger problem with the language-legesher-python package which is being taken care of in this issue in Atom's repo. This would be why you might be experiencing issues with this package.

Good news though, @wescran actually made an incredible contribution to create an issue every time there is an update in tree-sitter's tree-sitter-python repository.

Wish I had more information at this time ☹️!

[madiedgar]

Hey @Aadithyavarma! Thank you so much for your patience on this. We're still waiting for the issue to be resolved with Atom for the language-legesher-python package error, but I can give you a little update.

@wescran recently made adjustments to our GitHub Actions to create an issue every time there is a commit on the tree-sitter-python repository. This might better inform your efforts.

Our next big initiative within this repo is to template out the legesher keywords with ones from the desired dialect (french, arabic, etc.). I feel that a lot of the headway we'll make in this new feature^ will give us a better framework to continue this specific issue in comparing with tree-sitter-python.

Excited to work together in the future!