Review systems to automate code review

[gpsaggese]

We want to find a way to do the first line of defense for our PRs

Some ideas

• https://docs.github.com/en/copilot/using-github-copilot/code-review/using-copilot-code-review
• graphite.dev
• https://techcrunch.com/2019/08/06/deepcode-gets-4m-to-feed-its-ai-powered-code-review-tool/
• https://techcrunch.com/2024/08/15/coderabbit-raises-16m-to-bring-ai-to-code-reviews/

[Shaunak01]

Some info: https://docs.google.com/document/d/1cs50rwqxhRzQLt41brcPoQs5xL0QYOEFDcKHJB0NUo8/edit?tab=t.0#heading=h.rv0yjq809l2h

[gpsaggese]

Okay, let's start with Copilot and then try graphite. What's the next step? @Shayawnn, do you have the permissions to enable it or is it already on?

[gpsaggese]

@Shaunak01 can you change the name of the gdoc as HelpersTaskXYZ_... same as a PR. This is how we keep the gdoc in sync with the tasks

[gpsaggese]

We want a "bot" to add comments to the PR related to:

1. stylistic stuff (based on our code style)
2. look for mistakes ... to automate the "first" PR

You can also set it up on your side and we do a demo to decide which one is best and why

[gpsaggese]

This is in progress right? If so, let's mark it

[Shaunak01]

@gpsaggese Codecov has sent request to cmamp to get access. Can you check

[gpsaggese]

Accepted. We can start from //helpers since the regressions are faster

[Shaunak01]

@gpsaggese

https://github.com/github-copilot/purchase

When you activate copilot pro for the organization we should be able to use Copilot for automated code reviews (we need to add as a reviewer if not set as default) and other features as well. No setup required from git actions or code base.

[gpsaggese]

@gpsaggese

• [x] Read gdoc
• [x] Enable copilot

[gpsaggese]

Good work here.

Next steps are:

1. Convert the doc into a markdown with our script
   • I would remove the tables since they are difficult to render / edit in md and put them in bullet lists
2. Instead of the pic, for pricing let's summarize
   • Link to the page
   • Cost per user, 10 users, 20 users
3. Can we install the top 2-3 and give it a try (is there a free offer, even just for me)

@sonniki this can be passed to interns under @Shaunak01 supervision if you want

[Shaunak01]

Sure. We can also have other tasks for (docs -> md) if we want tasks for interns. @sonniki LMK thanks

[sonniki]

Sure. We can also have other tasks for (docs -> md) if we want tasks for interns. @sonniki LMK thanks

Yes please, feel free to file issues (one per each doc to be converted), and tag me there as FYI

[sonniki]

@aangelo9 Specs:

• Convert gdoc to a markdown in docs/, following instructions in https://github.com/causify-ai/helpers/issues/393#issuecomment-2819660255
• Provide a step-by-step plan on what we need to do to start experimenting with GitHub Copilot Code Review
  • Ideally, we want to be able to guide the automatic reviewer to our docs/stylistic requirements so that it "knows" what to look for

[aangelo9]

@Shaunak01 I have requested access for the gdoc.

[sonniki]

@Shaunak01 I have requested access for the gdoc.

Shared to quickly unblock. Do you have access now, @aangelo9 ?

[aangelo9]

@sonniki Yes, I do have access now. Thank you.

[aangelo9]

Approach to experiment with GitHub Copilot Code Review.

1. Enable GitHub Copilot and its code review settings
2. Load Markdown Guidelines into Copilot Code Review
   • Navigate to Copilot Code Review Settings
     • Go to your GitHub repo > Settings > Code & automation > Copilot > Code Review
   • Create or Edit a Guideline
     • Click Create guideline (or Edit an existing one).
     • Copy the content from your Markdown file (style guide) and paste it into the guideline text box.
   • Once saved, GitHub Copilot will use this natural-language content to guide its PR reviews.
3. Experiment Copilot's reviews to improve its comments and suggestions.
   • Create a pull request that contains code blocks that follows style conventions and code blocks that do not.
   • Assign Copilot as a reviewer.
   • If comments and suggestions are not desirable, we could:
     • Update coding guidelines to be clearer.
     • Add code examples in guidelines.
   • Repeat.

[gpsaggese]

@Shaunak01 do you have enough permissions for test driving this? Let me know if I need to do something.

BTW we can write a blog with our experience on using different review systems, so let's keep doing a good job and collecting material to make a good comparison.

[sonniki]

The gdoc content is now moved to docs/work_tools/all.code_review_tools.reference.md

[Shaunak01]

@gpsaggese I need admin permissions to see repo settings where I can enable copilot. So you will have to do this. Should not take much time

[gpsaggese]

I think it's set up already

This is what I see on a per-project basis

This is what I see at org level

When I go to a PR and I ask a review I get

Then I see

Two questions

1. How do we evaluate which one is best / more useful?
2. How to have someone set up these things and have a way for me to get a sense of how they work?

• Maybe the only solution is for @Shaunak01 to do as much legwork possible and then do a PP together to play with it

[Shaunak01]

@gpsaggese

We have Copilot enabled for our repository and it's working for PR reviews. However, Copilot is currently reviewing without knowledge of our specific coding standards from our .md style guide.

To fully activate Copilot with our rules, we need a small setup on the organization level (admin access needed):

Steps to upload our coding style guide:

1. Go to: Organization Settings → Copilot → Policies → Code Review Guidelines.
2. Click "Create Guideline" and upload our .md file (which contains naming conventions, docstring expectations, formatting rules, PR checklist, etc.). - Code style doc
3. Assign the uploaded guideline to the helpers repository.
4. Save.

After this setup, Copilot will use our project-specific rules to review PRs more intelligently and catch style issues automatically.

Once the guideline is uploaded, I will run a few PRs (good and intentionally bad ones) to evaluate Copilot’s review quality and suggest any refinements if needed.

[gpsaggese]

We can try to feed https://github.com/causify-ai/helpers/blob/master/docs/coding/all.coding_style.how_to_guide.md as it is but I'm not sure if we need to make that file more LLM-friendly. Can you experiment with starting a new file with LLM-friendly style.

I will give you more permissions.

[gpsaggese]

@sonniki can help

[gpsaggese]

I've also put together several targets in llm_transform.py that get this job done. I'll do a quick demo today so we can start using it

[gpsaggese]

Copilot is ok, but not great. I've filed

• https://github.com/causify-ai/helpers/issues/685
• https://github.com/causify-ai/helpers/issues/716
• https://github.com/causify-ai/helpers/issues/717

to use a mixed of automation and human stuff

[sonniki]

FYI @Shaunak01 I've added a doc with the ~full list of concise coding guidelines in docs/code_guidelines/all.automated_review_guidelines.reference.md As @gpsaggese mentioned, we might create our own automated PR reviewer but I think, it's still worth it to try to feed this doc into Copilot to see if it becomes better at reviewing our style

[gpsaggese]

Sure, let's give it to Copilot and see what happens. @Shaunak01 can you post where it needs to go and any link to relevant documentation for Copilot? E.g., is there a way to test a prompt off-line to help with debugging?

[Shaunak01]

https://github.com/causify-ai/helpers/issues/393#issuecomment-2837129758

[gpsaggese]

I've tried but it's very high level and only 600 chars.

I think the ai_reviewer.py approach is better. Closing this