Submission into the llvm-project

[peterwaller-arm]

This issue tracks what needs to happen for the mono-repo push. I intend to keep this message up-to-date. Please post suggestions below or edit the body of this message as appropriate.

Prerequisites:

• [x] Implement first cut of history rewriting (#854)
• [x] Contact llvm-dev and flang-dev to express intent.
• [x] Resolve question of whether to do git merge --no-ff onto the maser branch (llvm-dev reference).
  • Answer: Yes.
• [x] Have someone responsible for freezing the f18 repository (@tskeith @sscalpone).
• [ ] Nominate code owners for various parts of the flang repo and commit a CODE_OWNERS file (#881).
• [ ] Get green light.
• [ ] Finalize rewrite program.
• [ ] Complete the relicensing.
• [ ] Update the f18 docs to describe the new process for making changes.
• [ ] Decide date (~current proposal: 10:00am GMT, Mon 13th Jan; in time before 15th Jan LLVM-10 branch~ -- postponed for now, pending more LLVM integration planning).
• [ ] Ensure all F18 developers have LLVM commit access.

When the time comes:

• [ ] Close all outstanding PR's on old f18 repo. It will be the responsibility of the PR owner to raise a new review in Phabricator to continue the review in the monorepo if the PR is still live.
• [ ] Freeze submissions to flang-compiler/f18
• [ ] Contact Tom Stellard and Mike Edwards to temporarily suspend commit emails.
• [ ] Generate rewritten history
  • [ ] Pull latest f18/master and llvm-project/master
  • [ ] Run rewriting program
  • [ ] Depending on outcome of the question about git merge --no-ff, do a merge.
• [ ] Final manual sanity checking
  • Check that the name of whoever did the rewrite does not end up on the commits where they do not belong.
  • Check that the key promise of "the trees of the merge commits are the same after rewrite" holds.
  • Eyeball the history, make sure it follows on from the right point.
• [ ] git push llvm-project rewritten-history-v2-llvm-project-merge:master
• [ ] Announce merge complete on llvm-dev and flang-dev.
• [ ] Contact Tom Stellard and Mike Edwards to unsuspend commit emails.
• [ ] Proposed: Place one final commit in the flang-compiler/f18 repository telling users that the content has moved, as MLIR has done (#908).
• Future submissions now must go into llvm-project.
  • flang-compiler/f18 remains frozen for accepting future pull requests.
  • Future commit messages referencing f18 issues and pull requests should use the notation flang-compiler/f18#123 rather than simply #123.

[RichBarton-Arm]

I think some additional pre-requisites are:

• [ ] Ensure all F18 developers have LLVM commit access
• [ ] Nominate code owners for various parts of the flang repo and commit a CODE_OWNERS file

And an additional step for when the time comes

• [ ] Close all outstanding PR's on old f18 repo. It is the responsibility of the PR owner to raise a new review in Phabricator to continue the review in the monorepo if the PR is still live.

[peterwaller-arm]

Alex L posts a good argument for a merge commit on the mainline: http://lists.llvm.org/pipermail/llvm-dev/2019-December/137690.html

To summarize: this will prevent the builders from trying to build all 2,700 commits, instead they will just build the merge commit.

[klausler]

How much time will pass between the freezing of the existing f18 repository on github and the resumption of development in llvm-project?

[tskeith]

More pre-requisites:

• complete the relicensing
• update the f18 docs to describe the new process for making changes

[peterwaller-arm]

How much time will pass between the freezing of the existing f18 repository on github and the resumption of development in llvm-project?

@klausler From the technical side I don't see any need for a significant pause. We could freeze contributions last thing PST one day, and (assuming me executing) I could do the rewrite, validation and LLVM push first thing GMT, for a total "possibility to contribute" downtime measuring in a few hours, under ideal conditions and assuming good coordination. The rewrite, validation and push itself on the happy path would only require minutes..

I don't know if there is another argument for having a longer freeze.

More pre-requisites:

@tskeith I've added your suggestions above. Could you please clarify which docs need updating? I didn't see anything covering the mechanics on the flang side. Since it will be an llvm-project, I think we can just reference http://llvm.org/docs/Contributing.html#how-to-submit-a-patch for example.

[peterwaller-arm]

I'm proposing to do the rewrite, merge & push some time around 10:00am GMT, Mon 13th Jan. This would make it just in time for the scheduled LLVM 10 branch on the 15th Jan, but leaves another week of comment period when people start getting back from Christmas on the 6th Jan.

@tskeith are you happy to be responsible for freezing submissions to the f18 repository, or finding someone to do this? I guess it amounts to removing commit access to everyone, or changing who has permissions to push to the master branch to "the empty set".

I'm shortly going on leave until 6th Jan and will respond to as soon as I can thereafter.

[tskeith]

@tskeith I've added your suggestions above. Could you please clarify which docs need updating? I didn't see anything covering the mechanics on the flang side. Since it will be an llvm-project, I think we can just reference http://llvm.org/docs/Contributing.html#how-to-submit-a-patch for example.

Maybe in the main README. We just need to say somewhere not to submit PRs here any more and what to do instead.

[tskeith]

@tskeith are you happy to be responsible for freezing submissions to the f18 repository, or finding someone to do this? I guess it amounts to removing commit access to everyone, or changing who has permissions to push to the master branch to "the empty set".

Either @sscalpone or I should be able to do that.

[peterwaller-arm]

Maybe in the main README. We just need to say somewhere not to submit PRs here any more and what to do instead.

I guess this could be a single commit to the flang-compiler/f18 repository which doesn't get included in the llvm-monorepo push.

[RichBarton-Arm]

A couple more things that @kiranchandramohan and I were discussing

• What will we do wrt issue tracking once we have ported over? llvm-project has enabled github issues but they seem to be used quite sparingly compared to bugzilla. There was a thread on llvm-dev on this subject following the dev meeting which suggests a swift port over to github issues, but nothing that concludes this is the plan. Bugzilla still seems open to me. I assume we can simply start using llvm-project issues in the same way we were using F18's issues.
• What will we do with @DavidTruby's CI on F18? I think it is currently not working so I suggest we don't port it over to llvm-project until such time that it is working well.

[kiranchandramohan]

I have a few points/questions. Will the FIR changes be available before we submit to the LLVM repo? How will the current driver work with the LLVM make files? Will there be CMake changes for this? Is there a plan to send a final mail once the merge closes on what is the current status of the F18/Flang project to properly set expectations. (i.e we have parsing, semantic analysis but no code generation)

[peterwaller-arm]

Summary thoughts:

• I think we should start using the llvm-project/llvm issue tracker.
  • It seems the obvious thing to do.
  • I'll email to request an update on the plans for the issue tracker, but I would be surprised if it is going away. (Note: I'm not yet fully caught up on emails after the new-year yet, so I don't know if anything changed in the meantime).
• We should stop accepting new issues on flang-compiler/f18.
  • Issues newly opened here should be closed and the reporter redirected to the new place. It might also be possible to prevent this by tweaking permissions, but I am unsure.
• (not suggested) github supports transferring issues between repositories, but only between repositories in the same organization. So that's not possible unless f18 moves to llvm-project/f18, which I don't propose to do.
• Reminder: I'm assuming that flang-compiler/f18 stays where it is and all the old issues remain here so that they can be referenced by historical commits.

Responding to @RichBarton-Arm:

I assume we can simply start using llvm-project issues in the same way we were using F18's issues.

Yes. I don't see any impediment to this. It seems the obvious thing to do and least disruptive.

There are labels which begin A- (I assume so that they appear first when alphabetized), and we can create an A-flang label. It looks as though I have github permissions to do this and I can't see anyone objecting to making a label once we're merged. The label will make it easy to view just the flang issues.

There was a thread on llvm-dev on this subject following the dev meeting which suggests a swift port over to github issues, but nothing that concludes this is the plan.

I agree there was no concrete conclusion, but my reading of the thread is that it was warmly received and looked likely to be accepted.

The main concerns raised were how to arrange notifications so that one can subscribe to just what they're interested in. A solution which Tom Stellard proposed was that people could be notified on relevant projects according to the label, so issues labelled with the A-flang label would subscribe relevant flang contributors. I could co-ordinate to arrange that assuming things are going that way.

What will we do with @DavidTruby's CI on F18?

That's an interesting question, and something for the LLVM list I think. I'm not sure what the full impact of having a github-aware CI enabled on the flang tree will be. At a minimum it would require someone with appropriate permissions to turn it on for the llvm-project, and tweaking of the in-tree CI configuration, but I think more coordination and consideration would be required. I'm thinking in particular along the lines of "how do we prevent this getting in the way of developers not on the flang project?".

[peterwaller-arm]

Responding to @kiranchandramohan:

Will the FIR changes be available before we submit to the LLVM repo?

Not something I can speak to. I think that we should not block the LLVM repo submission on anything unless it is critical.

How will the current driver work with the LLVM make files? Will there be CMake changes for this?

As above, this will stay as it is, until it is fixed after the migration.

Is there a plan to send a final mail once the merge closes on what is the current status of the F18/Flang project to properly set expectations. (i.e we have parsing, semantic analysis but no code generation)

My intent is to announce that the merge is complete, once the above checklist is run through. The purpose of that is so that people can understand what has changed in the tree and git history. I can explain that this changes nothing in the LLVM build system for now, and the main impact is that there are more commits in the git history and a new flang directory.

I think it would be nice if someone could give a general update on the status of the project once it has landed but I think someone other than me (like @sscalpone?) might be better suited to give this.

[DavidTruby]

As regards the CI, I only ever intended the current github solution to be a stopgap until we merged and therefore could using the existing LLVM CI infrastructure. In addition the CI is still flakey even after the parsing.cc changes.

I think we should just disable the CI we have here after the merge and set up something new using the LLVM infrastructure (although this will probably be easier after our CMake changes to integrate with the monorepo build system go through).

[kiranchandramohan]

I am assuming the Readme.md file will be copied over. It will have to be mildly updated to reflect the repository location etc. We should ensure that the instructions are in a state so that someone can blindly follow the instructions and build and run tests. Will references to f18 be replaced with flang?

[DavidTruby]

I am assuming the Readme.md file will be copied over.

@peterwaller-arm has a suggestion for this in #908

[kiranchandramohan]

Issue 908 seems to be talking about the readme that remains in this repository. Here I am specifically talking about the readme that ends up in llvm-project.

[DavidTruby]

Here I am specifically talking about the readme that ends up in llvm-project.

Oh I see, sorry I misunderstood what you meant. I've opened a PR to reflect the changes I think need to be made to the README before we upstream, please comment on there if you think of anything else that needs changing!

[h-vetinari]

Has the name after merging been discussed/decided?

The posts on the llvm mailing list (last thread here) are always titled (my emphasis) "flang landing in the monorepo", while clearly talking about f18 and not flang.

My question is whether the current plan would be to have flang-compiler/f18 become llvm/f18?

Edit: I should have looked harder, sorry: the answer is no, it will be named llvm/flang.