github-readme-stats
github-readme-stats copied to clipboard
anuraghazra
Ranking System v2
Hello @anuraghazra :wave:
As #883, #455 and #1029, I've noticed that the function calculateRank doesn't work as expected.
The first problem is the use of a normal distribution instead of an exponential distribution, as pointed out by #960 . The second is that each metric has the same weight in the score, while some are clearly better indicators (e.g. stars). I've corrected both problems in this PR.
If you want to try it out before merging, I've hosted this version on Vercel.
Importantly, in #960, being extremely good in a single metric (e.g. 100k commits) while being terrible elsewhere still gets an "S+" rank. In my implementation, in order to get "S+", you need to be outstanding everywhere. However, you can still get "A+" or "S" if a single metric is poor (e.g. contributions or issues) but you have a lot of stars and followers.
@francois-rozet is attempting to deploy a commit to the github readme stats Team on Vercel.
A member of the Team first needs to authorize it.
![vercel[bot] avatar](https://avatars.githubusercontent.com/in/8329?v=4 "Published by a pub.dev verified publisher"){kind=small}
So in #960 seems like linustorvals is getting S+ rank which is good, since atleast we know a baseline that there are people who can get S+ but in this PR your changes seems to stretch the stats maybe a little bit too far?
Hi @anuraghazra 👋 I've edited the weights such that Linus Torvalds is now S+.
By the way S+ is top 2.5%, S 10%, A+ 25%, A 50% and B+ 75% and B 100%.
o in #960 seems like linustorvals is getting S+ rank which is good, since atleast we know a baseline that there are people who can get S+ but in this PR your changes seems to stretch the stats maybe a little bit too far?
thought it was great, in the current algorithm most are in the A+, even accounts with less than 5 commits and no stars get this rank, due to the huge amount of registered guests. and accounts with numerous commits, repositories and pull requests also remain with the same rank as this one.
By the way S+ is top 2.5%, S 10%, A+ 25%, A 50% and B+ 75% and B 100%.
This ratio seems well balanced.
This is one of the good ways of modeling the data to exponential distribution. I frequently work with datasets and most of them don't fit into normal distribution (I think the current ranking depends on fitting the data to a normal distribution).
As we currently have no idea on the whole data i.e, meta-data for all the github users with respect to commits, prs, stargazers etc..,. It would be difficult to obtain right parameter values and weights for the model to explain the distribution,
For fun I've tried to see what's the minimum count(MC) required for only one type of parameter (considering others to be zero count) to achieve a given rank.
| Rank | threshold | MC - Commits | MC - Prs | MC-Followers | MC-Stars |
|---|---|---|---|---|---|
| S+ | 2.5% | None | None | None | None |
| S+ | 10% | None | None | None | None |
| A+ | 25% | None | None | None | None |
| A | 50% | None | None | None | 300 |
| B+ | 75% | None | None | None | 83 |
| B | 100% | None | None | None | None |
Interestingly the weights are so strict that one can achieve only upto Rank A who has only a minimum of 300 stargazers.
I'll try to share more stats based on combinations of parameters.
Hello @Enforcer007, I'm not sure to understand what you want to say.
Do you think it is bad that one cannot get S+ (or even A+ in fact) by maximizing a single metric ?
Hi @francois-rozet,
The weights you used is a good start, as it restricts users to go higher in ranks who only focus on one kind type of parameter.
Just testing out few numbers with the ranking system
https://codesandbox.io/s/calculaterank-9nj6d?file=/src/index.js
One scenario which I'm seeing is that if someone is overall decent on all stats but has 0 stars gained they will always get B+ rank which isn't ideal.
See codesandbox for more info.
Hello @anuraghazra,
You seem to have a really warped view of what is a "decent" number of stars. Most users don't have any stars, let alone more than 100. "just a few thousand stars" is already incredibly outstanding. For instance, here you see that only 5000 users have more than 2750 stars and there are over 40 millions active users on GitHub.
Same for the number of commits, followers, PRs, etc. (2500 commits or 750 followers is not average at all). The values I have chosen were calculated (by hand) as the averages over the stargazers of one of my repos (francois-rozet/piqa). The stargazers of this repository (there are ~30k) would be much more representative. Since you are familiar with GitHub's API it shouldn't be too hard for you to gather the metrics of these users and then compute the averages.
You seem to have a really warped view of what is a "decent" number of stars. Most users don't have any stars, let alone more than 100
Exactly, thats why I said if someone is overall decent on all stats but has 0 stars gained they will always get B+ rank which is really unfair.
I'm talking about the first case
// Average user but with 0 stars will always get B+
console.log(
calculateRank({
totalCommits: 1500,
followers: 500,
prs: 150,
issues: 65,
stargazers: 0
})
);
You can reduce the weight of stars if you want. STARS_WEIGHT = 0.5 seems balanced.
Could this be published and enabled with an extra URL parameter - or with a 'v2' as part of the URL? This would not cause breaking changes for anyone, and we could carry on from there. 🚀 🌔
@francois-rozet Thanks a lot for your pull request! I however think the rank calculation should also take the number of contributions into account (see https://github.com/anuraghazra/github-readme-stats/pull/960#issuecomment-963171112).
@francois-rozet I reviewed your pull request and found one small bug in the tests. When that is fixed, I think it should be fine to be merged into the master branch.
Hello @anuraghazra, I have made this PR more than one year ago. To be honest, I find it disrespectful that you ignore all PRs fixing the rank calculation, both for PR authors and users who have been asking you to fix this problem for years. If you don't want to change the rank calculation, just say it on the README, but don't let people put efforts into fixing it for nothing.
If you do consider changing the rank calculation, please make a move.
There are only two PRs that propose good systems: #960 and #1186. They are similar, but in #960, being extremely good in a single metric (e.g. 100k commits) while being terrible elsewhere still gets an "S+" rank. In #1186, in order to get "S+", you need to be outstanding everywhere. However, you can still get "A+" or "S" if a single metric is poor (e.g. contributions or issues) but you have a lot of stars and followers.
I feel that #1186 is more balanced and both @rickstaa and @GaryStriving have approved the changes.
@francois-rozet, I understand your frustration. Unfortunately, @anuraghazra and I maintain this repo in our free time, so merging PR might go slightly slower than in a company repo. I, however, understand that one year is a bit long. 😅 I think my proposal in #1867 will speed up the merging process since it will allow us to include more maintainers since an organization gives more detailed permission control.
Having that said. The main difficulty with #1186 is that #960 addresses the same issue. As stated in the past, my preference is with #1186, but @anuraghazra hasn't yet stated an opinion. I am happy to merge #1186, but I would love a second opinion from @anuraghazra since it regards the core codebase. Maybe let's give @anuraghazra 30 days to respond, after which I will merge it. 🤔 We can always revert it back to the old version or #960. :upside_down_face:
@francois-rozet Since last month, we required prettier to be run on each PR (see https://github.com/anuraghazra/github-readme-stats/actions/runs/3111026424/jobs/5042849646). I created https://github.com/francois-rozet/github-readme-stats/pull/2 to fix the errors thrown by the prettier check. You can, of course, also run prettier yourself. Whatever you prefer. 🤓
Codecov Report
Patch coverage: 100.00% and project coverage change: -0.07 :warning:
Comparison is base (
7ec1a76) 97.53% compared to head (8d8d972) 97.47%.
:exclamation: Current head 8d8d972 differs from pull request most recent head 53005a5. Consider uploading reports for the commit 53005a5 to get more accurate results
Additional details and impacted files
@@ Coverage Diff @@
## master #1186 +/- ##
==========================================
- Coverage 97.53% 97.47% -0.07%
==========================================
Files 24 24
Lines 4630 4594 -36
Branches 423 426 +3
==========================================
- Hits 4516 4478 -38
- Misses 112 115 +3
+ Partials 2 1 -1
| Impacted Files | Coverage Δ | |
|---|---|---|
| src/calculateRank.js | 100.00% <100.00%> (+0.95%) |
:arrow_up: |
| src/fetchers/stats-fetcher.js | 91.94% <100.00%> (-1.18%) |
:arrow_down: |
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hello @rickstaa, it has been a bit more than a month. Is it finally time to merge? 🙏
Hello @rickstaa, it has been a bit more than a month. Is it finally time to merge? pray
@francois-rozet I think we can merge this if we set STARS_WEIGHTS to 0.75 or 0.5. @anuraghazra, what do you think?
Star weight 1.0
Star weight 1.0 & include_all_commits=true
@francois-rozet since this is one of our most requested features, I think it is essential it gets merged (see https://github.com/anuraghazra/github-readme-stats/issues/1935). Sadly, it affects everything that uses the public instance, so I would really like @anuraghazra opinion on it.
On https://github.com/anuraghazra/github-readme-stats/pull/960#issuecomment-914357213 (Sep 7, 2021) @anuraghazra wrote:
Will consider both of the PRs, and release it under experimental flags ?enable_experiments=new_ranking_system to get feedback from the community and roll out the change gradually.
This would not affect production as it would be offered through an experimental flag.
I maintain open-source projects and know it's annoying to get bugged on un-merged PRs but it's been 549 days: 1.5 years to the day and counting.
क्यों 🤷
On #960 (comment) (Sep 7, 2021) @anuraghazra wrote:
Will consider both of the PRs, and release it under experimental flags ?enable_experiments=new_ranking_system to get feedback from the community and roll out the change gradually.
This would not affect production as it would be offered through an experimental flag.
I maintain open-source projects and know it's annoying to get bugged on un-merged PRs but it's been 549 days: 1.5 years to the day and counting.
क्यों 🤷
My preferences still goes with this pull request instead. It is more balanced. I however still think star weight has to be updated. I didn't merge it since I wanted one last opinion of @anuraghazra. Additionally, I still think https://github.com/anuraghazra/github-readme-stats/issues/1867 might improve maintainability of the project. Currently, it is me and @anuraghazra who maintain it, but the user base has grown a lot since I became a collaborator.