gavel
gavel copied to clipboard
anishathalye
add multiple categories for projects and judges
This adds multiple categories. These are designed so there can be more than one judging session/category going on at the same time. nwHacks is planning on creating an overall category, a popular vote category and a category for each sponsor prize.
This is implemented by adding new models: Category, ItemCategory and AnnotatorCategory. mu, sigma_sq, prioritized, viewed and ignore have been moved into the per category models.
When a user encounters the judging interface they are asked to specify which category they want to judge for. If they only have one category available (/active) they are automatically redirected to that category.
You can also optionally set categories while importing via CSV.
The updated admin interface is a little bit rough, but workable. I think I stamped out most of the bugs I introduced, but I'd appreciate someone looking over it.
user interface
admin interface
Thanks again for the pull request. I think it'll be good to have some discussion about the design before accepting a change as fundamental as this, because once we decide to do category judging a certain way, it will be hard to change the design.
For people joining the discussion at this point, I'll summarize the implementation from this pull request as follows: this is effectively the same as running separate instances of Gavel for each category, where judges can have accounts on multiple instances (but of course this implementation has a much smoother user experience). I know some hackathons have been doing category judging this way. I'll refer to this way of implementing category judging as separate categories.
One alternative way to do category judging is to have a single overall judging pool, and then for category rankings, use relative rankings from the overall judging. This is the way HackMIT has been doing category judging. I'll refer to this as filtering categories.
Filtering categories
The method HackMIT has been using has a couple main advantages. It provides greater statistical confidence for the overall ranking with fewer judges, and it allows for reassignment of categories after judging (a feature that has come in handy at HackMIT, where we've had people submit their project to the wrong category and then come to us later to correct it).
The method has one main disadvantage: it makes the assumption that if a project X is ranked below a project Y in the "overall" category, then you also have X < Y within a category. This assumption may hold if you have categories like "best iOS project", "best web project", etc., but it may not hold if you have categories like "best design", "best use of deep learning", etc.
Separate categories
This method has the advantage that it gives good results even when different categories have very different judging criteria, which probably does happen in real hackathons. It also accommodates categories like "popular vote" where you can even have hackers vote on projects.
This method has the disadvantage that it splits judges between many different categories, so you get less statistically significant results, especially for the "overall" category. Also, you could have judges be judging multiple categories (e.g. an "overall" category and a "best ML hack" category, and they might see the same projects multiple times if they switch between categories during judging, which is a little unfortunate.
Which one is better?
Of the properties mentioned before, I think these are the most important ones:
- Providing high statistical confidence without requiring a lot of judges / not "wasting" votes
- Giving good results even when categories have very different judging criteria
And I think these features are nice to have, but not essential:
- Allowing for a "popular vote" category with hackers as judges
- Allowing for changing a project's categories in the middle of judging
Neither separate categories nor filtering categories gives us all the properties that I think are most important, so perhaps we need some hybrid of the two solutions. Do you have any ideas on how we could do this?
I'll also spend some time thinking about this, and I'll also try to get feedback from other hackathons that have used Gavel for category judging to see if their organizers have any feedback or ideas.
Thanks to @patins, @michaelsilver, and @jlin816 from the HackMIT team, who gave feedback on this pull request that was synthesized into this reply.
Filtering categories definitely seems like an important feature for certain usecases.
nwHacks 2018 just finished up today and we used the code in this PR. It works quite well for our use case. Most of our categories have different criteria, so it doesn't make sense to have one global ranking. In addition, submitted projects can be in multiple categories with different criteria so filtering doesn't really make much sense.
We had a category for Overall, Popular Vote and then one for each individual sponsor prize. Overall and Popular Vote had similar judging criteria where the individual sponsor prizes had arbitrary sponsor criteria. We didn't have any issues with number of judges except for one sponsor using it incorrectly. For the most part judges only were judging for a single category.
We did have some issues with people forgetting to add specific sponsor prizes so we added them to a new category mid judging without any issues (in fact one of those late additions ended up winning a prize).
Filtering categories doesn't really seem all that difficult to implement. It honestly seems easy enough to do that as an external second step. Just dump the entire listing for a "separate category" and then filter for the specific "filter category".
One approach might be to add "tags" to each project that are distinct from the separate categories. Under the category view you could then have an option to filter by a specific tag. This gives the desired behavior with minimal code changes.
Hmm, that makes sense. I talked this over with the HackMIT team, and nobody seemed to have any better ideas than the current proposal. So I think we should be able to accept this patch in more-or-less its current form.
I want to wait until I have time to properly review the code and test it out myself before merging, so it may be a week or two till this is merged.
Did this ever get merged? Or was this functionality added some other way? @anishathalye
Uh not yet, I lost track of it. I'll try to take a look at it soon. Got a work deadline coming up next week, should have a little bit of free time after that to review open issues and PRs for the projects I maintain.
Sounds good. We used it at SpartaHack V and I figured out a way to use it for multiple categories without making any changes to code using containers. I am writing a medium article about it. I can send to you if you think that would help.
Give me a chance to review it before you invest any more of your time. Hopefully I'll have time to look at it next week.
If you could provide feedback on the PR, that would be appreciated. I will briefly test it myself later as well.
Yeah, once you guys take a look at it I'll try rebasing + resolving any comments you have. Not sure how far master has drifted but should be fine.
On Wed, Jan 23, 2019, 08:58 Anish Athalye [email protected] wrote:
If you could provide feedback on the PR, that would be appreciated. I will briefly test it myself later as well.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/anishathalye/gavel/pull/64#issuecomment-456881630, or mute the thread https://github.com/notifications/unsubscribe-auth/AA3fMBLJ3jobKzwKtUueCKSeZsj0EgvVks5vGJSggaJpZM4RT74W .
We just used this PR at CUhackit and it worked well. We ended up doing a hybrid approach where the judges judge for best software, hardware, and social good using the different categories system of the PR. The judges then conferred and agreed on some derivative categories like best beginner and most ingenious. I had no issues with the category system other than teams not submitting to a category, but they were added mid-way through and were sufficiently voted on.
It'd be great if this feature could be merged—we're considering Gavel for HackTJ 7.0.
Ah I think this one fell through the cracks last time I made a big push on backlogs in my open-source projects (two months ago).
For now, you could merge this pull request into your own copy of Gavel (and let us know how things go 😀). In the meantime, I'll put this on my to-do list and get to it eventually.
I believe I merged correctly, but the instance isn't working—https://github.com/hacktj/gavel is what we tried. Upon navigating to /admin we saw "Internal Server Error" and in the logs, there was a Programming Error with SQLAlchemy
We also tried the prizes branch in https://github.com/nwplus/gavel/tree/prizes but that didn't work either, and we also tried the master branch in https://github.com/nwplus/gavel, which didn't work.
We'll probably write our own feature for this (it might be a little too specific to HackTJ but hopefully it'll help others). When it's ready, I'll make sure to open a PR.
Ah that's too bad that the straightforward merge didn't work. I guess the PR is pretty old (I should probably have tried to merge it sooner), and Gavel has diverged since then, so merging is probably not super straightforward? Though it's interesting that the prizes and master branches from nwplus/gavel didn't work either (were you sure to re-create the database between switches? the schema might have changed, I don't remember).
If you do write the feature for this and it isn't too much effort, I'd be happy to see a PR. Then when I can get around to reviewing this multi-category feature PR, I'll see which one is a better fit for merging.
Though it's interesting that the
prizesandmasterbranches from nwplus/gavel didn't work either (were you sure to re-create the database between switches? the schema might have changed, I don't remember).
Yup, I deleted and re-created the database each time. It seems like an API change is the only logical explanation for this, but in the requirements.txt file there are versions for all the dependencies, so I don't know why this would affect the project.
If you do write the feature for this and it isn't too much effort, I'd be happy to see a PR.
I was taking a look around and you mentioned in a few places that HackMIT uses filtering categories—but I don't think this feature is in this repository's codebase. If it isn't, would it be possible to open-source the version used by HackMIT?
Filtering categories doesn't actually require changes to the code. What I meant by filtering categories was that there is a single overall ranking (done using judges / Gavel), and then each project is associated with some subset of categories, e.g. for a "Best iOS Project" prize. The way the best iOS hack is chosen is by looking at the Gavel ranking of all the projects and considering only the subset that are relevant, so taking the top-ranked overall project that is also an iOS hack.
HackMIT's version has diverged a bit from this repo. I'm not sure exactly what features they're using / exactly how it differs, but the code is open-source in case you want to take a look (they seem to have various feature branches): https://github.com/techx/gavel
I see; sorry for the misunderstanding. We'll probably modify the code anyway because we'll want judges with certain backgrounds to judge certain categories, but the final product might be a little too specific to HackTJ, so I might not make a PR. Thanks for your help!
edit: https://github.com/HackTJ/live