Regarding adding rating points to score of organizers of a CTF and reflecting the organizers CTF skills

[dnivra]

I happened to notice that the organizers of Backdoor CTF(SDSLabs) have a score of 75.547 points till now and it seems that majority of(>99%) the points this year are from organizing Backdoor CTF than from participating in CTFs. Looking back at previous years, I see the same trend: majority points seem to be from running Backdoor CTF than from participation in CTFs. This might be the case with more teams too perhaps. I feel this is not fair considering that this doesn't necessarily reflect how good the team is in playing CTFs(which is probably the point of having a rating system isn't it?).

I think a solution might be to award rating points to organizers only if they have scored same amount of points by participating in CTF competitions. It would be good to hear more opinions on what people think about this issue and possible solutions too.

[Pharisaeus]

I honestly don't see why. If a team can organize a decent CTF then it deserves to get points for that. In fact, from experience, I know that preparing good tasks and managing a CTF is far more demanding than simply playing it.

[dnivra]

I don't have a problem with awarding teams points for organizing CTFs - definitely running a CTF is not a trivial task(and I've done it too). Earning nearly all their points only from organizing just 1 CTF is what I'm talking about. Also, in case of backdoor CTF(this year and before), it seems there were some problems with contest based on the comments on CTFTime.

[Pharisaeus]

Earning nearly all their points only from organizing just 1 CTF is what I'm talking about.

I don't get the problem. Some teams get nearly all their points from playing 1 or 2 CTFs per year, you want to somehow strip them from those as well? The whole point of change in the scoring scheme, to include only a couple of best scores the team has, was to promote teams who are strong, and not teams who play all the time.

I think a solution might be to award rating points to organizers only if they have scored same amount of points by participating in CTF competitions

This makes no sense to me. Top team organizes a CTF with some high rating, but it doesn't mean they could win a CTF of the same level, especially given the tough competition.

Look at ESPR, they organize CCC with rating of 90, which means they would have to win an event with rating at least 90 to get their points (according to what you propose). But there is no such event at all. DefCON was 80, Plaid was 86... Same goes for other top teams - they organize high quality CTFs, but winning an event of similar difficulty might be not doable for them.

I don't think this is an issue at all. You envy those teams the points? Organize your own, high quality event.

I'd say a bigger issue is with who and how can score the events. Sadly what happens now is that some low-level events are upvoted to their max (25 this year) by some noname/newbie teams, simply because they managed to get high in the ranking (due to total lack of any strong teams participating). At the same time good-quality events are harshly punished for any shortcomings, because stronger teams have higher expectations. Paradoxally this causes some weird noname events, with hardly any participants, and low-quality/low-difficulty tasks getting higher weight than some high-quality events.

All of this is due to the fact that any team who played or was in top50 last year can vote. I would combine those conditions instead -> you can vote only if you played (for obvious reasons) and at the same time your team was top 50 last year (so you're good enough to actually rate if the quality/difficulty was good).

[dnivra]

I don't get the problem.

Sorry about the unclear descriptions till now. I hope this example would help understand. If you see the profile of SDSLabs, they have 75.547 points as of writing. They earned most of their points(75.3 points to be exact) from running Backdoor CTF(which, based on the comments on CTFTime, had some problems) and only 0.247 from participating in CTFs. Because of the organizer points, they are ranked higher than teams such as GoN which earned all their points solely from playing CTFs and not running a contest. This was what I was trying to highlight.

Some teams get nearly all their points from playing 1 or 2 CTFs per year, you want to somehow strip them from those as well?

Of course not - that would be unfair since undoubtedly those teams are pretty good. I am only referring to teams who earn majority points only by organizing CTFs and very little from playing CTFs.

This makes no sense to me. Top team organizes a CTF with some high rating, but it doesn't mean they could win a CTF of the same level, especially given the tough competition.

I never said winning a CTF with the same rating: the organizers just need to earn same points as the rating for a CTF they organize(not even double; which is what's awarded to organizers) by participating in CTFs and solving challenges. Surely that's not a problem for a CTF team that's good? Additionally, if a team can't earn the same points(from all CTFs that will be scored) as the rating of a CTF they organize, I feel it could be an indicator that the event they organized may not be very good.

Sadly what happens now is that some low-level events are upvoted to their max (25 this year) by some noname/newbie teams, simply because they managed to get high in the ranking (due to total lack of any strong teams participating).

Yes this is definitely a problem. Additionally, some teams can organize CTF events which don't have any strong teams participating and they can boost their points using the organizer points from running the CTF.

At the same time good-quality events are harshly punished for any shortcomings, because stronger teams have higher expectations. Paradoxally this causes some weird noname events, with hardly any participants, and low-quality/low-difficulty tasks getting higher weight than some high-quality events.

A possible solution here might be to have a tier system perhaps similar to how Topcoder has a division 1 and division 2. Thus, top teams can be in division 1 perhaps and thus still have higher rated events(unless it was extremely bad). This would require more discussion and I think is probably a distinct issue than the one I originally posted.

Edit: Made some changes to improve clarity of my responses.

[Pharisaeus]

They earned most of their points(75.3 points to be exact) from running Backdoor CTF(which, based on the comments on CTFTime, had some problems)

1. Rating is now calculated for the NEXT year. So if event had issues this year, it will have lower rating next year. This was introduced to prevent people upvoting events based on their current performance (which still happens for new CTFs)
2. Backdoor is a good example for event getting bashed by strong teams for some trivial issues. The issues were: lack of category assigned to the tasks (cumbersome, but it's not a critical issue), periodical task release without clear schedule (again a bit pain in the neck), they got hit by DDoS at the beginning. Still, the quality of the tasks was reasonable (as far as I remember).

Either way: I have no issue with teams getting rewarded for organizing an event. How many points they get depends on the event quality.

Surely that's not a problem for a CTF team that's good?

I'm not sure. I can imagine people organizing a CTF at much higher level than their team can play. Especially when it comes to some medium teams. They can copy some ideas for tasks from top-tier events, and come up with tasks much above what they could solve by themselves.

Thus, top teams can be in division 1 perhaps and thus still have higher rated events(unless it was extremely bad)

Medium team can make a top-tier event (look for example at Hacklu or EKOParty), and strong team can make a bad CTF, so I don't think you can couple this.