JamBrain
JamBrain copied to clipboard
JamBrain
Karma system may promote bad feedback behavior
I saw talk on discord that there are users who give low quality feedback, and still have very good karma. I'm thinking that it could be that even low quality feedback picks up a love here and there and so if you blast it across the page you might gain more than if you write high quality appreciated feedback.
I have an idea on how to deal with this, but I need to investigate the current code
I had someone mail me complaining about users that "liked everything" in a post, and how this could be used to cheat.
On Wed., Aug. 22, 2018, 03:01 Martin Zackrisson, [email protected] wrote:
I saw talk on discord that there are users who give low quality feedback, and still have very good karma. I'm thinking that it could be that even low quality feedback picks up a love here and there and so if you blast it across the page you might gain more than if you write high quality appreciated feedback.
I have an idea on how to deal with this, but I need to investigate the current code
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ludumdare/ludumdare/issues/1748, or mute the thread https://github.com/notifications/unsubscribe-auth/ABf2CYlPgE5SwccZOIXIgzkAUSArTLoIks5uTQGwgaJpZM4WHGnq .
Simple fix would be
raw_feedback = min(42, love_total / comments_count * 10)
This would cap karma at 42, it's the general quality of your comments up to the cap of 4.2 hearts per comment.
So if you are a spammer, chances are you'll stay down about less than ten. Only thing this doesn't do is give you credit for having written many comments. But on the other hand, the other aspects of smart does that already in part because it tracks grades given, and so it would be open to only commenting really good on a few games and earning a whole bunch.
I think eventually my intent with the smart filter was to make karma from previous participants worth more, as a means of offsetting abuse, as it is most likely come from first-timer accounts.
Anyway, in general I think this needs more research. I'll try to remember to post a more useful transcript of the reported issue in the morning.
On Wed., Aug. 22, 2018, 03:18 Martin Zackrisson, [email protected] wrote:
So if you are a spammer, chances are you'll stay down about less than ten. Only thing this doesn't do is give you credit for having written many comments. But on the other hand, the other aspects of smart does that already in part because it tracks grades given, and so it would be open to only commenting really good on a few games and earning a whole bunch.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ludumdare/ludumdare/issues/1748#issuecomment-414936244, or mute the thread https://github.com/notifications/unsubscribe-auth/ABf2CfhKDcmQq8uibLOfFVTyHSoWIvU5ks5uTQXXgaJpZM4WHGnq .
@mikekasprzak Our team's game was affected by someone abusing the system and I think we sent you that e-mail.
I think the current smart balance system can be abused by griefers to target specific games (and our game was in fact targeted).
Especially this line:
$raw_given_feedback = commentLove_CountBySuperNodeNotAuthorKnownNotAuthor($node['parent'], $node['id'], $authors);
Let me give an concrete (albeit fictive, but likely) example: Lets say a griefing team has 3 people. If that team of were to specifically target another game, they could each leave a feedback then like every other feedback.
That would add 31.5 to the targeted game's smart_against balance.
(3 * 20 + 3) / 2.0 = 31.5.
Let's assume that the targeted team has received 20 feedback and left 40 feedback on others' games, their initial smart balance would be: (sqrt(40 * 100 / 20) * 10) - 100 = 41.42;
Then lets have our greifers come and like every comment and leave 3 dummy comments. (sqrt(40 * 100 / (20 + 31.5) * 10) - 100 = -72.13;
So effectively that team of only 3 people can successfully reduce another team's smart balance by more than 100 by simply liking their feedback comments. And since we cannot find who liked the feedbacks (likes are anonymous?), we cannot track the offenders and kindly ask them to revoke their likes. To the benefit of griefers, they can stay anonymous while reducing other games' visibility.
Of course the equations I used are simplified versions of what's currently in use but the general idea is here.
There's a cap for smart_against, but since there's also a cap for smart_for, the targeted team in this example will not expect to see a high smart balance for a long time.
Fortunately our game got the required amount of feedback before the arrival of the lone griefer, but it still has impacted us a lot. We have now given more than twice the feedback and rating than what we received but our smart balance is barely positive.
Now some live examples, this is our game's current metric data:
The number of feedback we received is 32. With 28 feedback liked by the griefer + 5 other likes not from us.
Our smart balance before this ordeal was a strong 24. We kept being in the front 2-3 pages since we reviewed and gave quality feedback to around 20 games daily. But after this we felt it was useless as each new feedback we gave only increased our smart balance by an insignificant amount and we just stopped reviewing games.
Compare our smart balance to another very similar entry (one of my favorite actually):
They have received 42 feedback. No griefers.
We reviewed more games, received less feedback and still have less smart balance thanks to the griefer.
A simple fix would to limit the commentLove part for each user per game entry, which means a single person can at most reduce someone else's smart balance by 1.
Sorry to hear and this does highlight for me at least that we have to run some simulations on this. I don't know the particulars of your case but the sad thing is that the griefer may well be unaware of what they did. They might have just liked your game and the feedback. And even if that was not the case, it is quite possible that someone accidentally destroys someone else's visibility.
Thinking about it, I don't think we should care at all about how much love you received. It's not promoting anything useful and we want people to be happy about love gotten and give as much love as they feel.
My idea now is something like:
karma = power(average(love), log2(n_comments + 2))
Is it possible to find who liked our comments so we can ask him to remove those likes? I hope it was unintentional and it would be nice for us to get some more feedback/ratings for the ongoing Ludum jam. Our game entry is submitted by me, the Ludum Dare account has the same name as my Github account (bloc97).
Mike is out in the woods and the only one who could check since they are only accessible in the DB. As intended. But I'm not really sure what the good/right procedure could be in fixing that short of changing the algorithm
Maybe something like the feedback friends algorithm could be used. It relied on the comment length. This could easily be abused though in theory, but I don't remember anyone abusing it during LD 36.
Again, like @local-minimum said, many people are probably doing this unaware. This also falls under the Education aspect of the site as I'm not sure many first-time jammers know what the algorithm really is, much less how hearts affect it.
Bringing back a trophy/badge system could also help incentivize helpful feedback.
It would be nice if we could get a cap on this soon, I can't upvote good feedback on the games I play unless I want it to be near-impossible for them to get any more ratings. I can affect the smart score by about 13 smart points by upvoting the good feedback on one game, which is a huge difference
Why does liking someone's comments reduce the smartness of a game? What even is smart score? A way to show games with higher smart score? So the algorithm thinks a game has gotten enough reviews if many comments are liked, and thus reduces its smart score?
"Smart" is just the name given to the game prioritization algorithm. The original algorithm was essentially "number_of_scores_received - number_of_scores_given". Put another way, it wasn't very smart. 😉
Admittedly making the data public and calling it a score probably causes more confusion than necessary, but we're rather trusting. 😉
On Thu, Apr 20, 2023, 22:31 Carlonn Rivers @.***> wrote:
Why does liking someone's comments reduce the smartness of a game? What even is smart score? A way to show games with higher smart score? So the algorithm thinks a game has gotten enough reviews if many comments are liked, and thus reduces its smart score?
— Reply to this email directly, view it on GitHub https://github.com/JammerCore/JammerCore/issues/1748#issuecomment-1517171722, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL7MCP4XITVHX3XDGZAVDLXCHWQLANCNFSM4FQ4NHVA . You are receiving this because you were mentioned.Message ID: @.***>
Good point. I'll dive into the algorithm over the weekend. Initially were making a hard pivot to get folks to prioritize commenting (since it was way too easy to get priority from giving ratings). That can most certainly be cleaned up.
On Fri, Jan 20, 2023, 17:19 LoneWolfHT @.***> wrote:
It would be nice if we could get a cap on this soon, I can't upvote good feedback on the games I play unless I want it to be near-impossible for them to get any more ratings. I can affect the smart score by about 13 smart points by upvoting the good feedback on one game, which is a huge difference
— Reply to this email directly, view it on GitHub https://github.com/JammerCore/JammerCore/issues/1748#issuecomment-1399022179, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL7MCMI7LCI5YISGVKFAJ3WTMFP5ANCNFSM4FQ4NHVA . You are receiving this because you were mentioned.Message ID: @.***>
Will this be fixed for Ludum Dare 54? I have quite honestly had a much harder time participating in Ludum Dare ever since Ludum Dare 51 when I was hit really hard by this bug.
It simply gives me a lot of anxiety knowing that, at any time, my game's karma could be tanked by anyone, anonymously, with no way to reverse it. When it happens to you, playing and rating games barely moves the karma meter, and it feels really bad, and it then requires the person affected to find other ways to reach 20 ratings, which is very stressful.
I would definitely appreciate it if it was fixed in time for play & rate for Ludum Dare 54. From what I've looked at in the source code it seems like it should be a pretty straightforward change, but I can't tell exactly what all the different variables mean for the rating formula, otherwise I would have opened a pull request by now.
Well, I don't actually know if I understand the code correctly, but I've created a pull request for this anyways. If I'm right that the "given_feedback" variable corresponds to the hearts left on comments on somebody's game, then my change should work.
Again I'm hoping this can be merged before Ludum Dare 54 (if the changes are correct).
Just to add on to this, here is another user already affected by this system for Ludum Dare 54: scroll down to the comment by cymusi
@mikekasprzak IDK what the best way to contact you is, but I definitely hope you take a look at this soon.