What will become of reputation?

(Personal opinion, not official anything.)

I'm one of the people who, from the beginning, wanted to change how we handle reputation, because while Someplace Else got some things right with it, their setup also produced some degenerate results that I don't want to replicate.

The biggest problem, IMO, was tying privileges to raw reputation without looking deeper. On one of the sites I used to moderate, we fairly often saw poor answers on popular questions shoot up in (net) votes, and one-time posters suddenly had privileges they didn't really know how to use. This created extra work for community curators. We were keen to not repeat that mistake here at Codidact, and we now have an abilities system that ignores reputation and instead looks at the underlying actions -- your posts, suggested edits, and flags. For the purpose of abilities, an answer at net +5 counts the same as one at net +500; we're looking for "positive" and aren't measuring how positive. We think that someone who contributes ten positive answers has shown more fluency than someone with a single very popular answer.

So great, we've done that -- abilities are decoupled from reputation. What, then, of reputation?

I'm not opposed to having a single "bragging rights" number, whether it's called reputation or score or karma or something else. Communities that want to display it can; those that don't can omit it or even turn it off entirely. And I hear the requests for different weightings; not all post types in all categories are created equal. We currently allow communities to set the rep gain/loss for votes on each post type, without regard to category. That's an artifact of the original architecture, before we added categories, and I grant that it is not ideal. I'd like to be able to allow better, but still simple, configuration of this bragging-rights number. I'd like to replace the current rep configuration with a matrix: categories on one axis, post types on the other, rep for upvotes and downvotes within. This would allow a community to say that papers are worth more than blog posts even though they're both articles, remove downvote penalties on meta because working out policies and such inherently involves posting unpopular options for voting, remove rep from a sandbox entirely, and so on.

If we make rep more configurable in this way, I would want us to agree that this is the only way to gain or lose rep. It's about bragging rights from the posts you contribute, as another answer argues. All of the other valuable contributions people make to a community (edits, curation, etc) fall outside the rep system, to avoid diluting its signal and to avoid making the code even more complex.

posted almost 4 years ago

CC BY-SA 4.0

Monica Cellio‭ staff

64 372 2259 828

Back in the real old days, we had a real long discussion about reputation and privileges (exhibit A, B, C and likely many other discussions in other threads).

What emerged from those discussions was mainly that:

• People want something to feel recognized for their contributions.
• People want to see, how helpful they are to the community.
• Some people like reputation because they know it.
• Reputation isn't good to decide, who's trustworthy to do moderation actions (editing, closing, deleting...), because it's focused on posts and expertise.
• Reputation can produce negative behaviors.

Based on that, we collectively concluded that "classical reputation" has a lot of flaws. It combines aspects of recognition, feedback and privilege. And while it is an acceptable compromise of these three factors; we thought we might be able to do better by splitting them up.

So we did.

We started thinking of replacements of reputation for purposes of determining trust and privileges. At first, we looked at Discourse's trust level model. Instead of an unbound, continuous reputation, you would only have ca. 5 levels, which would indicate how much you could be trusted. They would be calculated based on all kind of things: number of posts, number of flags, number of edits, number of votes and their qualitiy.

However, that system was -- in our opinion -- still flawed: You only have one path of progression. You couldn't distinguish users who were good at editing to users who were good at detecting off-topic posts. We started thinking about providing multiple ways to get one trust level (AKA X edits or X successful flags), but things would just get more complicated and it'd still not solve the core problem:

We can only trust you to do things you show to be trustworthy at.

Finally, we developed the Abilities system. The core idea is dead simple: If you show that you are continuously making good edit suggestions to post, you'll eventually be trusted to make edits without supervision. Likewise for moderation.

But that leaves us with recognition and feedback. This is actually hard. Many people feel appreciated by different stuff.

Ultimately, we came to the conclusion that we should show different statistical details about a user on their "user card", so that other users can see whether they are experienced contributors or not. (One thing we also considered, but which hasn't been worked on so far and which hasn't received more thought recently, is to show something like a "tag expert" badge. When you're making helpful contributions to the Python tag, you'll eventually get a "Python expert" badge under your user card. I still like this idea, but it needs a lot more consideration.)

This is what the recent update introduced. Reputation-only display was amended by more specific profile details, such as number of posts or received votes.

But, especially in this area, we've received a lot of criticism and objections. People just like having a small number going up when they make useful posts. Hence, we are not going to "remove reputation" for now. Instead, we are deprioritizing it and experimenting with other options, such as the one introduced yesterday. Also, in our long tradition of empowering the communities, we wanted to make these details as customizable as possible. Therefore, communities are able to decide which details to show. If a community wants rep and only rep, that's fine. If they desperately want to get rid of rep, now they finally can.

I'm seeing this change as an experiment. But I also think that the current solution (possibly combined with the expert-badge) is a good one and could be what we ultimately end up with. Only time will be able to tell.

This leaves us, last but not least, with feedback. Reputation is a nice feedback, because it goes up when you do nice things and it goes down when you do bad things. Reputation going up also has the advantage of making you feel nice.

I know that this has been removed right now without replacement. I know that this isn't optimal. I'm also a bit annoyed by it. But, we are going to add some feedback mechanism (whether something fancy or just a list of recent votes). It's high priority on our TODO list, but that still means that it'll be a while, because we are all only volunteers here. Maybe, I'll add the reputation number in the header again, at least for the time being.

So, don't worry, the feed will be back again soon. (:P)

posted almost 4 years ago

CC BY-SA 4.0

3y ago by Quasímodo‭

luap42‭ staff

17 68 568 98

Whatever it is called, we still need a nice simple single number that is a rough measure of how much good content someone has contributed, as judged by all others on that site. "Reputation" is a reasonable name that captures this concept well enough, but there are probably others out there that would do too. Maybe "awesomeness score", "godliness status", "total worth as a human being", or something ;-)

The core value of any site is its ability to provide good answers. That means you need to have domain experts willing to spend their free time providing this value for you. We can't pay them, but what we can do it give them public recognition. The system doesn't have to be perfect (what system is?), but these experts providing the core value of your site need something to keep them engaged. The awesomeness score reputation number is a good way to do that.

Reputation has to actually mean something, else it is just a useless "atta boy". Some of the abilities should be dependent on reputation. It's fine for someone to get edit privileges after having enough edits approved. It's quite another to give someone policy-setting powers that isn't vested in the site and hasn't proven it by providing a lot of content.

These sites are all still young with small numbers of users. By and large, most people here actually care about the sites. That will change if we are successful. That's OK by itself, but the fatal error is giving the future masses any real power.

These sites must NOT be democracies. The masses are their own worst enemy. It is to any one user's advantage to get their question answered regardless of the effect on the site. However in the long run, the health of the site is far more important than any one question.

When a site gets larger, there is clamor from the masses for more leniency, resistance to closing bad questions, accusations of being "unwelcoming", mistreating newbies, and the like. If these are acted upon, then site quality declines, the experts get disillusioned, feel unappreciated, and eventually leave.

This is exactly what has happened to several SE sites when SE started valuing clicks over quality, probably because they associated clicks with revenue. They forgot that it was that quality that attracted the clicks in the first place. No matter how much the masses bitch and moan about actions to keep quality high, they are still ultimately there because of that quality.

To solve this, you give the core of resident experts the power to keep the place clean and set the policy. Sure, others can help with things like edits to fix bad wording or spellings, deleting inappropriate comments, etc, but the policy-setting powers need to be in the relatively few hands of the core group that has shown a commitment to the place by consistently providing good content. This isn't as much a reward for the service, as a recognition that these are the people that "make" the place, and need to be given the freedom to run it according to their vision.

That means there needs to be a way to identify the top committed contributors. This is the other aspect to "reputation". Means of figuring reputation need to be designed for this purpose. When done right, it will serve the public recognition purpose of reputation well enough, even if just as a byproduct.

Reputation must therefore measure "amount of good content contributed, as judged by the consumers of that content". SE got this largely right, although a few tweaks would make it better. Reputation should be:

1. Gained from upvotes on answers to questions in domain-content categories. This means you don't earn rep by asking questions, nor by anything in meta. You earn rep by providing good answers. How much others agreed or disagreed with things you said on meta shouldn't matter.
2. Lost by some types of downvotes on answers in domain-content categories. This is the reverse of #1, largely. One problem on SE they never really addressed was malicious and vandalous downvotes. I have felt for a long time that the best way to prevent that, and get honest downvotes in general is to make them all public. However, there is strong resistance to that. A good compromise would be to allows two types of downvotes, anonymous and "signed". Anonymous downvotes affect answer score, and therefore sort order, but nothing more. Signed downvotes also affect rep, but require a comment explaining the reason, or a "me too" on someone else's downvote comment.
3. Can be different per category. For example, votes on meta generally shouldn't affect rep at all, although that is settable as with other categories. On EE we have a Papers category. Writing a paper is more work, and it's held to a higher standard than questions and answers in the Q&A category. In that case, votes on the question (there are no answers in Papers) should mean more, both up and down. Overall, each category has mod-selectable numbers for how much upvotes and (signed) downvotes mean on questions and answers, for a total of 4 tweak factors. No special case then needs to be made for meta or any other category.

Reputation must also be simple. Just a single number is best. After all, we're really just trying to measure one thing, which is roughly "appreciated contribution level". Multiple numbers just confuse and/or dilute that.

Another tweak relative to SE would be having rep be based on "recent" contributions. Put another way, the rep from old contributions slowly decreases over time, eventually falling to 0. The point of this is to avoid making someone "eternal moderator for life". If someone hasn't been active on the site for a few years, they should no longer have a say in how it is run. This also gets around the problem of it being impossible for new active contributors to become part of the core group. If you provide a lot of good content over two years or so, you should be "in".

The exact parameters need to be tweakable, in no small part because it will take experience to find good values. A good start might be everything in the last year (or 2 years?) counts full, then decays to half its value every year after that. Or, maybe just a linear downgrade to 0 over 5 years after the initial year or two. The exact mechanism isn't important, at least not right now.

posted almost 4 years ago

CC BY-SA 4.0

Olin Lathrop‭

46 122 342 122