Welcome to Codidact Meta!
Codidact Meta is the meta-discussion site for the Codidact community network and the Codidact software. Whether you have bug reports or feature requests, support questions or rule discussions that touch the whole network – this is the site for you.
We need to talk about Abilities
The problem
Almost three years ago we introduced Abilities to Codidact as an improved system for granting activity-based permissions, such as allowing users who have made consistently great edit suggestions to edit posts without requiring approval. I'll defer to the linked posts on how Abilities function in detail.
I do still believe that coupling permissions with specific experience in the area of the permission is a good idea and should be kept as the basic premise of the permission system.
However, there are a three major issues I see with how our abilities system works currently:
-
The Ability thresholds are calculated using the Wilson score formula we use for posts. This means, that the minimum number of (positive) actions needed and the minimum percentage of positive actions are directly coupled.
In some cases this coupling can be inappropriate for the intended purposes. For example, a community might want a user to have suggested many edits to demonstrate experience but want to require only an average approved percentage of like 75%. In our current system this would not be possible.
In some cases communities might want the opposite. They would want a high percentage of 95% edits approved, but because they have low activity they only want a small sample of 5 edits (which would of course mean that once one edit is rejected your amount of edits that need to be approved would rise drastically). This would also not be possible with the current system.
-
Currently abilities are directly coupled with permissions. This means, that the system will check whether or not you have a certain ability and will infer your permission from this. This becomes a problem once a permission should be inferred from multiple abilities, because then the every time a permission is checked the code would have to keep in mind how exactly abilities bind to permissions, and if multiple permissions are coupled into one ability, because they are then affixed together as an all-or-nothing thing. However one might want to separate permissions from another for moderation purposes. A user might need to be banned from reviewing edits while their power to edit posts should not be restricted.
-
The current system is based on specific existing abilities hard-coded in the application. However communities might want to change the ability progression to better match their individual needs.
A solution?
Therefore I am proposing some tweaks to how abilities are implemented:
-
There are two factors: "privileges" and "abilities" (or "trust levels").
-
Privileges are technical allowances, such as "may edit other people's posts without review", "may review suggested edits" or "may vote to close". A user can have or not have a privilege at any time.
-
Abilities are steps on the progression ladder. Technically an ability consists of one or multiple privileges and a "win condition". If you earn an ability, you will be granted the privileges it contains.
-
Privileges can be individually suspended by moderators (as abilities can be suspended for time/infinity right now). Abilities cannot be suspended because the only thing that goes beyond suspendable privileges are "bragging rights".
-
Abilities are reached when its win condition is reached. A win condition may consist of multiple requirements, each of which must be completed. Requirements could be, for example,
- minimum user age
- obtaining a specific ability
- minimum post count
- minimum positively received post count
- minimum percentage of posts positively received
- minimum (suggested) edits
- minimum approved/applied edits
- minimum percentage of suggested edits approved
- minimum flag count
- minimum helpful flag count
- minimum percentage of helpful flags
- minimum close/delete votes
- minimum effective close/delete votes
- manual appointment (i.e. this ability can only be awarded manually)
As before, moderators would have all such privileges.
Something similar to the current ability progression should be set as standard, so that we have an ability trees not unlike this:
- everyone (general path)
- established user (general path)
- editing posts (edit path)
- editing tags (edit path)
- closing posts (mod path)
- deleting posts (mod path)
- editing posts (edit path)
- moderators
- established user (general path)
However communities would be free to change this progression as they like.
I think that these tweaks would allow better bootstrapping of communities and would allow it to better react to community size with configuring the ability system.
Discussion
What do you think? Do you have any other ideas on how to improve abilities?
I've been thinking about abilities again lately (and for a while), because they're too hard to earn but just tweaking th …
1mo ago
Considering both timeframe and last N events We could base awarding of an ability on both proportion of events in a t …
1mo ago
My main area of concern would be your "Problem", point 2. You seem to have put a lot of thought into that one as well, b …
1y ago
Testing different approaches When considering what approach to take to awarding abilities, or what specific values to u …
1mo ago
I want moderators to be able to grant or suspend abilities at fairly granular levels, because sometimes that person who' …
1y ago
coupling permissions with specific experience in the area of the permission is a good idea and should be kept as the bas …
1y ago
6 answers
You are accessing this answer with a direct link, so it's being shown above all other answers regardless of its score. You can return to the normal view.
Testing different approaches
When considering what approach to take to awarding abilities, or what specific values to use for the thresholds, it might be worth drafting database queries for each approach being considered, and a few different threshold values for each approach.
We can then take advantage of the fact that there is a body of data built up already. Even though much of this data is confidential, someone already trusted with database access could run the various queries and share the overall totals without divulging anything private. For example: "With approach A people would have taken on average 5 months and 76 events to gain this ability, whereas with approach B people would have taken on average 4 months and 62 events to gain this ability".
Trying this out for a few different size communities which have had different traffic levels over time could give an idea of what settings it is useful to have, and what defaults to start them out at. Since these totals don't need to be private, results from each approach could be posted in separate answers for discussion and voting. If the database queries are staggered over time to avoid too much load on the server, these results could even be in the form of charts that show how the outcomes vary as each threshold is varied.
0 comment threads
I've been thinking about abilities again lately (and for a while), because they're too hard to earn but just tweaking the Wilson numbers won't fix it. The fundamental problem is that a single negative event sticks to you forever and requires many more positive events to compensate. I sometimes look at active users, wondering "why doesn't this person have such-and-such ability yet?", and the answer is one declined flag two years ago or something like that, despite dozens of accepted ones since.
What problems would arise if abilities, instead of using Wilson scores, were formulated as:
- At least N positive events within timeframe T1
- And no more than M negative events within timeframe T2
The events would be the same as what we use now: posts, suggested edits, flags. Maybe those should be changed too, but I'd like to see if we can just adjust how we count the things we already have for the abilities we already have and if that would make things run more smoothly for our communities.
I originally wrote this as a single timeframe, but I think for some things we'll want to consider the positive record for longer than the negative record. Maybe the timeframes will end up being the same in some cases. If we're going to do this at all, having two values rather than one doesn't seem too complicated.
I'm thinking this as the logic for granting an ability, not logic for doing ongoing assessment to grant or revoke. Currently, abilities are not auto-revoked if you do something that would have caused you to no longer qualify. Moderators can suspend or revoke abilities if needed.
2 comment threads
My main area of concern would be your "Problem", point 2. You seem to have put a lot of thought into that one as well, but I have a number of questions. I'm first most interested in fixing the top level groupings. Are the security levels basically just mods and users? I'm seeing staff tags as well. Does that carry any particular organizational, process, or security meaning in this context? And can you think of any other possible "exception" groups (e.g. guest)?
It looks like you're doing a good job of organizing the "privilege structures". I was going to stop and ask for that before all else - then I scrolled down. So kudos there. However, there are a number of potential layers of abstraction between "privilege" and a high-level classification like mod, and it's probably a good idea to flesh out any other similarities between users that might impact or be proxies for their privileges.
What about crossover between groups? I.e. can a mod be denied some mod privilege, either temporarily or permanently, but still need to be "a mod" for some reason? Or is a fully trusted user essentially equivalent to a mod in all access within a site? Once a privilege is unlocked, can it be removed again from the frontend? In particular, can adjustments to the qualifications, or "win conditions" cause existing privileges to be added or removed?
On a personal level (i.e. as an end user), I didn't really get excited about the idea of the abilities system because it seems potentially ornerous. If it "just works", then in principle it does seem like a rational concept. But realistically, I don't actually want to have to "prove" to anyone that I know how to do things. While I can accept that it's sometimes necessary, it easily becomes monotonous.
In that vein, I feel like there's a significant missing component here, which is probably best encapsulated by a concept of "general competence and maturity". I'm a grown-up, and I want to be treated as one. We can't guarantee that's true of everyone coming to that platform, and in fact we must expect it won't be the case for many. But once I feel like I've put in the leg work to prove it, I don't want to be hassled anymore - pretty much at all.
The key is that the software be designed to make the same kind of decisions that a human mentor would if they were manually assigning permissions each time. Yes, if I've only ever posted answers, then suddenly go to edit someone's question for the first time, then it's hard to judge my competence or decide the risk. And there probably do need to be some minimal safeguards in place as you move into new tasks on the site. However, the risk is significantly lower if I've posted 50 good answers over a year, with lots of upvotes, than if I just made my account yesterday. That's why SE's reputation/privilege system works at all.
So I think there needs to be a concept of a general "user trust level", which unlocks at least some of the more ornerous safeguards after sufficient contributions in any area of the site. I also think that the abilities system, while innovative, and interesting to the nerds among us, should really be a hidden "under layer". If end users feel like they have to spend time learning to understand it before jumping in - we've already lost.
Considering both timeframe and last N events
We could base awarding of an ability on both proportion of events in a timeframe, and proportion of events in the last N events. I'll use concrete numbers because I find that clearer, but these are just examples - the actual numbers could be different (and different per community):
In order to gain this ability, you need to have 95% positive events over the last 3 months, and 95% positive events over the last 30 events.
You don't get the ability until both of these requirements are met at the same time, but once you get the ability you don't lose it unless a moderator removes it manually.
This means that on a low traffic community 3 months may not be long enough to get 30 events, so gaining the ability may take longer than 3 months.
Similarly on a high traffic community you can't get the ability in a day just because you were able to process 30 events in a day (which means spammers won't get abilities unless they play the long game and behave consistently well for 3 months).
This is similar to the current system where you need a hard minimum such as 30 events, but now once you go past that minimum, older events stop counting against or towards you. Having the timeframe stops someone from covering up bad behaviour with rapid production of good events afterwards, but still allows bad behaviour to age away eventually. If you have negative events, they cease to count against you once both 30 more events and 3 months have elapsed.
Simplicity
I expect this could be made fairer by making it more complex, but personally I would lean towards keeping it as simple as possible while still taking into account both aspects (last M months and last N events). Wilson score is not immediately intuitive for most people, but the extra complexity is worthwhile in order to present answers in a meaningful order. The same does not apply to gaining abilities - the exact order in which different people gain an ability does not seem important in the way that the order of answers does. For abilities I would prefer to have something sufficiently simple that it can be displayed both graphically and textually in a way that is immediately clear.
0 comment threads
I want moderators to be able to grant or suspend abilities at fairly granular levels, because sometimes that person who's answered a ton of questions well also has a tendency to make noisy edits or leave rude comments, and I want moderators to be able to take the smallest action that addresses a problem.
As an admin who just yesterday tried to adjust the settings for abilities on a new site... yeah, it's complicated, and I felt like I couldn't express what I wanted. When deciding whether a user should have an ability, I want the system to be able to take into account two things:
- how many times the user has done the related action
- what the "success" proportion was
If somebody has suggested 100 edits and 3 of them were rejected, I probably don't care -- that's an awful lot of good edits, and everyone makes mistakes occasionally. Or maybe the edit shouldn't have been rejected. On the other hand, if someone has suggested 5 edits and 3 of them were rejected, the user probably shouldn't be able to make unreviewed edits just yet.
So we don't want raw numbers, but percentages alone aren't enough. I want to be able to express something like "at least 10 suggested edits, and 75% approval". Pure Wilson scoring doesn't let me express that.
If we had this kind of expressiveness, then I think a lot of the pain points for the current abilities system would decrease or go away. We have higher (default) thresholds on some things that could be disruptive if unfettered, but those higher scores implicitly raise the minimum number of events to overcome a single mistake.
I agree with you (and another answer) that some other factors should contribute too, including age of the account, overall level of activity, and (maybe) activity on other network communities. Abilities can currently depend on three factors: post scores, number of suggested edits, and number of flags. I either suggested or advocated for those factors (I don't remember now) and at the time I thought they would be sufficient, but now I don't think they are.
I think what you're proposing that we group what are currently called "abilities" (to be renamed "privileges") into larger buckets called "abilities" (new meaning), so that you could define something like:
New user:
requires: (nothing)
grants: Daily Posts = 5, Comment on Own Posts
Established user:
requires:
account_age: 30d
non_negative_posts: 10
suggested_edits: 5
grants: Edit, Comment Anywhere, Daily Votes = 30
And also that we could have specific privilege criteria, that don't have to depend on the broad abilities, like:
Edit:
suggested_edits: 10
edit_ acceptance_rate: 75%
Would this be expressive enough to allow moderators to set the limits that make sense for their communities? Would it be uncomplicated enough to use? Would we be able to bake in some sort of validation or advisory feedback if the ability definitions and privilege definitions are inconsistent?
coupling permissions with specific experience in the area of the permission is a good idea and should be kept as the basic premise of the permission system.
I disagree. This is the basic flaw of the current abilities/permission system. It is trying to be overly specific and misses the forest for the trees.
Basically, we only want people that know what they're doing to be allowed to do things that could damage the site. That's basically all there's to it. The problem then comes down to determining "know what they're doing".
The problem with our current system is that it dives deep into irrelevant minutia in trying to determine "knows what what they're doing" along various different orthogonal axes.
For example, you're not trusted to edit a post if you haven't edited enough previously that others agree were edited well. But, having performed previous edits is a very narrow way to measure "knows what they're doing" when it comes to edits. What you really want is someone that knows a good post from a bad one. Simply writing a bunch of good posts should be good enough to demonstrate that.
All the various "knows what they're doing" attributes are closely linked. If someone has been around a site long enough and has a track record of positive contributions, we should assume they "know what they're doing" and give them the powers to help with the janitorial work.
Some actions have more potential for damaging a site than others, so there should be a higher "knows what they're doing" threshold to allow them. For example, editing others' posts doesn't need a high bar. Most edits are uncontroversial spelling or grammar fixes. If someone changes meaning against the author's intent, then the edit can be rolled back and a warning sent by a mod.
On the other hand, closing questions is often controversial even among experienced users with solidly positive track records. Users should be selected carefully to receive that permission.
In summary, lots of different successful positive actions on a site should increase one's "knows what they're doing" factor, which in turn should unlock various permissions. These things aren't anywhere as orthogonal as our current system treats them.
As an example, I have the highest rep on the Physics site, by more than 2x the next user. I've written 36 answers with an average of 2.0 upvotes per answer, and only one with a negative score. That should be plenty good enough to be trusted to know what a good question looks like, even though I haven't written any question. I've also done 21 edits, but am still not trusted to make edits without them being approved by someone else. That's ridiculous!
As a real world analogy, the current system is akin to saying you're not trusted to review books without having written a best-seller. Just having read many books doesn't somehow give you the experience to be able to rate the books you've read. That makes no sense in the real world, and not much more sense here on Codidact.
I don't necessarily dislike the idea of certain privileges being tracked orthagonally to the "knows what they're doing" factor, as you called it. But at some point, the latter should be sufficient to gain access to most privileges.
I totally agree and should have pointed that out myself. Testing your ability to carry out specific actions properly can be useful to allow you to perform those actions without supervision when we've got nothing else to judge you by. However, once you accumulate enough "know what you're doing" factor, privileges should unlock regardless of the mix of specific actions you've taken.
1 comment thread
0 comment threads