AI Chat Moderation

Overview

XBans reads every chat message through three layers, and all three feed the same escalation ladder.

• Chat rules — measurements, not guesses. Block capitals, a held key, a link, a listed word, a word written in two alphabets at once. No training data and no threshold involved.
• The classifier — a Naive Bayes model that weighs words, for the phrasing a rule cannot anticipate.
• Near misses — what scored just under the bar, added up over time, so nobody can sit below it indefinitely.

The model ships trained on 1,569 samples across six categories: clean, toxicity, cheating, scam, griefing and spam. The samples are mostly English, with some French, Spanish and German. That is enough for phrasing and intent, but a model is a poor place to keep a list of insults, and your community's own words belong in chat-rules.words below.

How It Works

The Bayesian classifier works by analyzing the probability that individual words and word combinations appear in toxic vs. clean messages. When a player sends a chat message:

1. The message is normalized (lowercased, special characters stripped, leet-speak decoded)
2. The message is tokenized into individual words and bigrams (word pairs)
3. Each token's probability of appearing in toxic vs. clean messages is looked up from the trained model
4. The probabilities are combined using Bayes' theorem to produce a toxicity score (0.0 to 1.0)
5. If the score exceeds the configured threshold, the message is flagged as toxic

Confidence is spread over the length of the message rather than saturating, so a single word and a paragraph are scored on the same scale. Two settings decide whether anything happens:

chat-ai:
  # How sure the model must be before a category is acted on.
  min-confidence: 0.5
  # How close "clean" may get to that category before the message is left alone,
  # as a fraction of it. At 0.75, a message scored 0.60 toxic is skipped once
  # clean reaches 0.45.
  clean-margin: 0.75

clean-margin is a ratio and not a fixed number on purpose. Absolute probabilities move with how much was written, so a fixed number only ever suits one message length. Lower it to act more readily, raise it to leave anything ambiguous alone.

Use /xbans ai test <message> to see the category, the confidence and the words behind a verdict before committing to a threshold.

Chat Rules

A word model is blind to everything that is not about the words. A message in block capitals, a key held down, an address written as "discord dot gg", a word spelled with Cyrillic letters that look Latin. These are measurements rather than guesses, and each one hands its verdict to the same category as the model, so the escalation below applies either way.

chat-rules:
  enabled: true
  # Capitals and repetition only. "OK" is not shouting, and the share of capitals
  # in a two-letter message means nothing. Listed terms, links and mixed alphabets
  # are checked whatever the length: a short insult is still an insult.
  min-length: 8
  caps:
    enabled: true
    ratio: 0.75        # share of the LETTERS that must be capitals
    category: "spam"
  repetition:
    enabled: true
    run: 6             # a run of this many identical characters is a held key
  links:
    enabled: true
    category: "advertising"
  homoglyphs:
    enabled: true
    category: "spam"

Listed terms

Three lists, and the difference between them matters.

  words:
    enabled: true
    category: "toxicity"
    # Searched anywhere, including across a split.
    list:
      - connard
      - salope
      - bitch
    # Compared as whole words, for terms that live inside ordinary ones.
    whole-word:
      - con      # concert, conclure, condition
      - pd       # "coup de", "beaucoup de", "trop dur"
      - ass      # assez, assiette, assign
    # Innocent words that really do contain a listed term.
    allow:
      - salopette
      - dispute
      - depute

A term in list is found however it was written. The message is read with every separator, accent, digit and repetition removed, so sal ope, s a l o p e, sal0pe and saloooope all reduce to the same thing. Ambiguous characters are expanded into every reading rather than guessed, so 1 and | are each tried as both i and l.

That strength is also why some terms cannot go there. Removing the spaces means two ordinary words glued together can produce a short term by accident, which is why anything of two letters belongs in whole-word. Put a term in whole-word and it is safe, but it no longer survives being split, so keep that list short and reserve it for genuine collisions.

allow is the third tool, for a real word that happens to contain a listed term. An exception is recognised as a word, not as a run of letters, so dispute is spared without fils de pute being spared with it.

Any language

Normalisation keeps every alphabet, so a Russian, Greek or Arabic term is matched the same way a French one is, split and disguised forms included. Letters that carry no accent to strip are spelled out, so German ß, Danish ø and Icelandic þ match a message written properly. The digit and symbol substitutions are Latin by nature and only ever apply there.

Near Misses

A message scoring just under min-confidence used to be dropped and forgotten, so somebody who learned where the bar sat could sit under it indefinitely. Confidences are now added up per category over a window, and enough of them count as one infraction.

chat-ai:
  near-miss:
    enabled: true
    # Only messages scoring at least this much are remembered.
    floor: 0.6
    # Accumulated confidence that counts as one infraction.
    threshold: 2.0

With these values, four messages scoring 0.5 each are worth one infraction, while a single one is not. The running total is kept for 30 minutes after the last message in a category. The infraction it produces is an ordinary one and enters the same ladder as everything else.

Escalating Actions

When a message is flagged, XBans doesn't just block it. It applies escalating actions based on how many violations the player has accumulated, configured per category (toxicity, spam, scam):

chat-ai:
  enabled: true
  min-confidence: 0.5
  monitored-categories:
    toxicity:
      reset-after: "1d"
      actions:
        1: "warn {player} {reason}"
        2: "mute {player} 10m {reason}"
        3: "mute {player} 1h {reason}"
        5: "mute {player} 1d {reason}"
        10: "ban {player} 7d {reason}"
    spam:
      reset-after: "6h"
      actions:
        1: "warn {player} {reason}"
        2: "mute {player} 5m {reason}"
        3: "mute {player} 30m {reason}"
        5: "mute {player} 3h {reason}"
    scam:
      reset-after: "0"
      actions:
        1: "warn {player} {reason}"
        2: "mute {player} 1h {reason}"
        3: "ban {player} 7d {reason}"
    advertising:
      reset-after: "1d"
      actions:
        1: "warn {player} {reason}"
        2: "mute {player} 30m {reason}"
        3: "mute {player} 1d {reason}"

Actions are full XBans command strings with {player} and {reason} placeholders. {reason} resolves from lang/<code>.yml using the key chat-ai-reason-<category>-<threshold>, so messages can be customized and translated. The violation counter is per-player per-category and resets after the configured reset-after duration (use "0" to never reset).

Custom Training Data

The model is trained from a single file, plugins/XCore/addons/XBans/ai-training.yml. It holds one list per category, and the categories are the ones the escalation ladder uses.

clean:
  - hello everyone
  - anyone want to trade
toxicity:
  - you are worthless
  - kys noob
cheating:
  - he flies through walls
scam:
  - send me your items first and i will pay after
griefing:
  - i blew up his base
spam:
  - buy buy buy buy buy

Add your own lines under the category they belong to, then run /xbans ai train. Real messages from your own chat logs are worth far more than invented ones.

The commands

• /xbans ai stats — sample counts per category and the size of the model
• /xbans ai test <message> — the category, the confidence, the words behind the verdict, and which of the three checks would stop it
• /xbans ai train — rebuild the model from the file
• /xbans ai reset — throw the model away and start from what ships