Add lemmatization confidence scoring (e.g., based on frequency or match level).

[linuxscout]

trafficstars

[linuxscout]

✅ Why Add Confidence Scoring?

When Qalsadi returns multiple possible lemmas or a single one from ranked options, adding a confidence score helps:

Use Case	Benefit
🧪 Downstream NLP (e.g., tagging)	Filter or weight based on confidence
🧠 Human-in-the-loop review	Prioritize uncertain cases
📈 Error analysis or benchmarking	Identify weak predictions

---

🎯 How Confidence Might Be Computed

1. By Frequency Rank (if Qalsadi uses solution ordering)

• If lemma is chosen from solutions[0], it’s more confident.

• Confidence could be a function of:

  • Frequency
  • Rank in list
  • Relative score compared to other solutions (if available)

2. By Match Quality

• Use heuristics like:

  • Exact surface match? ✔️ High
  • Rule-based lemma? ⚠️ Medium
  • Fallback default? ❌ Low

3. By Source

• Lexicon match > rule > guess

---

✅ Example Output

[
  {
    "word": "المعلمين",
    "lemma": "معلم",
    "pos": "noun",
    "confidence": 0.92
  },
  {
    "word": "كتبوا",
    "lemma": "كتب",
    "pos": "verb",
    "confidence": 0.78
  }
]

---

🔧 How to Add (Conceptually)

In your lemmatizer:

solutions = self.get_solutions(word)
best = solutions[0]
confidence = self._compute_confidence(solutions)

return {"lemma": best["lemma"], "confidence": confidence}

Where _compute_confidence() could be something like:

def _compute_confidence(self, solutions):
    if not solutions:
        return 0.0
    return max(0.1, 1.0 / (1 + len(solutions) - 1))  # inverse of rank

Or assign weights based on frequency info if available.

---

🚩 Note

Qalsadi would need either:

• frequency metadata in its dictionaries (or ranked solutions)
• Or heuristics to infer confidence from rules/source

[linuxscout]

✅ TODO List: Lemmatization Confidence Scoring

📦 Data & Structure

• [ ] Review how lemmatizer.py selects lemmas from solutions.
• [ ] Identify where frequency or ranking of solutions is available.
• [ ] Decide where to insert confidence score in the output structure.

🔍 Confidence Strategy

• [ ] Define a scoring model based on:

  • [ ] ✅ Solution rank (e.g. top solution = 1.0, second = 0.8, etc.)
  • [ ] Optional: Match type (lexicon match, rule, fallback)
  • [ ] Optional: Frequency (if available in dictionary)

• [ ] Implement compute_confidence(solutions: list) → float

🛠 Code Integration

• [ ] Modify lemmatization result to include "confidence" field.
• [ ] Ensure scoring does not affect logic — just adds metadata.
• [ ] Add an optional flag --with-confidence (CLI) or confidence=True (API)

✅ Output

• [ ] Update ResultFormatter to include confidence if available.
• [ ] Add support to --profile lemmas and main to show it.
• [ ] Ensure as_json(), as_csv() etc. handle the field.

🧪 Testing

• [ ] Write unit tests for compute_confidence()
• [ ] Add example tests showing output with confidence scores.

📚 Documentation

• [ ] Document the scoring method in README.md or CLI --help.
• [ ] Provide example with JSON including confidence.

[linuxscout]

✅ Function: compute_confidence()

Place this inside your Lemmatizer class (or wherever solutions are ranked):

def compute_confidence(self, solutions):
    """
    Compute confidence score based on solution rank.

    - Top-ranked solution: 1.0
    - Next ranks: linearly decreasing (e.g., 0.9, 0.8, ...)
    - Minimum threshold: 0.1
    """
    if not solutions:
        return 0.0

    rank = 0  # first solution
    max_conf = 1.0
    step = 0.1  # decrease per rank

    return max(max_conf - (rank * step), 0.1)

You can also modify it to assign lower scores to fallback/rule-based results if needed.

---

✅ Integration (Minimal)

In your lemmatizer method (e.g., lemmatize_text or lemmatize_word):

results = []
for word in words:
    solutions = self.analyze(word)
    if not solutions:
        lemma = word
        conf = 0.0
    else:
        best = solutions[0]
        lemma = best.get("lemma", word)
        conf = self.compute_confidence(solutions)
    
    results.append({
        "word": word,
        "lemma": lemma,
        "pos": best.get("pos") if solutions else None,
        "confidence": conf
    })

---

✅ CLI Option

In argparse, add:

parser.add_argument("--with-confidence", action="store_true",
                    help="Include confidence score in lemmatization output")

Then pass with_confidence=args.with_confidence to your lemmatizer.

---

✅ Formatter Update

In ResultFormatter, check:

if any("confidence" in result for result in self.results):
    self.used_fields.append("confidence")

---

✅ Output Example

[
  { "word": "كتبوا", "lemma": "كتب", "pos": "verb", "confidence": 1.0 },
  { "word": "المعلمين", "lemma": "معلم", "pos": "noun", "confidence": 0.9 }
]