kolosal-cli icon indicating copy to clipboard operation
kolosal-cli copied to clipboard

Improve Loop Detection UX and Fix False Positives

Open drestanto opened this issue 1 month ago • 0 comments

Issue: Improve Loop Detection UX and Fix False Positives

Summary

Loop detection triggers false positives, blocking legitimate workflows. Error messages provide no actionable guidance.

Current Error Message

ℹ A potential loop was detected. This can happen due to repetitive tool calls or other model behavior. The request has been halted.

Problems: Vague, no guidance, frustrating - users don't know what went wrong or how to fix it.


Bug: False Positive - Content Loop

Scenario (User Reported, Try to Reproduce it but can't):

  1. User: "Create a simple lightweight accounting app"
  2. Kolosal: Shows implementation plan, asks "Do you agree?"
  3. User: "Yes, I agree"
  4. ERROR: "potential loop detected" - before any file was written!
  5. Workaround: Had to ask for MD file with plan first, then it worked

Root Cause (Speculative): Content loop detection uses 1-char sliding window on 50-char chunks. Structured plans with repeated phrases trigger false detection.

// loopDetectionService.ts - analyzeContentChunksForLoop()
this.lastContentIndex++; // ❌ Increments by 1, not chunk size

When model generates:

1. Create main.py with the following structure...
2. Create database.py with the following structure...
3. Create models.py with the following structure...

The repeated "Create X with the following structure" pattern, combined with overlapping 50-char chunks (sliding by 1 char), quickly hits the threshold of 10 matches.


Additional Concern: LLM Judge

After 30 turns, an external LLM judges if conversation is "stuck".

Problem: 30 turns = ~10 file operations (read + edit + verify). Complex but legitimate tasks get flagged.


Proposed Fixes

Fix 1: Better Sliding Window (High Priority)

// Change from 1-char to half-chunk increment to reduce false matches
this.lastContentIndex += Math.floor(CONTENT_CHUNK_SIZE / 2); // 50% overlap max

Fix 2: Adjust Thresholds

Parameter Current Proposed
CONTENT_LOOP_THRESHOLD 10 15
LLM_CHECK_AFTER_TURNS 30 50

Fix 3: Informative Error Messages

// Instead of generic message, show specific info + tips:
{
  type: 'CHANTING_IDENTICAL_SENTENCES',
  message: 'Loop detected: Model generating repetitive content',
  tips: [
    'Try rephrasing your request to be more specific',
    'Use /clear to reset conversation context'
  ]
}

Fix 4: Add Loop Context to Event

// Current - no context
yield { type: GeminiEventType.LoopDetected };

// Proposed - include details for better error messages
yield {
  type: GeminiEventType.LoopDetected,
  value: { loopType, repeatedContent, reasoning }
};

Checklist

High Priority:

  • [ ] Fix sliding window increment (1 char → half chunk)
  • [ ] Increase CONTENT_LOOP_THRESHOLD to reduce false positives
  • [ ] Add loop context to LoopDetected event
  • [ ] Show specific error messages with actionable tips

Medium Priority:

  • [ ] Increase LLM_CHECK_AFTER_TURNS threshold
  • [ ] Add user config options for thresholds

References

  • packages/core/src/services/loopDetectionService.ts - Detection logic
  • packages/cli/src/ui/hooks/useGeminiStream.ts - Error message display

drestanto avatar Nov 30 '25 11:11 drestanto