CyberScraper-2077 icon indicating copy to clipboard operation
CyberScraper-2077 copied to clipboard

Feature/mcp

Open kanalive opened this issue 1 year ago • 1 comments

MCP Server and Client for CyberScraper

(Sorry for the multiple PRs this weekend! This PR introduces another idea: implementing CyberScraper as an MCP server. This allows it to be integrated with Claude Desktop and enables anyone to write their own MCP client to interact with CyberScraper's capabilities)

This PR adds Model Context Protocol (MCP) support to CyberScraper 2077, allowing the scraper to be used as a service with standardized inputs and outputs.

Changes:

  • Add MCP server implementation (cyberscraper_mcp.py) that wraps the WebExtractor functionality
  • Create Python MCP client (cyberscraper_mcp_client.py) for simple integration
  • Support passing API keys via the protocol rather than environment variables
  • Include proper error handling and task status tracking
  • Add documentation for integration with other systems

Benefits:

  • Enables integration with systems that support MCP (like Claude Desktop)
  • Allows headless/serverless operation of the scraping functionality
  • Makes it easy to build applications on top of CyberScraper's capabilities
  • Preserves all the AI-powered extraction features of the original

All tests passing. This addition doesn't modify any existing functionality - it simply adds new capabilities for services integration.

kanalive avatar Mar 23 '25 04:03 kanalive

I will test them out soon, as I don't have much time these days!

itsOwen avatar Mar 24 '25 00:03 itsOwen