Implement CORS handling with --disable-web-security in BrowserManager…

[Ahmed-Tawfik94]

Summary

This change enhances support for the --disable-web-security Chromium flag in Crawl4AI to properly bypass CORS restrictions during JavaScript execution. Previously, using this flag in BrowserConfig.extra_args would fail because it requires a persistent browser context and a dedicated user data directory. The fix detects when --disable-web-security is present, automatically creates a temporary user data directory if none is provided, and launches the browser using launch_persistent_context instead of the standard launch method.

This resolves issues where XMLHttpRequest calls in custom JS code would be blocked by CORS policies, even when the flag was intended to disable them. Fixes #695 (CORS blocking XMLHttpRequest in JS execution).

List of files changed and why

• browser_manager.py - Modified the start() method in BrowserManager to detect --disable-web-security in extra_args, create a temp user_data_dir if needed, and use launch_persistent_context for proper CORS bypass functionality.
• test_browser_manager_cors.py - Added comprehensive pytest tests to verify CORS bypass functionality and ensure no regression in browser manager behavior.

How Has This Been Tested?

• Created and ran pytest tests in test_browser_manager_cors.py that execute JavaScript code using XMLHttpRequest to fetch a CSV file from raw.githubusercontent.com (a cross-origin URL that normally blocks CORS).
• Verified that with --disable-web-security enabled, the XMLHttpRequest succeeds and returns data; the tests confirm the fix works in headless mode.
• Tested normal browser launch functionality to ensure no regression when the flag is not used.
• All tests pass, demonstrating that the CORS bypass works as expected while maintaining backward compatibility.

Checklist:

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation (documentation update can be done in a follow-up PR if needed)
• [x] I have added/updated unit tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes