tiny-web-crawler icon indicating copy to clipboard operation
tiny-web-crawler copied to clipboard

Published 20 hours ago • verified publisher icon indrajithi

Feature: Support for crawling dynamic javascript heavy site

Open indrajithi opened this issue 1 year ago • 5 comments

Description:

Enhance the existing web crawler to support crawling and extracting content from websites that rely heavily on JavaScript for rendering their content. This feature will involve integrating a headless browser to accurately render and interact with such pages.

Objectives:

  • Enable the crawler to fetch and parse content from JavaScript-heavy sites.
  • Use a headless browser to render JavaScript content. (explore playwright-python)
  • Ensure compatibility with the existing crawler structure and options.
  • Maintain the ability to switch between the default fetching method and the headless browser.

Design Considerations:

  • Single Headless Browser Instance:
    • Use a single instance of a headless browser to handle multiple asynchronous requests, reducing resource consumption.
  • Concurrency Management:
    • Utilize asyncio and a semaphore to manage concurrent requests within the same browser context.
    • Integrate the asynchronous fetching logic with our existing web crawler structure.
  • Error Handling:
    • Ensure proper error handling and resource cleanup. (no zombie browsers, they are already headless :p)
    • Fall back to default fetching mode when there is a error with the headless browser. (keep the user informed)

indrajithi avatar Jun 14 '24 16:06 indrajithi

Blocked by #17

indrajithi avatar Jun 16 '24 12:06 indrajithi

What does blocked mean? I'd like to work on this but do you think I shouldn't?

Mews avatar Jun 17 '24 19:06 Mews

@Mews You can work on this. I want to complete #17 before picking this up. I have merged the MR for that. Although there are a few more things to be done for #17. I believe this issue can be unblocked.

Since this is going to be relatively bigger story. Let us first discuss the approach and spec out the requirements and acceptance criteria.

indrajithi avatar Jun 17 '24 19:06 indrajithi

Alright makes sense I'll work on the v1 milestone too then. I'll pick this up when v1 gets released 👍

Mews avatar Jun 17 '24 19:06 Mews

You can create Issues for things in #24 you find interesting and pick it up. Meanwhile I will spec out some details in this Issue. Also I think we should have this in v1.

indrajithi avatar Jun 17 '24 20:06 indrajithi