crawl4ai icon indicating copy to clipboard operation
crawl4ai copied to clipboard

Getting different bugs on two of the versions.

Open EntropyDev opened this issue 1 year ago β€’ 1 comments

Please look into following errors for respective versions.

  1. Environment Using Python 3.12.2 environment at: venv Name: crawl4ai Version: 0.3.731

Error Process HTML, Failed to extract content from the website: "LINK", error: 'NoneType' object has no attribute 'select'

( After pip install 'crawl4ai==0.3.74' --force-reinstall )

  1. Environment Using Python 3.12.2 environment at: venv Name: crawl4ai Version: 0.3.74 Requires: aiofiles, aiosqlite, beautifulsoup4, colorama, html2text, litellm, lxml, numpy, pillow, playwright, python-dotenv, rank-bm25, requests, snowballstemmer, tf-playwright-stealth, xxhash

Error np.float_ was removed in the NumPy 2.0 release. Use np.float64 instead

EntropyDev avatar Dec 02 '24 16:12 EntropyDev

I later tried uninstalling all the pip packages from the environment and then set crawl4ai==0.3.74 and reinstalled everything from requirements. And now, it seems to be working.

EntropyDev avatar Dec 03 '24 07:12 EntropyDev

@EntropyDev Thx for trying Crawl4ai. Your errors likely stemmed from outdated dependencies and missing elements in the older versions of crawl4ai. Upgrading to a newer stable release (e.g., 0.4.1) and reinstalling the requirements in a clean environment fixes these issues. In particular:

  • The NoneType error often comes from malformed or empty HTML. Later versions have improved handling.
  • The np.float_ issue occurs if you’re using NumPy 2.0.0 or later. Reinstalling in a fresh environment with pinned dependencies ensures a compatible NumPy version.

After upgrading and reinstalling, as you mentioned, it now works.

unclecode avatar Dec 09 '24 06:12 unclecode