! crawl4ai The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested 0.0s

[QuangTQV]

I am experiencing an issue when trying to run the Crawl4AI Docker image. The error message indicates that the requested image's platform does not match the detected host platform. Here are the details:

Host Platform: linux/amd64 Requested Image Platform: linux/arm64/v8 Error Message: exec /usr/local/bin/uvicorn: exec format error

[unclecode]

@QuangTQV Thanks for reporting this. I'll check and update within a day or two.

[jefcolbi]

Hi. I encounter the same error

[sol5288]

I have the same issue. I'm waiting for an update. thank you unclecode

[unclecode]

I've nearly found the issue. I hope to update the Dockerfile this week to support the nuances between the three operating systems. @QuangTQV @jefcolbi @sol5288

[jefcolbi]

Thank you. I tried building the container with docker build its succeeded but upon start it gives me an error.

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/main.py", line 324, in <module>
    app.mount("/mkdocs", StaticFiles(directory="site", html=True), name="mkdocs")
  File "/usr/local/lib/python3.10/site-packages/starlette/staticfiles.py", line 56, in __init__
    raise RuntimeError(f"Directory '{directory}' does not exist")
RuntimeError: Directory 'site' does not exist

[unclecode]

@jefcolbi I think I need to move documents to another branch. The documentation you see is because I mounted library documentations to /mkdocs. To fix this, first install the mkdocs command line and then build the documentation, which will regenerate the site folder for you. If you have mkdocs, just run 'mkdocs build' and it will handle it. Thanks for pointing this out, I'll separate this from Docker to avoid such issues.

[ivanberry]

I have this problems too. is there any method to fix it?

[unclecode]

Hey everyone, there have been some updates on the Docker folders and they're going to be in the new versions 0.3.74 which will be out by tonight or tomorrow, perhaps. Please try it and let me know if you have any issues with the Docker. @jefcolbi @ivanberry @QuangTQV

[jefcolbi]

I can't wait for it

[roliii]

It still doesn't work for me. WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested exec /usr/local/bin/uvicorn: exec format error

[andnig]

Hey folks, I think we need all to chill a bit, @unclecode is doing great work and as he mentioned 2 days ago, he is already working on it. The provided docker images are probably built on an ARM MacBook and therefore only work for arm for now. But that's the great thing about open source, we can help ourselves 😁

Until there is official support for more docker platforms, just follow these steps to build your own image. It's really easy.

Building crawl4ai docker image

1. Let's clone the repository to our machine:

    git clone https://github.com/unclecode/crawl4ai.git
   

2. Next, navigate into the cloned repository:

    cd crawl4ai
   

3. Open the file main.py and remove the following line from the file.

    app.mount("/mkdocs", StaticFiles(directory="site", html=True), name="mkdocs")
   

   This line would mount the Crawl4AI documentation and provide it as part of the web server. However, first we don't need it and secondly it would require a second build step, which we want to avoid for now.

   Note: This might change in the future. So if you can't find this line in the file, don't worry, just continue with the next steps.

4. Now execute the following command to build the docker image:

   docker build -f Dockerfile -t pondhouse/crawl4ai:beta-1 --build-arg INSTALL_TYPE=all .
   

(instead of pondhouse/crawl4ai:beta-1 please use your own image tag)

Note: instead of setting the INSTALL_TYPE to all, you can also set it to basic. The full version is required for AI powered extraction whereas if you just want the web-crawling without AI extraction, the basic version is sufficient.

Hope this helps for now.

[unclecode]

@andnig Thanks for the kind words 😍! I just updated the repo, version 0.3.74 is out now. The Docker stuff will take a bit more time, but next week it’ll be my main focus.

For now, I’ve added a one-click deployment on DigitalOcean, which seems fine, and resolved issues with the “site” folder. To be honest, I haven’t tested it as thoroughly as I’d like, so please help and report any issues. What I can promise is that in 2024, we’ll have a beautiful, efficient, blazing-fast Docker image! 🤞🤞💪💪

[jefcolbi]

Good work @unclecode @andnig thanks for the trick

[unclecode]

Hey everyone, right now I tested the Docker, and it seems to work on Windows, Linux, and Mac. Also, to run it on arm64, make sure to set the platform like this:

# Install from hub
docker run --platform linux/arm64 -p 11235:11235 unclecode/crawl4ai:basic

# Install from local
docker build -t crawl4ai:local \
  --build-arg INSTALL_TYPE=basic \  # Options: basic, all
  --platform linux/arm64 \
  .

If you want to use docker-compose, you can manually add the "platform" field and set the value you want.

Please check it and let me know if you face any issue, then share the error message plus your operating system, specs, so I can replicate the error.

[ivanberry]

version: '3'

services:
  crawl4ai:
    image: unclecode/crawl4ai:basic
    environment:
      - MAX_CONCURRENT_TASKS=5
      - platform=linux/amd64/v3
    ports:
      - "11235:11235"
    volumes:
      - ./data:/app/data
      ```
      ```
      [+] Pulling 1/1
 ✔ crawl4ai Pulled                                                                                                                                   6.5s 
(crawle4ai) a:~/app/crawl4ai$ docker compose up -d
[+] Running 1/1
 ✔ Container crawl4ai-crawl4ai-1  Started                                                                                                            0.0s 
(crawle4ai) a:~/app/crawl4ai$ docker compose ps
NAME                IMAGE               COMMAND             SERVICE             CREATED             STATUS              PORTS
(crawle4ai) a:~/app/crawl4ai$ docker compose up -d
[+] Running 2/0
 ✔ Container crawl4ai-crawl4ai-1                                                                                                                          [+] Running 2/21s 
 ✔ Container crawl4ai-crawl4ai-1                                                                                                                          [+] Running 2/21s 
 ✔ Container crawl4ai-crawl4ai-1                                                                                                                          [+] Running 2/21s 
 ✔ Container crawl4ai-crawl4ai-1                                                                                                                          [+] Running 2/21s 
 ✔ Container crawl4ai-crawl4ai-1                                                                                                                          [+] Running 2/21s 
 ✔ Container crawl4ai-crawl4ai-1                                                                                                                          [+] Running 2/21s 
 ✔ Container crawl4ai-crawl4ai-1                                                                                                                          [+] Running 2/21s 
 ✔ Container crawl4ai-crawl4ai-1                                                                                                                              Started0.1s 
 ! crawl4ai The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested 0.0s

Thanks for your efforts. but I am still not start the service. I did the image pull already

[jefcolbi]

@ivanberry uses the steps of @andnig it works well.

[QuangTQV]

Hey everyone, right now I tested the Docker, and it seems to work on Windows, Linux, and Mac. Also, to run it on arm64, make sure to set the platform like this:

# Install from hub
docker run --platform linux/arm64 -p 11235:11235 unclecode/crawl4ai:basic

# Install from local
docker build -t crawl4ai:local \
  --build-arg INSTALL_TYPE=basic \  # Options: basic, all
  --platform linux/arm64 \
  .

If you want to use docker-compose, you can manually add the "platform" field and set the value you want.

Please check it and let me know if you face any issue, then share the error message plus your operating system, specs, so I can replicate the error.

Docker still has a platform error; I don't want to have to modify the code. @unclecode

[unclecode]

@ivanberry you're welcome and would you please let me know your os platform specs? I have to be able to replicate the error

[unclecode]

@QuangTQV I try to set up a virtual server with your host platform: linux/amd64 and test over there, would you please confirm this specs?

[QuangTQV]

@QuangTQV I try to set up a virtual server with your host platform: linux/amd64 and test over there, would you please confirm this specs?

I am still getting this error: ! crawl4ai The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested 0.0s

[unclecode]

@QuangTQV Ok I will set up a virtual machine with host platform: linux/amd64 and test it this weekend. No matter what I get it work finally haha

[edenbuaa]

@QuangTQV I try to set up a virtual server with your host platform: linux/amd64 and test over there, would you please confirm this specs?

I am still getting this error: ! crawl4ai The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested 0.0s

same issue.

[unclecode]

Hello everyone, I had to create two virtual machines, one with ARM64 and the other one with AMD64, and I created two sets of Docker images. To make it more clear and easier to use, right now both are working properly under testing. Here, I have created shared documentation on how to use it. Please try it and let me know. I hope this finally puts an end to this Docker challenge.

@QuangTQV @edenbuaa @roliii @jefcolbi @ivanberry

Option 1: Docker Hub (Recommended)

Choose the appropriate image based on your platform and needs:

For AMD64 (Regular Linux/Windows):

# Basic version (recommended)
docker pull unclecode/crawl4ai:basic-amd64
docker run -p 11235:11235 unclecode/crawl4ai:basic-amd64

# Full ML/LLM support
docker pull unclecode/crawl4ai:all-amd64
docker run -p 11235:11235 unclecode/crawl4ai:all-amd64

# With GPU support
docker pull unclecode/crawl4ai:gpu-amd64
docker run -p 11235:11235 unclecode/crawl4ai:gpu-amd64

For ARM64 (M1/M2 Macs, ARM servers):

# Basic version (recommended)
docker pull unclecode/crawl4ai:basic-arm64
docker run -p 11235:11235 unclecode/crawl4ai:basic-arm64

# Full ML/LLM support
docker pull unclecode/crawl4ai:all-arm64
docker run -p 11235:11235 unclecode/crawl4ai:all-arm64

# With GPU support
docker pull unclecode/crawl4ai:gpu-arm64
docker run -p 11235:11235 unclecode/crawl4ai:gpu-arm64

Need more memory: Add --shm-size:

docker run --shm-size=2gb -p 11235:11235 unclecode/crawl4ai:basic-amd64

Test the installation:

curl http://localhost:11235/health

Option 2: Build from Repository

Build the image locally based on your platform:

# Clone the repository
git clone https://github.com/unclecode/crawl4ai.git
cd crawl4ai

# For AMD64 (Regular Linux/Windows)
docker build --platform linux/amd64 \
  --tag crawl4ai:local \
  --build-arg INSTALL_TYPE=basic \
  .

# For ARM64 (M1/M2 Macs, ARM servers)
docker build --platform linux/arm64 \
  --tag crawl4ai:local \
  --build-arg INSTALL_TYPE=basic \
  .

Build options:

• INSTALL_TYPE=basic (default): Basic crawling features
• INSTALL_TYPE=all: Full ML/LLM support
• ENABLE_GPU=true: Add GPU support

Example with all options:

docker build --platform linux/amd64 \
  --tag crawl4ai:local \
  --build-arg INSTALL_TYPE=all \
  --build-arg ENABLE_GPU=true \
  .

Run your local build:

# Regular run
docker run -p 11235:11235 crawl4ai:local

# With increased shared memory
docker run --shm-size=2gb -p 11235:11235 crawl4ai:local

Test the installation:

curl http://localhost:11235/health

[RemeLards]

The crawl endpoints doesn't work for me when I have mkdocs packages to install on the Dockerfile and the line @andnig said on the "main.py",because it says PlayWright wasn't installed. Had to remove both to make it work perfectly fine on Ubuntu 22.04

[unclecode]

@RemeLards would you plz share your Dockerfile after editing? Btw You need to rebuild the image and make sure all steps in the Dockerfile are followed. The existing Dockerfile already installs mkdocs and runs playwright install chromium, which ensures that Playwright is properly set up. Also confirm that mkdocs build is executed before you start the server. If you made any changes to the Dockerfile or the build steps, rebuild the image from scratch, and then run your container again. This will ensure that both mkdocs and Playwright are correctly installed and that the documentation is built before the server starts. Please let me know if you have issue with this.

[aravindkarnam]

Closing this issue due to inactivity. Reopen a new issue if the problem still exists.