bootcamp
bootcamp copied to clipboard
milvus-io
[BUG]: Reverse Image Search Very Slow
Is there an existing issue for this?
- [X] I have searched the existing issues
Current Behavior
When I index an image via the API using http://127.0.0.1:5000/img/upload?url=[imageUrl]
It takes approximately 60 seconds to respond successfully. I've run this multiple times.
When I perform a search using Web Client it takes approximately 60 seconds for a successful response to come back.
Expected Behavior
Based on what I've read I would expect results to come back nearly instantly.
Steps To Reproduce
Follow the steps here https://github.com/milvus-io/bootcamp/tree/master/solutions/reverse_image_search/quick_deploy to launch a Docker container.
Then attempt to index an image via a URL using the api like such http://127.0.0.1:5000/img/upload?url=[imageUrl]
Software version
I'm not sure what versions these are running. I'm just launching the docker file.
version: '3.5'
services:
etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.0
networks:
app_net:
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2020-12-03T00-03-10Z
networks:
app_net:
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.0.2
networks:
app_net:
ipv4_address: 172.16.238.10
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
ports:
- "19530:19530"
depends_on:
- "etcd"
- "minio"
mysql:
platform: linux/amd64
container_name: img-search-mysql
image: mysql:5.7
networks:
app_net:
ipv4_address: 172.16.238.11
environment:
- MYSQL_ROOT_PASSWORD=123456
ports:
- "3306:3306"
webserver:
container_name: img-search-webserver
image: milvusbootcamp/img-search-server:towhee0.6
networks:
app_net:
ipv4_address: 172.16.238.12
environment:
MILVUS_HOST: '172.16.238.10'
MYSQL_HOST: '172.16.238.11'
volumes:
- ./data:/data
restart: always
depends_on:
- standalone
- mysql
ports:
- "5000:5000"
webclient:
container_name: img-search-webclient
image: milvusbootcamp/img-search-client:1.0
networks:
app_net:
ipv4_address: 172.16.238.13
environment:
API_URL: 'http://127.0.0.1:5000'
ports:
- "8001:80"
healthcheck:
test: ["CMD", "curl", "-f", "http://127.0.0.1:8001"]
interval: 30s
timeout: 20s
retries: 3
networks:
app_net:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.16.238.0/24
gateway: 172.16.238.1
### Anything else?
_No response_
Note, my dataset only currently consists of 2 images, which are very small images.
After looking into this further I'm assuming this has to do with the fact that I'm running this on a Mac M1 Silicon and the fact that these images are built using the AMD64.
I set this up on my Mac Mini that has an intel based processor and it works 100% better! Search responses are still taking 600ms to respond so that's not super promising. Not sure if there are ways to increase that since I only have 4 images in the index right now.
Hi @selected-pixel-jameson, I think
Maybe you are right, it shows the warning. Is there any doc about this assuming, can you share it?
I set this up on my Mac Mini that has an intel based processor and it works 100% better! Search responses are still taking 600ms to respond so that's not super promising. Not sure if there are ways to increase that since I only have 4 images in the index right now.
I think the most time-consuming is embedding, you can test the performance in your own machine.
And you can try to update the embedding pipeline or use GPU to improve:
- redefine the embedding_pipeline
Run dokcer exec -ti img-search-webserver bash to go into the container and modify server/src/encode.py:
import towhee
class Resnet50:
def __init__(self):
self.pipe = (towhee.dummy_input()
.image_decode()
.image_embedding.timm(model_name='resnet50')
.tensor_normalize()
.as_function())
def resnet50_extract_feat(self, img_path):
return self.pipe(img_path)
then you need to install the latest towhee with pip install towhee=0.8.0, finally restart the container with docker restart img-search-webserver .
- set GPU
If there is GPU in your mechine you can set GPU when starting the container, first modify the docker-compose.yaml, add the following line on webserver image(after line 68):
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
Then you need to rerun docker-compose up -d which means you need to start new server. And then you can run dokcer exec -ti img-search-webserver bash to go into the container and run the following python code to check the torch.cuda:
$ python3
>>> import torch
>>> torch.cuda.is_available()
True
If it prints true, it means towhee will automatically use GPU for embedding. It is more recommended to use the source code to run it on your local machine.
Please let me know if the solution above works for you, actually it works in my env :)
Thank you for the response.
I'm getting an error when I try to run docker-compose up -d after I modify docker-compose.yaml.
version: '3.5'
services:
etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.0
networks:
app_net:
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2020-12-03T00-03-10Z
networks:
app_net:
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.0.2
networks:
app_net:
ipv4_address: 172.16.238.10
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
ports:
- "19530:19530"
depends_on:
- "etcd"
- "minio"
mysql:
container_name: img-search-mysql
image: mysql:5.7
networks:
app_net:
ipv4_address: 172.16.238.11
environment:
- MYSQL_ROOT_PASSWORD=123456
ports:
- "3306:3306"
webserver:
container_name: img-search-webserver
image: milvusbootcamp/img-search-server:towhee0.6
networks:
app_net:
ipv4_address: 172.16.238.12
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
environment:
MILVUS_HOST: '172.16.238.10'
MYSQL_HOST: '172.16.238.11'
volumes:
- ./data:/data
restart: always
depends_on:
- standalone
- mysql
ports:
- "5000:5000"
webclient:
container_name: img-search-webclient
image: milvusbootcamp/img-search-client:1.0
networks:
app_net:
ipv4_address: 172.16.238.13
environment:
API_URL: 'http://127.0.0.1:5000'
ports:
- "8001:80"
healthcheck:
test: ["CMD", "curl", "-f", "http://127.0.0.1:8001"]
interval: 30s
timeout: 20s
retries: 3
networks:
app_net:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.16.238.0/24
gateway: 172.16.238.1
I had to update my docker-compose version. Apparently it needs to be at least 1.28+ https://stackoverflow.com/questions/66298638/devices-properties-is-not-allowed-while-creating-docker-compose-with-nvidia
I'm now receiving this error.
Recreating img-search-webserver ... error
ERROR: for img-search-webserver Cannot start service webserver: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown
ERROR: for webserver Cannot start service webserver: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown
ERROR: Encountered errors while bringing up the project.
Which I believe is due to the fact that my computer does not have an NVIDA GPU. But I'm not sure.
So I was unable to get the GPU settings enabled. But I did update the embedding in the encode.py file.
However, this has not improved in the indexing at all. In fact it has gotten worse. It now takes about 4 seconds to perform an index and still takes about 600 - 700ms to return a result.
@selected-pixel-jameson Can you try the latest code for reverse image search?