This repo is deprecated. We are developing a new tool for visualize anns index. Take a look at https://github.com/zilliztech/feder

vector-index-visualization-tool

Overview

We provide visualizations for the process of ANNS. Currently, it supports IVFFlat and HNSW.

• Backend server
  • Vector data store and mapping
  • Index building / search
  • Projection utils
• Web for visualizations
  • Design for IVFFlat and HNSW
  • Technicals: including voronoi, d3-force-links/radius/collision, DR project methods (pca, mds, t-sne, umap)

Quick start

Preparing

• New conda env (python >= 3.6)

$ conda create -n myenv python=3.6
$ conda activate myenv

• faiss >= 1.7.1 (Installing Faiss via conda)

# CPU-only version
$ conda install -c pytorch faiss-cpu

# GPU(+CPU) version
$ conda install -c pytorch faiss-gpu

# or for a specific CUDA version
$ conda install -c pytorch faiss-gpu cudatoolkit=10.2 # for CUDA 10.2 

• hnswlib4vis

  • Download and install wheel directly.
  • compile C++ to py (hnswlib4vis).

• Dataset (images and embedding vectors)

  • images from VOC2012
    • put image_files("VOCdevkit/VOC2012/JPEGImages") to "./server/data/images/"
  • vectors embedding by ResNet
    • put vector.csv to "./server/data/"
  • you can also use your own data including images and embedding vectors.

Build and Run

• Clone this repo

$ git clone [email protected]:zilliztech/data-visualization-tool.git

• Run server

$ cd server/
$ pip install -r requirements.txt
$ python api.py

• Run web

$ yarn start

IVFFlat Visualization

The visiting records of vectors in IVFFlat comes from the python api of faiss. (details on server/index/faiss_ivf.py)

HNSW Visualization

Getting the visiting records of vectors in HNSW comes by modifying the source cpp of hnswlib, then compiling to python. (details on hnswlib4vis and server/index/hnswlib_hnsw.py)

Join us

Welcome to Zilliz!

Acknowledgments

• faiss
• hnswlib
• d3