iqdb

iqdb copied to clipboard

iqdb - Image Query Database system [email protected]

Distributed under the terms of the GNU General Public License, see the file COPYING for details.

1. Compiling

• install either the libGD+libpng+libjpeg, or the ImageMagick dev package Using GD, on Debian: libgd2-xpm-dev libpng12-dev libjpeg62-dev Using ImageMagick, on Debian: libmagick9-dev, on Fedora: ImageMagick-devel)
• check the top of the Makefile for settings
• run make
• if it fails, fix the Makefile or something

2. Running

The iqdb program has two operating modes: database maintenance and query server.

The maintenance mode is to update image databases: add and remove images, set image properties and retrieve database stats. The query server mode listens on a TCP port for query commands and returns matching image IDs.

a) Maintenance

Two major modes of calling the maintenance mode:

$ iqdb add foo.db

This mode expects a list of image IDs and filenames on stdin, in this format: <ID> [ ]:

The ID is the image ID in hexadecimal. If width and height are not specified, the values from the filename are used instead. The filename can be any image format supported by ImageMagick; the image will be resized to 128x128 and added to the database. Duplicate IDs are ignored. Note that iqdb does not remember the filename associated with an image ID, you are responsible for keeping track of that. It refers to an image exclusively by the ID.

A more complex mode allows add, removing, updating and querying images for multiple databases at once:

$ iqdb command foo.db bar.db baz.db

This mode expects commands on stdin. For a full list see do_commands() in iqdb.cpp. The argument refers to the database on the command line, with the first argument being dbid=0. Commands of particular interest are:

add <dbid> <imgid> [<width> <height>]:<filename>
	See above.
remove <dbid> <imgid>
	Remove the given image from the DB.
rehash <dbid>
	This commands updates the coefficient bucket sizes.
	It is normally not needed, and is inefficient because
	it needs to read in the entire database.
	(As a special feature, `iqdb rehash foo.db' can
	be used to upgrade an old version of the DB.)
done
	Quit and save all databases to their original file.
quit
	Quit and save all databases to their original file.

The maintenance commands (see listen mode below) are also supported in command mode, but perhaps not very useful here.

Changes are automatically saved in the original file upon EOF on stdin, or when sending the "done" or "quit" commands.

DO NOT interrupt the program with Ctrl-C or similar until the database is closed or it may become corrupted.

b) Server mode

In query server mode, iqdb loads the databases into memory in read-only mode to allow the fastest image queries. No database modifications are possible.

$ iqdb listen [IP:]port [-r] [-d=] [-s<IP/host>...] foo.db bar.db baz.db

Listens on the given IP:port (default localhost if no IP given) for commands, after loading the given databases. If -r is specified and the port is currently in use by another instance of iqdb, it will load the database and then terminate that instance, to reduce the downtime while loading the databases (and to leave the old server running in the rare case that it should crash while starting up). The debug level may be set with -d. The -s option allows limiting iqdb queries to the given source IPs. This is useful if you cannot bind iqdb to localhost, for instance because the query script and iqdb run on different hosts. Then you can limit iqdb requests to the host where the script is running. Note that you should probably also specify "-s<host-IP>" with the IP as given to the listen argument, to allow requests from the local host, for instance to add images to the database and to make the -r option work.

$ iqdb listen2 [IP:]port [options...] foo.db bar.db baz.db

Same as above, but listens on the given port and one port below it (i.e. if port 5588 is specified, also listens on 5587). The lower port is higher priority and all pending requests are serviced before the other port.

To end a request, send the "done" command.

Of particular interest are the following commands:

query <dbid> <flags> <numres> <filename>
	Find numres images most similar to given filename. If the
	filename starts with ':', use e.g. "./filename" to disambiguate
	it from the literal image data query below.
	Flags is a bitmask:
		0 = normal operation
		1 = file contains a sketch, use different weights
		2 = force grayscale match, discard color information
		8 = consider image width as set ID instead and
		    return only the best match for each set
		16= discard common coefficients (those present in at
		    least 10% of all images), this finds a
		    near-identical match much faster but the
		    non-matching images have less similarity

query <dbid> <flags> <numres> <:size>
	As above, but if the filename argument starts with ':', the
	given number of bytes of literal image data starts on the next
	line after the command. This allows querying without first
	writing the image data to a file, and to query images that don't
	locally exist on the host running the iqdb server.

multi_query <dbid> <flags> <numres> [+ <dbid2> <flags2> <numres2> +...] <filename>
multi_query <dbid> <flags> <numres> [+ <dbid2> <flags2> <numres2> +...] <:size>
	Merge query results from multiple databases. This adjusts the
	individual scores to deal with varying similarity noise levels
	in different databases (the noise level depends on the number
	of images in the DB and how self-similar they are).

sim <dbid> <flags> <numres> <id>
	Like the "query" function, but queries for images similar to
	the image with the given ID. This only works if the database
	has been loaded in "readonly" mode, e.g. using the "load"
	command described below.

query_opt <option> <arguments...>
	Set advanced options for the next query. Options are reset after
	the next query or multi_query command. Available options are:
	mask <and> <xor>
		Consider image height as bitmask and only return images
		for which ((bitmask & and) ^ xor) == 0. In other words,
		for each bit that is set in "and", the image mask's bit
		must match that of the "xor" mask. This can be used to
		limit search results to given rating masks for example.
	mindev <min.std.dev.>
		Return only results that are the given standard deviation
		above the noise level. Nothing is returned if there are
		no relevant results at all.

The "add" and "remove" commands are now supported as well, however they only modify the memory representation of the DB and cannot be saved back to disk later. They allow you to update the server without restarting it but require that you also update the DB file directly.

Additionally, since version 20081123 the listen mode also supports DB maintenance commands. In two-port mode these will only be accepted on the lower (high-priority) port.

load <dbid> <mode> <filename>
	Loads the given DB file into the given dbid (which must not
	be in use yet). The mode can be one of the following:
		simple	default listen mode, supports fast queries
			as well as adding/removing images
		readonly requires more memory but supports image
			ID queries (needed for duplicate finding)
		alter	default command mode, supports fast updates
			of DB files but no queries
		normal	full functionality but requires much memory
			and has slow queries, mainly useful for
			upgrading old DB versions

drop <dbid>
	Drops the given database. With drop and load commands in the
	same connection, a database will be reloaded or replaced
	atomically, however the server will be blocked until loading
	is complete. When loading a large DB it is recommended to
	start a new copy of the server with the -r option instead.

db_list
	Lists all loaded databases with dbid and filename.

The server has the following possible responses:

000 iqdb ready
	Displayed whenever iqdb is ready to
	accept a command.
100 <text>
	General informational message.
101 <key>=<value>
	Specific parseable information.
102 <dbid> <filename>
	Response to db_list command.
200 <imgid> <score> <width> <height>
	Query result.
201 <dbid> <imgid> <score> <width> <height>
	Multi-query result.
202 <original id>=<std.dev> <dupe1 id>:<sim1> [...]
	Duplicate finder result.
300 <text>
	General error message.
301 <exception> <description>
	Non-fatal error message (e.g. invalid image ID, or
	unreadable image file)
302 <exception> <description>
	Fatal error message (e.g. corrupted database)

3. Querying

The file iqdb.php holds sample PHP code to connect to a running iqdb server and queries it for similar images.

4. Converting

Since version 20090612, iqdb will automatically detect the integer sizes used for writing a given image database. It will automatically convert them to its internal format on reading. Writing a non-native image database is not possible.

This version also by default is compiled in 64-bit mode to make the image database file platform-independent.

It is possible, but not recommended however, to have iqdb convert an image database of an older version into this new platform-independent format. It's better to rebuild the database from scratch. If you do want to convert the database, first make a backup of it, in case the conversion should fail. Then run "iqdb rehash ".

If you haven't turned off debugging, this should output the following: Loading db (converting data sizes)... Database loaded from , has images. After loading, it will save the database again in the platform-independent format. Pay particular attention to the number of images it reports. If this isn't the number you expect, the conversion probably didn't work. In that case you will have to rebuild the database from scratch using the new version.