Incorrect output during Query

[innovate-invent]

I have two files where some tokens are disjoint between the files (regardless of the set they belong to). The following line filters any tokens from the query set that are not present in the index file:

https://github.com/ekzhu/SetSimilaritySearch/blob/348a3c9f45f05bfe92ac36d40d557b860eecb83b/SetSimilaritySearch/search.py#L74

This generates erroneous Jaccard calculations as the filtered size of the query set is being used.

I have included some data to reproduce this: jaccard.csv query.txt index.txt

set_ID_x	set_ID_y	set_size_x	set_size_y	similarity
CP000438.1:1214715	PAGI-9|6|6	5	1	0.5

Here is a reduced dataset: index.txt query.txt

Debug Output

2022-07-12 17:16:30,568: Reading set tuples from /home/nolan/Projects/curated_GIs/gene_order.txt (reversed=False)...
2022-07-12 17:16:30,591: 24737 tuples read.
2022-07-12 17:16:30,591: Creating sets...
2022-07-12 17:16:30,600: 379 sets created.
2022-07-12 17:16:30,601: Reading set tuples from /home/nolan/Projects/curated_GIs/wild_gis/pseudomonas/genes_blast_sets.tabular (reversed=False)...
2022-07-12 17:16:30,672: 67023 tuples read.
2022-07-12 17:16:30,672: Creating sets...
2022-07-12 17:16:30,682: 3010 sets created.
2022-07-12 17:16:30,687: Building search index on 379 sets.
2022-07-12 17:16:30,687: Building SearchIndex on 379 sets.
2022-07-12 17:16:30,687: Start frequency transform.
2022-07-12 17:16:30,687: Applying frequency order transform on tokens...
2022-07-12 17:16:30,700: Done applying frequency order.
2022-07-12 17:16:30,700: Finish frequency transform, 14048 tokens in total.
2022-07-12 17:16:30,700: Start indexing sets.
2022-07-12 17:16:30,705: Finished indexing sets.
2022-07-12 17:16:30,705: Finished building search index.
2022-07-12 17:16:30,705: Find pairs with similarity >= 0.5.
2022-07-12 17:16:30,705: 29 original tokens and 10 tokens after applying frequency order.
2022-07-12 17:16:30,705: 1 candidates found.
2022-07-12 17:16:30,705: 0 verified sets found.
2022-07-12 17:16:30,705: 9 original tokens and 5 tokens after applying frequency order.
2022-07-12 17:16:30,705: 0 candidates found.
2022-07-12 17:16:30,705: 0 verified sets found.
2022-07-12 17:16:30,705: 29 original tokens and 11 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 25 original tokens and 7 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 53 original tokens and 18 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 31 original tokens and 11 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 5 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 3 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 3 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 24 original tokens and 13 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 44 original tokens and 14 tokens after applying frequency order.
2022-07-12 17:16:30,706: 1 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 13 original tokens and 5 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 1 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 19 original tokens and 7 tokens after applying frequency order.
2022-07-12 17:16:30,706: 0 candidates found.
2022-07-12 17:16:30,706: 0 verified sets found.
2022-07-12 17:16:30,706: 69 original tokens and 22 tokens after applying frequency order.
2022-07-12 17:16:30,706: 1 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 3 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 1 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 11 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 34 original tokens and 11 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 55 original tokens and 18 tokens after applying frequency order.
2022-07-12 17:16:30,707: 1 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 9 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 13 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 1 original tokens and 0 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 55 original tokens and 20 tokens after applying frequency order.
2022-07-12 17:16:30,707: 1 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 31 original tokens and 8 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 15 original tokens and 8 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,707: 23 original tokens and 8 tokens after applying frequency order.
2022-07-12 17:16:30,707: 0 candidates found.
2022-07-12 17:16:30,707: 0 verified sets found.
2022-07-12 17:16:30,708: 5 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 5 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 3 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 5 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 17 original tokens and 8 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 5 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 17 original tokens and 7 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 5 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 24 original tokens and 8 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 9 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 11 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 1 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 21 original tokens and 13 tokens after applying frequency order.
2022-07-12 17:16:30,708: 1 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 11 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,708: 0 candidates found.
2022-07-12 17:16:30,708: 0 verified sets found.
2022-07-12 17:16:30,708: 32 original tokens and 12 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 23 original tokens and 6 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 11 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 40 original tokens and 20 tokens after applying frequency order.
2022-07-12 17:16:30,709: 3 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 25 original tokens and 10 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 19 original tokens and 7 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 17 original tokens and 6 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 44 original tokens and 13 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 3 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 9 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,709: 5 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,709: 0 candidates found.
2022-07-12 17:16:30,709: 0 verified sets found.
2022-07-12 17:16:30,710: 23 original tokens and 6 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 50 original tokens and 17 tokens after applying frequency order.
2022-07-12 17:16:30,710: 1 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 11 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 3 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 3 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 9 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 5 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 7 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 9 original tokens and 5 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 35 original tokens and 17 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 1 original tokens and 0 tokens after applying frequency order.
2022-07-12 17:16:30,710: 0 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,710: 55 original tokens and 17 tokens after applying frequency order.
2022-07-12 17:16:30,710: 1 candidates found.
2022-07-12 17:16:30,710: 0 verified sets found.
2022-07-12 17:16:30,711: 17 original tokens and 6 tokens after applying frequency order.
2022-07-12 17:16:30,711: 0 candidates found.
2022-07-12 17:16:30,711: 0 verified sets found.
2022-07-12 17:16:30,711: 15 original tokens and 7 tokens after applying frequency order.
2022-07-12 17:16:30,711: 0 candidates found.
2022-07-12 17:16:30,711: 0 verified sets found.
2022-07-12 17:16:30,711: 30 original tokens and 14 tokens after applying frequency order.
2022-07-12 17:16:30,711: 1 candidates found.
2022-07-12 17:16:30,711: 0 verified sets found.
2022-07-12 17:16:30,711: 23 original tokens and 7 tokens after applying frequency order.
2022-07-12 17:16:30,711: 0 candidates found.
2022-07-12 17:16:30,711: 0 verified sets found.
2022-07-12 17:16:30,711: 11 original tokens and 6 tokens after applying frequency order.
2022-07-12 17:16:30,711: 0 candidates found.
2022-07-12 17:16:30,711: 0 verified sets found.
2022-07-12 17:16:30,711: 39 original tokens and 15 tokens after applying frequency order.
2022-07-12 17:16:30,711: 1 candidates found.
2022-07-12 17:16:30,711: 0 verified sets found.
2022-07-12 17:16:30,711: 5 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,711: 0 candidates found.
2022-07-12 17:16:30,711: 0 verified sets found.
2022-07-12 17:16:30,711: 18 original tokens and 5 tokens after applying frequency order.
2022-07-12 17:16:30,711: 0 candidates found.
2022-07-12 17:16:30,711: 0 verified sets found.
2022-07-12 17:16:30,711: 13 original tokens and 5 tokens after applying frequency order.
2022-07-12 17:16:30,711: 0 candidates found.
2022-07-12 17:16:30,711: 0 verified sets found.
2022-07-12 17:16:30,711: 17 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,711: 0 candidates found.
2022-07-12 17:16:30,711: 0 verified sets found.
2022-07-12 17:16:30,712: 75 original tokens and 29 tokens after applying frequency order.
2022-07-12 17:16:30,712: 2 candidates found.
2022-07-12 17:16:30,712: 0 verified sets found.
2022-07-12 17:16:30,712: 83 original tokens and 30 tokens after applying frequency order.
2022-07-12 17:16:30,712: 0 candidates found.
2022-07-12 17:16:30,712: 0 verified sets found.
2022-07-12 17:16:30,712: 16 original tokens and 5 tokens after applying frequency order.
2022-07-12 17:16:30,712: 0 candidates found.
2022-07-12 17:16:30,712: 0 verified sets found.
2022-07-12 17:16:30,712: 13 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,712: 0 candidates found.
2022-07-12 17:16:30,712: 0 verified sets found.
2022-07-12 17:16:30,712: 14 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,712: 0 candidates found.
2022-07-12 17:16:30,712: 0 verified sets found.
2022-07-12 17:16:30,712: 23 original tokens and 9 tokens after applying frequency order.
2022-07-12 17:16:30,712: 0 candidates found.
2022-07-12 17:16:30,712: 0 verified sets found.
2022-07-12 17:16:30,712: 7 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,712: 0 candidates found.
2022-07-12 17:16:30,712: 0 verified sets found.
2022-07-12 17:16:30,712: 15 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,712: 0 candidates found.
2022-07-12 17:16:30,712: 0 verified sets found.
2022-07-12 17:16:30,712: 1 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,712: 0 candidates found.
2022-07-12 17:16:30,712: 0 verified sets found.
2022-07-12 17:16:30,712: 41 original tokens and 14 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 59 original tokens and 20 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 43 original tokens and 17 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 5 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 9 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 7 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 3 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 32 original tokens and 9 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 9 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,713: 0 candidates found.
2022-07-12 17:16:30,713: 0 verified sets found.
2022-07-12 17:16:30,713: 39 original tokens and 11 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 24 original tokens and 9 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 13 original tokens and 5 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 9 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 1 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 13 original tokens and 6 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 3 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 5 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 21 original tokens and 7 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 1 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 117 original tokens and 51 tokens after applying frequency order.
2022-07-12 17:16:30,714: 4 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 35 original tokens and 11 tokens after applying frequency order.
2022-07-12 17:16:30,714: 1 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,714: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,714: 0 candidates found.
2022-07-12 17:16:30,714: 0 verified sets found.
2022-07-12 17:16:30,715: 52 original tokens and 16 tokens after applying frequency order.
2022-07-12 17:16:30,715: 1 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 9 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 15 original tokens and 9 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 23 original tokens and 7 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 1 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 13 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 5 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 5 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 17 original tokens and 5 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 39 original tokens and 12 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 11 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 11 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,715: 5 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,715: 0 candidates found.
2022-07-12 17:16:30,715: 0 verified sets found.
2022-07-12 17:16:30,716: 79 original tokens and 26 tokens after applying frequency order.
2022-07-12 17:16:30,716: 1 candidates found.
2022-07-12 17:16:30,716: 0 verified sets found.
2022-07-12 17:16:30,716: 15 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,716: 0 candidates found.
2022-07-12 17:16:30,716: 0 verified sets found.
2022-07-12 17:16:30,716: 3 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,716: 0 candidates found.
2022-07-12 17:16:30,716: 0 verified sets found.
2022-07-12 17:16:30,716: 3 original tokens and 1 tokens after applying frequency order.
2022-07-12 17:16:30,716: 0 candidates found.
2022-07-12 17:16:30,716: 0 verified sets found.
2022-07-12 17:16:30,716: 11 original tokens and 4 tokens after applying frequency order.
2022-07-12 17:16:30,716: 0 candidates found.
2022-07-12 17:16:30,716: 0 verified sets found.
2022-07-12 17:16:30,716: 9 original tokens and 3 tokens after applying frequency order.
2022-07-12 17:16:30,716: 0 candidates found.
2022-07-12 17:16:30,716: 0 verified sets found.
2022-07-12 17:16:30,716: 29 original tokens and 9 tokens after applying frequency order.
2022-07-12 17:16:30,716: 0 candidates found.
2022-07-12 17:16:30,716: 0 verified sets found.
2022-07-12 17:16:30,716: 3 original tokens and 2 tokens after applying frequency order.
2022-07-12 17:16:30,716: 0 candidates found.
2022-07-12 17:16:30,716: 0 verified sets found.
2022-07-12 17:16:30,716: 17 original tokens and 6 tokens after applying frequency order.
2022-07-12 17:16:30,716: 0 candidates found.
2022-07-12 17:16:30,716: 0 verified sets found.
...
2022-07-12 17:16:30,944: 17 original tokens and 6 tokens after applying frequency order.
2022-07-12 17:16:30,945: 0 candidates found.
2022-07-12 17:16:30,945: 0 verified sets found.
2022-07-12 17:16:30,945: 11 original tokens and 5 tokens after applying frequency order.
2022-07-12 17:16:30,945: 0 candidates found.
2022-07-12 17:16:30,945: 0 verified sets found.
2022-07-12 17:16:30,945: Found 21 pairs.
2022-07-12 17:16:30,945: Average query time: 7.890807433777869e-05.
2022-07-12 17:16:30,945: Median query time: 7.890807433777869e-05.
2022-07-12 17:16:30,945: 90pct query time: 0.00011157989501953125.