Vec1

Tests on Publicly Available Data
Login

This page contains some tests run on publicly available datasets distributed for testing ANN systems. The following datasets are used:

All four datasets are distributed with sample query vectors and results. "sift1m" is distributed with dedicated training data (separate from the dataset itself), and the others are trained by sampling the main dataset. In all cases training and building the index are done with 16 threads. The "Training time" reported below is the wall-clock time the system spent running vec1_train() to create the mode. The "Build time" is the wall-clock time spent running the 'rebuild' command to build the index.

Queries are run with a variety of values for parameters K and nprobe and the recall and throughput (queries/second) reported. The reported throughputs are for a single thread only.

Two flavours of recall are reported - recall@1 and recall@10. Recall@1 is the proportion of queries for the single nearest neighbour that do, in fact return the true nearest neighbour. So a recall@1 of .916 means that if 1000 queries for the nearest neighbour are run, 916 of them return the true nearest neighbour.

Recall@10 is the average proportion of the true 10 nearest neighbours actually returned when the index is queried for the best 10 matches. i.e. if of the 10 results the query returns 7 of them are actually in the best 10 matches, the recall@10 is 0.7. The order of results returned does not matter - only the proportion that are part of the actual 10 best matches.

The SQL used for each query is:

        SELECT * 
        FROM vec1tbl($query, '{K: $K, nprobe: $nprobe}')
        ORDER BY vec1_l2_distance($query, vec1tbl.vector) 
        LIMIT $recall

Where $query is the query vector, $K is replaced by query parameter K, $nprobe by query parameter nprobe and $recall with the desired recall measure (1 or 10). For the "landmark-dino-768-cosine" index, vec1_cos_distance() is used instead of vec1_l2_distance(). In cases where $K==$recall the ORDER BY and LIMIT clauses are omitted from the query.

The results below were obtained on an AMD 5950X CPU based Linux workstation.

Dataset: sift-128-euclidean (1,000,000 128d vectors)

Vec1 Version: version 0.5 (AVX2, multi-threaded)
Index Parameters: {codesize:16, nbucket:1024, distance: "L2", opq: 0}
Training time: 2.43s (100,000 samples, 16 threads)
Build time: 1.32s (16 threads)

Recall@ nProbe K=1 K=10 K=50 K=100 K=200 K=300
Recall QPS Recall QPS Recall QPS Recall QPS Recall QPS Recall QPS
@10 16 0.558 6430 0.882 4666 0.919 3712 0.928 2661 0.929 2099
@10 32 0.567 3559 0.916 2925 0.962 2514 0.975 1985 0.977 1652
@10 48 0.569 2506 0.924 2161 0.973 1927 0.987 1594 0.990 1367
@10 64 0.569 1962 0.927 1729 0.976 1567 0.992 1340 0.995 1173
@1 16 0.462 6533 0.879 5885 0.949 4769 0.953 3817 0.954 2741 0.954 2158
@1 32 0.467 3630 0.902 3421 0.981 3013 0.987 2615 0.987 2014 0.987 1656
@1 48 0.468 2533 0.908 2426 0.989 2184 0.996 1948 0.997 1609 0.997 1380
@1 64 0.469 1945 0.909 1907 0.991 1733 0.998 1600 0.999 1354 0.999 1176

Dataset: imagenet-clip-512-normalized (1,281,167 512d vectors)

Vec1 Version: version 0.5 (AVX2, multi-threaded)
Index Parameters: {codesize:32, nbucket:1024, distance: "L2", opq: 1, residual: 0}
Training time: 30.0s (100,000 samples, 16 threads)
Build time: 7.88s (16 threads)

Recall@ nProbe K=1 K=10 K=50 K=100 K=200 K=300
Recall QPS Recall QPS Recall QPS Recall QPS Recall QPS Recall QPS
@10 16 0.531 3132 0.902 2523 0.959 2172 0.977 1702 0.980 1401
@10 32 0.533 1897 0.910 1648 0.968 1484 0.988 1246 0.992 1078
@10 48 0.534 1362 0.912 1225 0.971 1133 0.991 988 0.995 879
@10 64 0.534 1067 0.912 980 0.971 919 0.992 821 0.996 744
@1 16 0.426 3119 0.881 2675 0.984 2123 0.992 1998 0.992 1644 0.993 1399
@1 32 0.426 1917 0.884 1844 0.987 1664 0.995 1495 0.995 1255 0.996 1083
@1 48 0.428 1369 0.887 1331 0.990 1236 0.998 1138 0.998 992 0.999 883
@1 64 0.428 1071 0.888 1049 0.991 986 0.999 923 0.999 824 1.000 746

Dataset: landmark-dino-768-cosine (760,757 768d vectors)

Vec1 Version: version 0.5 (AVX2, multi-threaded)
Index Parameters: {codesize:48, nbucket:1024, distance: "cos", opq: 1, residual: 0}
Training time: 44.0s (100,000 samples, 16 threads)
Build time: 8.64s (16 threads)

Recall@ nProbe K=1 K=10 K=50 K=100 K=200 K=300
Recall QPS Recall QPS Recall QPS Recall QPS Recall QPS Recall QPS
@10 16 0.582 2495 0.905 2022 0.941 1754 0.950 1410 0.951 1196
@10 32 0.590 1571 0.927 1354 0.967 1218 0.978 1036 0.980 905
@10 48 0.593 1173 0.934 1048 0.976 963 0.988 844 0.990 755
@10 64 0.594 927 0.936 851 0.979 793 0.992 706 0.993 642
@1 16 0.554 2528 0.910 2233 0.971 1816 0.975 1691 0.975 1392 0.975 1206
@1 32 0.557 1580 0.919 1525 0.985 1363 0.989 1226 0.990 1040 0.990 909
@1 48 0.559 1159 0.924 1128 0.991 1034 0.995 949 0.996 831 0.996 741
@1 64 0.558 940 0.926 919 0.993 854 0.997 792 0.998 695 0.998 630

Dataset: agnews-mxbai-1024-euclidean (769,382 1024d vectors)

Vec1 Version: version 0.5 (AVX2, multi-threaded)
Index Parameters: {codesize:32, nbucket:1024, distance: "L2", opq: 1, residual: 0}
Training time: 58.2s (100,000 samples, 16 threads)
Build time: 13.3s (16 threads)

Recall@ nProbe K=1 K=10 K=50 K=100 K=200 K=300
Recall QPS Recall QPS Recall QPS Recall QPS Recall QPS Recall QPS
@10 16 0.619 2776 0.891 2176 0.931 1833 0.948 1387 0.952 1126
@10 32 0.625 2042 0.906 1708 0.951 1478 0.969 1174 0.974 973
@10 48 0.627 1608 0.911 1409 0.957 1252 0.977 1033 0.982 876
@10 64 0.627 1383 0.913 1200 0.959 1091 0.979 919 0.985 790
@1 16 0.616 2746 0.914 2336 0.959 1827 0.967 1726 0.969 1366 0.969 1125
@1 32 0.623 2065 0.928 1971 0.977 1719 0.986 1496 0.988 1182 0.988 976
@1 48 0.623 1646 0.932 1571 0.982 1405 0.992 1246 0.994 1022 0.994 868
@1 64 0.622 1393 0.933 1339 0.983 1215 0.992 1092 0.995 915 0.995 786