# Vector Search Benchmark[eting] Philipp Krenn, Elastic | AI Engineer World's Fair 2025 | 14:10 Source: https://www.youtube.com/watch?v=YrUBFXa1KUY Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/vector-search-benchmark-eting Published: 2025-06-27 Tags: benchmarks, evals, search ## TL;DR - Vendor benchmarks can make almost any system look faster by choosing a favorable workload, configuration, or statistic. - Vector search benchmarks must measure result quality alongside speed, because approximate nearest neighbor settings can trade precision and recall for latency. - A useful benchmark matches your data, queries, hardware, read-write ratio, acceptable latency, and configuration, then runs automatically and reproducibly over time. ## Summary Philipp Krenn explains why vector search benchmarks often produce contradictory claims. Vendors can choose a workload that favors their system, use read-only data even when real workloads include writes, select convenient hardware and memory settings, compare different software versions, or report an exceptional query as an overall result. Vector search adds another problem: approximate nearest neighbor search can return results faster by reducing search effort, so latency numbers are meaningless without precision and recall. Krenn recommends automated, reproducible benchmarks that run regularly, such as Elastic's nightly benchmarks, to catch gradual performance regressions. He also argues that teams need to build their own benchmarks around their data size, data structure, queries, read-write ratio, latency target, and hardware. Elastic's Rally tool lets users define data and queries in a track and tune settings. Even a flawed vendor benchmark can reveal the workload where that vendor believes its system performs well. ## Key ideas ### A vendor can make almost any vector system look faster [00:03](https://www.youtube.com/watch?v=YrUBFXa1KUY&t=3s) Krenn says benchmark claims are contradictory because nearly every vendor has published results showing it is faster than competitors, while other benchmarks show the reverse. Glossy charts often hide the choices behind the result. A company may define a scenario that suits its own system, test many scenarios, keep the one where it wins, and generalize that result into a broad claim such as being 40% faster. The chart may be attractive, but the workload may have little connection to a reader's system. ### The benchmark workload must match the system being built [01:43](https://www.youtube.com/watch?v=YrUBFXa1KUY&t=103s) Many vector search benchmarks use read-only data in an optimized format because that is easier to reproduce. Real applications often read and write data, so the benchmark needs an appropriate read-write ratio and the other parameters that shape the workload. Krenn says vendors can also benefit from choices such as shard size, total data size, memory allocation, compression, instance configuration, and access patterns. These choices may be intentional or may simply reflect what the vendor knows how to configure well. ### Filtering can make approximate vector search slower [03:42](https://www.youtube.com/watch?v=YrUBFXa1KUY&t=222s) Krenn describes a counterintuitive behavior in HNSW and approximate nearest neighbor search. In a conventional data store, a restrictive filter can reduce the number of records that must be ranked and make a query faster. With vector search, filtering can require the system to inspect more candidates before finding enough results that survive the filter. A benchmark can therefore favor one system by choosing filters that match its optimizations, or by omitting filtering behavior that matters to the real application. ### Speed without result quality is an incomplete benchmark [07:16](https://www.youtube.com/watch?v=YrUBFXa1KUY&t=436s) Approximate nearest neighbor systems expose a direct tradeoff between search effort and result quality. Different parameters or implementations can produce different precision and recall, so comparing latency without recording those settings compares different outcomes. Krenn says it is possible to return poor results very quickly, which is not useful if the application needs accurate retrieval. He recalls that even Elastic benchmarks were affected when people forgot to include quality measurements. ### Statistics can turn one unusual query into a broad claim [08:37](https://www.youtube.com/watch?v=YrUBFXa1KUY&t=517s) Krenn describes a benchmark with about 20 measurements where two systems were similar in 18 or 19 cases. One system had an optimization that made it much faster on a narrow case, possibly an ascending versus descending sort on one data type. The benchmark presentation adjusted the statistics so that this edge case helped create a large overall difference. The resulting headline claimed that one system was many times faster, even though the advantage came from one specific query. ### Regular automated runs catch slow performance regressions [09:50](https://www.youtube.com/watch?v=YrUBFXa1KUY&t=590s) Krenn recommends automated and reproducible benchmarks that run continuously. Elastic uses a broad set of nightly benchmarks and compares changes made during the day. This catches gradual regressions that are easy to miss when each individual change has a small effect. He compares the process to slowly heating a frog: a 1% slowdown today and another 2% slowdown later can eventually cause a large performance loss without a clear moment when anyone notices it. ### Teams need benchmarks built around their own workload [11:28](https://www.youtube.com/watch?v=YrUBFXa1KUY&t=688s) Krenn says there is no substitute for running a benchmark that matches the application. Teams need to specify their data size, data structure, read-write ratio, query shapes, acceptable latency, and hardware. Elastic's Rally tool lets users create a track containing the data and queries, then tune settings or hardware and measure what their setup can deliver. The same kind of workload must be applied to each system under evaluation if the comparison is meant to guide a purchase or architecture decision. ### A flawed benchmark can still reveal a vendor's preferred use case [12:58](https://www.youtube.com/watch?v=YrUBFXa1KUY&t=778s) Krenn advises against dismissing every result from a flawed benchmark. The benchmark may show the scenario where a vendor believes its system is strongest, the behavior it wants to emphasize, and the conditions under which it expects to perform well. That information can help teams form follow-up tests. The benchmark should not be accepted as a general answer, but its workload and reported strengths can still provide clues about what to measure next. ## Notable quotes - "Don't trust the glossy charts." (01:20) - "You can produce crap results very quickly, but that is probably not the point of what you do in your benchmarks." (07:58) - "You will need to do your own benchmarks." (11:28) - "There is no way around doing that yourself." (11:44) ## Tools & references mentioned - Elastic - Elasticsearch - Kibana - Beats - Logstash - HNSW - Rally - Volkswagen - precision and recall ## Who should watch - You are comparing vector databases and need results that reflect your own queries, data, hardware, and latency target. - Your team relies on vendor benchmark charts and needs to understand how filtering, approximate search settings, and result quality can change the conclusion. - You maintain a search system and want a practical case for automated nightly benchmarks that catch gradual regressions. ## Related talks - [Benchmarks: The Good, the Bad, and the Ugly](https://aietalks.com/talks/benchmarks-the-good-the-bad-and-the-ugly) (Ali Khial, G2i, 12:49) - [The Art & Science of Benchmarking Agents](https://aietalks.com/talks/the-art-science-of-benchmarking-agents) (Vincent Chen, Snorkel AI, 23:25) - [Evals Are Broken, Use Them Anyway](https://aietalks.com/talks/evals-are-broken-use-them-anyway) (Ara Khan, Cline, 19:04) - [Rethinking What State-of-the-Art Means](https://aietalks.com/talks/rethinking-what-state-of-the-art-means) (Bertrand Charpentier, Pruna AI, 19:36) - [Building Metrics that Actually Work](https://aietalks.com/talks/building-metrics-that-actually-work) (David Karam, Pi Labs, 40:28)