The goal of that experiment is to measure (subjectively) the relevancy of 4 search ranking algorithms to retrieve useful content.
It does compare the traditional well known algorithms: Okapi BM25 implemented in a most
common search engine software, Google ranking (very uniq but the dominant player here and a reference point) and 2 approaches using
sentence embeddings: ELSER from elastic.co and all-MiniLM-L6-v2 from sbert.net. Text embeddings combined with vector search
and applied to semantic search is an approach intensively discussed in recent times, following the progress of LLM.
The dataset is ~9000 automotive related questions searchable here https://www.carsguide.com.au/ask-the-guide.
questions are relatively small: median length is 54 words
The dataset is previously indexed for each algo, only the question are indexed (ie not the answers), except for Google.
16 queries or questions ran for each algorithm manually prepared covering some popular common questions in that dataset
4 have exact matches with an existing question, a control test
4 are reformulations of existing questions, but using synonyms, to test the semantic capacity
4 are non-existing/random questions in the dataset to test “overall” capacity
4 are few keywords-search having multiple matches in the dataset to test the “Google-way” to query.
Trimmed down to the first 5 hits
Google experiment is special because it couldn’t be run (obviously) automatically in a test environment like the others:
was run manually on google.com with Firefox browser based in Australia, in incognito mode, in October 2023 (no bard)
considering organic results only (ie disregard sponsor links and featured results) overridden
Using search operator site to limit results to the indexed questions site:www.carsguide.com.au/car-advice/q-and-a/
Google index contains not only the question but also the response on the page (and all the other related page content). And we cannot change that! I don’t think it should impact much the ranking since the test queries are crafted against the questions, but it should be keep in mind.
Search Queries
3 types of questions:
1 to 4: exact matches
5 to 8: reformulation of existing questions
9 to 12: non-existing/random questions
13 to 16: keywords search
How to Use Bluetooth in a Suzuki Swift?
What is the best oil type for my Ford Ranger, and is it possible to change the oil myself?
Where are Range Rovers made?
Who owns Rolls Royce?
What is the normal oil for a Honda CR-V and is it an easy DIY job to change?
original: What’s the correct oil type for my Honda CR-V, and is it tricky to change it yourself?
Is it legal in Victoria to sell a car without a RWC, if so, what are the correct steps?
original: In Victoria, can I sell my car without a roadworthy and, if so, what is the correct legal process?
Why electric cars are not builtin with PV?
original: Why don’t electric cars have solar panels?
What does the acronym LDV mean?
original: What does LDV stand for?
How to change engine oil on my Toyota Prius?
What is the difference between Diesel, Regular Petrol and Unleaded 91, Unleaded E10?