how to determine the accuracy of a semantic search engine?

Question

my cousin has created a semantic search engine and he claims that his search engine is the most accurate.

I've seen many semantic search engine and they all look the same to me, because they are not designed to give you results based on the keyword you type.

So if you are creating a semantic search engine, how to to determine the accuracy of its results?

Answer 1

Actually sarnold's suggestion is not far off the mark.

What you would typically do is to take a whole bunch of people and have them try out a bunch of standard queries. Or if you wanted to make the experiment fairer you might let each user pick their own queries to avoid any accusation of bias (because you could pick standard queries you knew your engine was good at answering).

For each query the user would look through the first 10 or so results and say whether they thought each result was relevant or not (you may want to have users score on a scale rather than just yes/no).

Then for each of the queries you can calculate accuracy scores, depending on exactly how you set up the experiment Precision and Recall may be the most appropriate measures though these rely on having a known expected answer which you may not necessarily have. It may be simpler and more appropriate to calculate a simple percentage accuracy.

To determine whether your search engine was better than your competitors you'd have the same people perform the same queries on those search engines scoring in the same way. Having done this you can then calculate and compare the scores for the search engines against your own.