Will indexing improve varchar(max) query performance, and how to create index

Question

Firstly, I should point out I don't have much knowledge on SQL Server indexes.

My situation is that I have an SQL Server 2008 database table that has a varchar(max) column usually filled with a lot of text.

My ASP.NET web application has a search facility which queries this column for keyword searches, and depending on the number of keywords searched for their may be one or many LIKE '%keyword%' statements in the SQL query to do the search.

My web application also allows searching by various other columns in this table as well, not just that one column. There is also a few joins from other tables too.

My question is, is it worthwhile creating an index on this column to improve performance of these search queries? And if so, what type of index, and will just indexing the one column be enough or do I need to include other columns such as the primary key and other searchable columns?

Answer 1

It's not worthwhile creating a regular index if you're doing LIKE '%keyword%' searches. The reason is that indexing works like searching a dictionary, where you start in the middle then split the difference until you find the word. That wildcard query is like asking you to lookup a word that contains the text "to" or something-- the only way to find matches is to scan the whole dictionary.

You might consider a full-text search, however, which is meant for this kind of scenario (see here).

Answer 2

The best analogy I've ever seen for why an index won't help '%wildcard%' searches:

Take two people. Hand each one the same phone book. Say to the person on your left:

Tell me how many people are in this phone book with the last name "Smith."

Now say to the person on your right:

Tell me how many people are in this phone book with the first name "Simon."

An index is like a phone book. Very easy to seek for the thing that is at the beginning. Very difficult to scan for the thing that is in the middle or at the end.

Every time I've repeated this in a session, I see light bulbs go on, so I thought it might be useful to share here.

Answer 3

you cannot create an index on a varchar(max) field. The maximum amount of bytes on a index is 900. If the column is bigger than 900 bytes, you can create the index but any insert with more then 900 bytes will fail.

I suggest you to read about fulltext search. It should suits you in this case

Answer 4

The best way to find out is to create a bunch of test queries that resemble what would happen in real life and try to run them against your DB with and without the index. However, in general, if you are doing many SELECT queries, and little UPDATE/DELETE queries, an index might make your queries faster.

However, if you do a lot of updates, the index might hurt your performance, so you have to know what kind of queries your DB will have to deal with before you make this decision.